Enterprise Mainframe Change Data Capture (CDC) to Apache Kafka with tcVISION and Confluent

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc. and Ram Dhakne, Solutions Engineer at Confluent

___Mainframe_To_Kafka_Confluent

This blog focuses on using Treehouse Software’s tcVISION to replicate data in real time between mainframes and Confluent, allowing for new use cases and truly setting data in motion.

Why mainframe modernization? Benefits and use cases

Mainframe data stores often hold large amounts of complex and critical data in proprietary legacy formats, making this data difficult to extract and incompatible with modern databases, data types, and data tools.

Enterprises are looking to take advantage of the latest cloud services, such as analytics, artificial intelligence (AI) and machine learning, scalable storage, security, high availability, etc., or move data to a variety of newer databases. Additionally, many customers want to modernize their application on a cloud or open systems platform without disrupting the existing critical work on the legacy system.

How tcVISION syncs legacy data for the cloud

tcVISION is a data replication software product that performs real-time synchronization of mainframe data sources and cloud and open systems, allowing critical mainframe data to be consumed by a variety of leading cloud services.

tcVISION supports many mainframe data sources for both online and offline scenarios. Data can be replicated from IBM Db2 z/OS, Db2 z/VSE, VSAM, IMS/DB, CA IDMS, CA Datacom, or Software AG ADABAS. tcVISION can replicate data to many targets including Confluent Platform, Apache Kafka®, AWS, Google Cloud, Microsoft Azure, PostgreSQL, Snowflake, etc. To learn more, see the complete list of supported tcVISION sources and targets.

tcvision-mainframe-to-confluent-cloud-data-replication-1536x1042

tcVISION focuses on CDC (change data capture) when transferring information between mainframe data sources and cloud and open systems databases and applications. Through innovative technology, changes occurring in any mainframe application data are tracked and captured, and then published to a variety of cloud and open systems targets.

tcVISION stores metadata in a relational database and the tcVISION manager components are administered by the tcVISION control board, a Windows GUI interface, which can be installed on premises or in the cloud. This allows tcVISION users to create metadata, create and control replication scripts, and control database interactions. tcVISION’s architecture is designed to minimize mainframe resource utilization.

Using the tcVISION control board, the most complex transformations can be specified, and it facilitates the mapping of the mainframe copybooks, redefines, data dictionaries, data catalogs, codepages, data type mapping, and more via the user-friendly interface. The repository editor allows users to control data transformations.

What is Confluent?

Confluent Cloud is a real-time data in motion platform that can be deployed in any public cloud, in any region of your choice. It comes with an SLA and uptime of 99.95%, and fully managed components like ZooKeeper, Kafka brokers, 120+ Kafka connectors, Schema Registry, and ksqlDB so you can leverage it on any cloud without having to worry about how it runs and scales.

Kafka Connect, Connect API, connectors, and tcVISION IBM Db2 connector

Kafka comes with three core APIs:

  • Kafka producer/Consumer API
  • Connect API
  • KStreams API

Kafka Connect is a tool for scalably and reliably streaming data between Kafka and other data systems. It makes it simple to quickly define connectors that move large data sets into and out of Kafka. Kafka Connect can ingest entire databases or collect metrics from all your application servers into Kafka topics, making the data available for stream processing with low latency. Kafka Connect connects APIs under the hood with fully managed connector support in Confluent Cloud.

Step-by-step guide on how to use tcVISION and Confluent

This example discusses the integration of tcVISION replication of data from Db2 to Confluent Cloud.

Set up tcVISION access to Confluent

Create an account with Confluent to make a Confluent user ID/password; the user ID is generally your email address. To sign on to Confluent, go to the Confluent Cloud login and enter your user ID:

Confluent Cloud welcome page

Then, enter your password:

Enter your password

When you log in, you’ll be in a Confluent environment called “default”:

Confluent environment called “default”

A Confluent environment is a type of container that holds clusters which in turn hold topics. If you are familiar with messaging systems, Confluent/Kafka will seem familiar. A cluster will need to be created to serve as a target for the data produced by tcVISION. The first attribute to be selected is the type of cluster. Confluent offers three types: Basic, Standard, and Dedicated. For the purposes of this demonstration, Basic will be used. A Basic cluster does not incur charges for simply existing, but does for data transmission and data storage.

Select "Basic cluster" and begin configuration

Select Begin configuration.

Select a cloud provider

Here, a cloud provider can be chosen—AWS, Google Cloud, or Microsoft Azure. For this example, AWS is used. Select Continue and the characteristics of the new cluster are displayed, which we’ve named “tcVISION_cluster_0”:

Cluster characteristics

After entering your payment information (not shown), you can click on the cluster name to launch the cluster overview.

Cluster overview

In order to use Confluent with tcVISION, the user must provide tcVISION with information about the cluster they intend to use. Specifically, the user must supply the hostname and port of the Confluent AWS virtual machine, and the credentials needed to access the cluster.

Confluent refers to the hostname and port as a bootstrap server. There can be multiple bootstrap servers for the purpose of load balancing, but a single server is used for this demonstration.

To find bootstrap server information, click Cluster Settings on the left-hand side:

Cluster settings

The bootstrap server will be listed under “Identification,” and includes both the AWS hostname and the port.

Credentials in Confluent consist of an API Key and an API Secret. These are generated for the cluster and take the place of the Confluent user ID and password used to log in. To generate a key/secret pair, click API Access on the left:

API Keys page

Followed by Create Key:

Select API Key scope

For this example, we use “Global Access” here, so click Next:

API Key and secret

Pay particular attention to the tip about saving the key and secret somewhere safe, because once this panel is exited, there is no way to display the secret again. A descriptive string for this key/secret pair can be filled in. The key or secret text to be copied can be selected, or use the convenient icons at the end of the field to copy. Once the key/secret has been safely stored, check the box that says it has been done, and click Save. You will return to the “API Keys” panel, and the key is now displayed:

API Key displayed

Set up Confluent and define the topic

The last thing to do is define a topic within the cluster. Confluent producers have the capability to define their own topics within a cluster, but this capability can be disabled by a Confluent configuration and is disabled in the configuration used here.

Go back to the cluster Overview:

Cluster Overview

On the left sidebar, click Topics:

Topics

Then Create Topic:

Create a topic

The topic name is filled in (“CONFLUENT_CLOUD_TOPIC1”), overriding the number of partitions from 6 to 1, since that is what the Confluent demo uses. Click Create with defaults:

Cloud topic

A topic is now available, which can be populated with Db2 data.

Set up tcVISION and run a bulk load of Db2 data

tcVISION’s control board is a Windows graphical user interface (GUI) that allows users to configure the replication stream between various database platforms, including the IBM mainframe and Confluent. Using the control board and built-in wizards, users can define the metadata and the mappings between the mainframe and target.

The following sequence of screens shows the steps required to create the tcVISION metadata and scripts for replicating mainframe Db2 z/OS data to Confluent.

Access the tcVISION control board:

tcVISION control board

Log on to Db2 z/OS:

Db2 z/OS

Create metadata that is specific to the input (Db2) and output (Kafka) and the replication definition. In this example, the Db2 table is mapped to the Confluent Cloud Kafka topic using JSON:

Import of structure definitions

The tcVISION metadata wizard asks for the information required for the replication of the mainframe database to Confluent Cloud. For Db2 z/OS, it asks for the mainframe Db2 subsystem:

Source type for structure definition import

Db2 subsystem

tcVISION presents the tables contained in the Db2 z/OS catalog on the mainframe. Select the schemas and associated tables for replication:

Select the schemas and associated tables for replication

Once the required tcVISION wizard-based screens are completed, the tool automatically defines the mappings between the source and target. tcVISION’s metadata import wizard creates a default mapping that handles data type conversion issues, such as EBCDIC to ASCII, Endianness conversion, codepages, redefines data types, and more:

Default mapping

tcVISION data scripts are created through wizards. Data scripts control the replication of data from the source (Db2 z/OS) to the target (Confluent Cloud Kafka JSON). tcVISION bulk load scripts are a type of data script that performs the initial load of the Kafka topic. The following script shows data being accessed directly from the mainframe Db2 z/OS database. Another alternative to reduce MIPS consumption is to read the data from a Db2 image copy.

Data script

Bulk load script running:

Bulk load script running

After execution of the bulk load script, replication statistics of the Db2 bulk load into the Confluent Cloud Kafka topic can be viewed:

Replication statistics of the Db2 bulk load

Now that the topic has been loaded with data from Db2, it can be displayed in Confluent. To do this, navigate to the topics panel again:

Notice that there are now statistics indicating that the tcVISION producer uploaded some data to the topic. On the horizontal menu, switch from “Overview” to “Messages” to display the messages (data records) that the tcVISION bulk load placed in the topic. The display can be filtered in various ways, but for this example, the default is used: “Jump to Offset,” which says “start displaying sequentially from this offset.” Here, an offset of 0 (start at the beginning) is specified, since we just want to verify that the Db2 data uploaded by tcVISION was actually delivered:

Messages (data records) from tcVISION bulk load

Run a change script in tcVISION to show the changes in Confluent

To capture ongoing changes to Db2 in real time, a Db2 z/OS CDC replication script is created.

This script captures the changes on the Db2 z/OS side and applies them into the repository where the output target is Confluent Cloud topic.

Replication script

Replication script

Target database Confluent Cloud topic

The CDC replication is initiated from the tcVISION control board. The tcVISION control board shows a graphical representation of the replication:

Graphical representation of the replication

The CDC replication is now actively capturing and replicating data changes whenever they occur on the Db2 z/OS side. You can test it by making a change in the Db2 z/OS table:

 
********************************* Top of Data **********************************
---------+---------+---------+---------+---------+---------+---------+---------+
UPDATE SXE1.TVKFKATB                                                    00010004
SET DEPT = '696969'                                                     00040029
WHERE PERS_ID = 5;                                                      00050004
---------+---------+---------+---------+---------+---------+---------+---------+
DSNE615I NUMBER OF ROWS AFFECTED IS 1                                           
DSNE616I STATEMENT EXECUTION WAS SUCCESSFUL, SQLCODE IS 0                       
---------+---------+---------+---------+---------+---------+---------+---------+
--COMMIT;                                                               00060019
---------+---------+---------+---------+---------+---------+---------+---------+
DSNE617I COMMIT PERFORMED, SQLCODE IS 0                                         
DSNE616I STATEMENT EXECUTION WAS SUCCESSFUL, SQLCODE IS 0                       
---------+---------+---------+---------+---------+---------+---------+---------+
DSNE601I SQL STATEMENTS ASSUMED TO BE BETWEEN COLUMNS 1 AND 72                  
DSNE620I NUMBER OF SQL STATEMENTS PROCESSED IS 1                                
DSNE621I NUMBER OF INPUT RECORDS READ IS 4                                      
DSNE622I NUMBER OF OUTPUT RECORDS WRITTEN IS 17                                 
******************************** Bottom of Data ********************************

This change is processed and replicated by tcVISION. The tcVISION control board shows the statistics highlighting that one update was performed:

Display of extended statistics

Checking in Confluent, the Db2 z/OS change has successfully been propagated to the Confluent Cloud topic:

Db2 z/OS change successfully propagated to Confluent Cloud topic

tcVISION and Confluent are better together

With tcVISION’s groundbreaking Db2 CDC connector and Confluent’s ability to serve as the multi-tenant data hub, this combination creates a very powerful solution to aggregate data from multiple sources and have data published into various Kafka topics. Sourcing events from any kind of Db2 via a connector into Confluent will set data in motion for the entire organization. Simplicity and agility are key elements of the tcVISION and Confluent “better together” story.


__001_TSI_LOGO

Video: tcVISION Demonstration…

In this video, we show a tcVISION overview, then a demonstration of replication of mainframe data on AWS RDS for PostgreSQL:

Contact Treehouse Software for a tcVISION Demo Today!

No matter where you want your mainframe data to go – the Cloud, open systems, or any LUW target – tcVISION from Treehouse Software is your answer.

Just fill out the Treehouse Software tcVISION Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.

Treehouse Software Customer Success: ETS uses tcVISION for Real-Time Synchronization Between their Mainframe IDMS Data and AWS RDS for PostgreSQL

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

ETS_Graphic

This blog focuses on a current Treehouse Software customer – ETS. Headquartered in Princeton, New Jersey, ETS is a private, nonprofit organization with approximately 3,000 employees devoted to educational measurement and research. ETS develops and administer a broad range of educational products and services for government agencies, academic institutions and corporations, including the TOEFL® and TOEIC® tests, the GRE® General and Subject Tests, and the Praxis® assessments. At nonprofit ETS, our belief in the life-changing power of learning is at the root of everything we do — it’s behind the tools we develop to move learning forward, the research that inspires educational progress and the commitment we make to enable opportunity for learners everywhere. We’re with you on the journey to what’s possible.

Business Background

ETS products and services are available to institutions, businesses, organizations and governments in more than 180 countries around the world. The top industries served by ETS are K–12 Education, Higher Education, English-language Learning, Career Development, and Consulting Services.

Business Issue

Most of ETS’s high volume critical application data is stored on an IBM mainframe in IDMS databases.  The technology is very old, therefore it is difficult to recruit and retain qualified technical personnel to maintain applications.  ETS is moving to Cloud-based computing which will allow them to retire the mainframe environments and modernize the applications.  The data is used and shared across several applications.  ETS required a solution that would allow them to continue, uninterrupted, daily operations on their mainframe while replicating data to their AWS Cloud platform, where they could develop modern application features.  This solution enables ETS to maintain demanding daily processing while they modernize and develop innovative Cloud solutions to meet and exceed customer requirements.

The Technology Solution

ETS_Diagram

Treehouse Software and the ETS team developed a rigid testing plan to implement tcVISION and performed a Proof of Concept to measure the effectiveness of the data replication, considering the high volumes of data changes on the source databases.  We collaborated on architecture requirements and installation steps.  There were many considerations associated with this process, including monitoring, alarming, configuration options, high availability, measuring the impact to existing mainframe database performance, restart capability, and security.  Concurrently, a team of subject matter experts worked on data mappings and translation of database designs from the IDMS network databases to AWS PostgreSQL relational databases.  The goal was to be able to replicate two very large IBM mainframe IDMS databases real-time on two Cloud-based PostgreSQL databases. Implementation was done in phases, starting with one non-production database being replicated to the Cloud.  High-volume testing was performed on the source database to simulate peak processing, replicating millions of transactions to the target PostgreSQL databases.  Many technical challenges were encountered and resolved with outstanding technical assistance from the Treehouse Software support team.  Once in production, the tcVISION product was able to deliver real-time data to the Cloud platform with no interruptions to the customer’s daily processing. The customer was then able to develop modern application features and functions in the Cloud to achieve independence from the legacy mainframe systems.  Using new Cloud-based capabilities enabled the customer to be more agile with meeting new requirements.


__001_TSI_LOGO

Video: tcVISION Demonstration…

In this video, we show a tcVISION overview, then a demonstration of replication of mainframe data on AWS RDS for PostgreSQL:

Contact Treehouse Software for a tcVISION Demo Today!

No matter where you want your mainframe data to go – the Cloud, open systems, or any LUW target – tcVISION from Treehouse Software is your answer.

Just fill out the Treehouse Software tcVISION Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.

Treehouse Software Customer Case Study: A Large Airline’s Real-time Data Synchronization Between IBM Mainframe Adabas and Oracle RDBMS

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

_0_Airline_Manitenance

This blog focuses on a current Treehouse Software customer – a major airline that is a long-time user of Software AG’s Adabas database on their mainframe system.

Business Background

This U.S. based airline is one of the largest domestic air carriers, and during peak travel seasons, they operate thousands of weekday departures within a network of hundreds of destinations in the United States and several other countries.

Business Issue

The airline’s IT modernization team was looking for a technology solution that enabled them to move their Adabas data off of the mainframe to more modern applications. However, their mainframe contains vital airline maintenance and parts data that must always be highly available, so rather than performing a complete migration from the mainframe, Treehouse technical representatives and the customer decided to utilize a data warehouse and real-time mainframe data replication. This architecture allows the mainframe legacy teams to maintain existing critical applications, while the modernization team develops applications on the newly created Oracle-based RDBMS. All teams at the airline determined that tcVISION real-time mainframe data replication was the perfect fit for meeting their goals.

The airline requires that tcVISION must support online and batch Adabas transactions and provide data replication between Adabas and their new Oracle RDBMS. Additionally, their large databases contain millions of rows (> 50 million) that must be supported, as well as support for database change audit requirements (datetime and type of operation), transaction management, and notification of exception events. There must also be support for configuration management between development, QA, and production.

The tcVISION Technology Solution...

The following is a high-level view of the airline customer’s tcVISION data replication architecture:

___Airline_tcV_Overview

  • Adabas: Mainframe data source containing business critical information replicated to the RDBMS.
  • Oracle: Open Platform RDBMS chosen by customer as replication target for both data warehouse and modernization project. The tcVISION Manager also uses Oracle as a repository for the metadata (field mappings).
  • tcVISION Manager: Software component deployed on both source and target systems. It is responsible for provisioning resources for:
    • Processing scripts
    • Metadata import
    • Scheduling
  • tcSCRIPT: Software component deployed on both source and target systems. It works in conjunction with the tcVISION Manager to:
    • Perform initial load of Adabas into Oracle
    • Ongoing near realtime CDC (Change Data Capture) and replication from Adabas to Oracle
    • tcSCRIPT processes data from:
      • Direct from the DBMS (initial load) or from DBMS extracts
      • Adabas Protection log (PLOG)
  • tcVISION Control Board: Software component deployed on a Windows machine which provides graphical user interface for a single point of control to administer the tcVISION environment:
    • Metadata import, creates metadata rules governing relationship between mainframe and open platform data structures
    • Replication rule maintenance
    • DDL creation
    • Creation of ETL and replication processes
    • Start, stop, and schedule replication processes

Customer Outcomes

All requirements were met by tcVISION, which led to a successful project implementation.  Here is a look at the customer’s reported outcomes and benefits:

Business Outcome Customer Benefit
Data warehouse is always in sync with Adabas using tcVISION data replication Improved timeliness and reliability of reporting data
Reduced usage of mainframe MIPS Reduced cost
All data required for modernization project was successfully replicated to target environment Increased business efficiently and agility

__TSI_LOGO

Contact Treehouse Software for a Demo Today…

No matter where you want your mainframe data to go – the cloud, open systems, or any LUW target – tcVISION from Treehouse Software is your answer.

_0_Treehouse_tcV_Cloud_OpenSystems

Just fill out the Treehouse Software Product Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.


Further reading:

Many more mainframe data migration and replication customer case studies can be read on the Treehouse Software Website.

Why is High Availability so Important for Mainframe Data Modernization on the Cloud?

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

Many customers embarking on Mainframe-to-Cloud data replication projects with Treehouse Software are looking at high availability (HA) as a key consideration in the planning process. All of the major Cloud platforms have robust HA infrastructures that keep businesses running without downtime or human intervention when a zone or instance becomes unavailable. HA basic principles are essentially the same across all Cloud platforms.

In this blog, our example shows how the AWS Global Infrastructure and HA is architected with Treehouse Software’s tcVISION real-time mainframe data replication product. A well planned HA architecture ensures that systems are always functioning and accessible, with deployments located in various Availability Zones (AZs) worldwide.

The following example describes tcVISION‘s HA Architecture on AWS. During tcVISION ’s Change Data Capture (CDC) processing for mainframe data replication on the Cloud, HA must be maintained. The Amazon Elastic Compute Cloud (Amazon EC2), which contains the tcVISION Agent, is part of an Auto Scaling Group that is spread across AZs with Amazon EC2 instance(s).

tcVISION and AWS overall architecture…

___tcVISION_AWS_Overall_Architecture

Upon failure, the replacement Amazon EC2 instance tcVISION Agent is launched and communicates its IP address to the mainframe tcVISION Agent. The mainframe tcVISION Agent then starts communication with the replacement Amazon EC2 tcVISION Agent.

Once the Amazon EC2 tcVISION Agent is restarted, it continues processing at its next logical restart point, using a combination of the LUW and Restart files. LUW files contain committed data transactions not yet applied to the target database. Restart files contain a pointer to the last captured and committed transaction and queued uncommitted CDC data. Both file types are stored on a highly available data store, such as Amazon Elastic File System (EFS).

tcVISION and AWS HA architecture…

___tcVISION_AWS_HA_Architecture

For production workloads, Treehouse Software recommends turning on Multi-AZ target and metadata databases.

To keep all the dynamic data in an HA architecture, tcVISION uses EFS, which provides a simple, scalable, fully managed elastic file system for use with AWS Cloud services and on-premises resources. It is built to scale on-demand to petabytes without disrupting applications, growing and shrinking automatically as you add and remove files, eliminating the need to provision and manage capacity to accommodate growth.

More information on AWS HA


Treehouse Software helps enterprises immediately start synchronizing their mainframe data on the Cloud, Hybrid Cloud, and Open Systems to take advantage of the most advanced, scalable, secure, and highly available technologies in the world with tcVISION

tcVISION supports a vast array of integration scenarios throughout the enterprise, providing easy and fast data replication for mainframe application modernization projects and enabling bi-directional data replication between mainframe, Cloud, Open Systems, Linux, Unix, and Windows platforms.

tcV_Arch01


__TSI_LOGO

Contact Treehouse Software for a Demo Today…

Just fill out the tcVISION Product Demonstration Request Form and a Treehouse representative will contact you to set up a time for your tcVISION demonstration. This will be a live, on-line demonstration that shows tcVISION replicating data from the mainframe to a Cloud target database.

Starting a Mainframe Data Replication Project? Consider Your Use Cases Carefully

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software

___Data_Center_To_Targets_Overview

Planning for Real-time and Bi-directional Mainframe Data Replication with tcVISION

A customer’s mainframe data may be utilized by many interlinked and dependent programs that have been in place for several years, and sometimes, decades, so unlocking the value of this legacy data can be difficult. Therefore, careful planning must occur for a mainframe data modernization project, beginning with identifying uses cases for the project and a Proof of Concept (POC) of data replication software, such as tcVISION, the Mainframe-to-Hybrid Cloud and Open Systems data replication product from Treehouse Software.

This blog serves as a general guide for organizations planning to replicate their mainframe data on Cloud and/or Open Systems platforms using tcVISION.

Questions and Considerations for Your Use Cases

A general principle should be to prove out the data replication technology (tcVISION) by using identified use cases. Listed below are some examples of questions and use cases to assist customers in planning and experiencing successful Mainframe-to-Cloud and/or Open Systems data replication projects:

  • What is/are the mainframe source database(s)? Obvious, yes, but the software solution vendor and outside consultants need this information.
  • What are the critical issues you need to test? Are there any areas that you believe that would be challenging for the vendor? Examples would be specific transformations, data types (e.g., BLOB), required CDC SLAs, field/column changes, specific security requirements, data volume requirements, etc. Document all of the critical test items.
  • Select the minimal set of files (generally 3-10) with representative conditions that will enable you to test all of your critical items. If there are multiple source databases, ensure specific test use cases are defined for each source.
  • What are the target databases? Will you be replicating to a Cloud database manager, such as AWS RDS? Are there additional requirements to replicate to S3, Azure BLOB, GCP Cloud Storage, Kinesis, and Kafka? What needs to be tested?
  • What are your bulk or initial load requirements? tcVISION can load data directly from mainframe databases, mainframe unloads, or image copies. What are the data volumes? Do you have sufficient bandwidth between your mainframe manager and on-premises or Cloud VM to handle your volume requirements?
  • What are your Change Data Capture (CDC) requirements?
  • Do you have plans for bi-directional replication in the future. What are your specific requirements? Since bi-directional replication can be complex and greatly lengthen a modernization project, the customer generally will perform one bi-directional use case for conceptual proof. Will this suffice for your organization?
  • What are your specific high availability requirements? Can they be handled by a technical discussion, or is a specific use case required?
  • What are your general security requirements for data at rest and data in transit? Do you have any specific security regulations to follow, such as HIPPA or FIPS? What are your PII / data masking requirements?
  • What are your schema requirements? For example, tcVISION creates a default schema based on your input mainframe data. Major changes to the default schema usually require a staging database.
  • Do you have staff available to perform the required tasks for the project? For example, for the length of a tcVISION POC you will need part-time staff, 2-4 hours per day. A part-time mainframe administrator will generally require 2-8 elapsed hours. Other staff will include Windows/Linux/Cloud administrators. 2-4 hours of project management may also be required.
  • Are business data transformations required? tcVISION handles minor transformation via point and click (e.g., date format transformations). Major transformations can require C++ or product scripting.
  • Are there any triggers or stored procedures? tcVISION performs CDC replication processing using a database that utilizes these database features.

Of course, each project will have unique environments, goals, and desired use cases. It is important that specific use cases are determined and documented prior to the start of a project and a tcVISION POC. This planning will allow the Treehouse Software team and the customer develop a more accurate project timeline, have the required resources available, and realize a successful project. 

More About tcVISION from Treehouse Software…

__Plans_To_Reality

tcVISION supports a vast array of integration scenarios throughout the enterprise, providing easy and fast data migration for mainframe application modernization projects. This innovative technology offers comprehensive abilities to identify and capture changes occurring in mainframe and relational databases, then publish the required information to an impressive variety of targets, both Cloud and on-premises.

tcVISION acquires data in bulk or via CDC methods from virtually any IBM mainframe data source (Software AG Adabas, IBM Db2, IBM VSAM, IBM IMS/DB, CA IDMS, CA Datacom, and sequential files), and transform and deliver to a wide array of Cloud and Open Systems targets, including AWS, Google Cloud, Microsoft Azure, Confluent, Kafka, PostgreSQL, MongoDB, etc. In addition, tcVISION can extract and replicate data from a variety of non-mainframe sources, including Adabas LUW, Oracle Database, Microsoft SQL Server, IBM Db2 LUW and Db2 BLU, IBM Informix, and PostgreSQL.


__TSI_LOGO

Contact Treehouse Software for a tcVISION Demo Today…

Simply fill out our Product Demonstration Request Form and a Treehouse representative will be contacting you to set up a time for your requested demonstration.

What’s Your Mix?

Real-time and Bi-directional Data Replication Between Mainframes and Virtually Any Target

Cloud004_Swirl

tcVISION from Treehouse Software — Enterprise ETL and Real-Time Data Replication Through Change Data Capture (CDC)

Planning a data replication project between Mainframe, Cloud, and Open Systems platforms? tcVISION supports a vast array of integration scenarios throughout the enterprise, providing easy and fast data migration for mainframe application modernization projects. This innovative technology offers comprehensive abilities to identify and capture changes occurring in mainframe and relational databases, then publish the required information to an impressive variety of targets, both Cloud and on-premises.

tcVISION acquires data in bulk or via CDC methods from virtually any IBM mainframe data source (Software AG Adabas, IBM Db2, IBM VSAM, IBM IMS/DB, CA IDMS, CA Datacom, and sequential files), and transform and deliver to a wide array of Cloud and Open Systems targets, including AWS, Google Cloud, Microsoft Azure, Confluent, Kafka, PostgreSQL, MongoDB, etc. In addition, tcVISION can extract and replicate data from a variety of non-mainframe sources, including Adabas LUW, Oracle Database, Microsoft SQL Server, IBM Db2 LUW and Db2 BLU, IBM Informix and PostgreSQL.

Whatever your mix, Treehouse Software has got it covered with tcVISION.


__TSI_LOGO

Moving the right data to the right place at the right time

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.

Quickly Begin Replicating Mainframe Data on Cloud and Open Systems During a tcVISION Proof of Concept.

by Joseph Brady, Director of Business Development / Cloud Alliance Leader at Treehouse Software, Inc.

___Start_Moving_Now_Graphic

Customers can start moving mainframe data within days during a tcVISION POC…

An online tcVISION Proof of Concept (POC) is approximately 10 business days, with the customer providing a representative subset of data, use cases, and goals for the POC. A Treehouse Software consultant will assist in downloading and installing tcVISION, and conduct a limited-scope implementation of a tcVISION application. This application uses customer data and executes on customer facilities, in a non-production environment. A document is provided beforehand that outlines the requirements and agenda for the POC.

By the end of the POC, customers can begin replicating mainframe data to their Cloud or Open Systems target database.  It can happen that fast!.

About tcVISION

More Cloud, Open Systems, and Systems Integration partners are recommending tcVISION, Treehouse Software’s Mainframe-to-Cloud data replication product for modernization projects. tcVISION focuses on changed data capture (CDC) when transferring information between mainframe data sources and Cloud and Open System databases and applications. Through an innovative technology, changes occurring in any mainframe application data are tracked and captured, and then published to a variety of RDBMS and other targets.

tcVISION_Overall_Diagram_Cloud_OS

Further reading…

Treehouse Software is an AWS, Google Cloud, and Microsoft Technology Partner, and the AWS Partner Network published a blog about tcVISION, which describes how tcVISION allows legacy mainframe environments to continue, while replicating data on highly available and secure Cloud platforms.


__TSI_LOGO

Contact Treehouse Software for a tcVISION Demo Today…

Simply fill out our Product Demonstration Request Form and a Treehouse representative will be contacting you to set up a time for your requested demonstration.

Treehouse Software is Helping Government Agencies with Mainframe Adabas Data Take Advantage of Cutting Edge Cloud-based Technologies

by Joseph Brady, Director of Business Development / Cloud Alliance Lead at Treehouse Software, Inc.

___Gov_IT_Cloud

Many government agencies have large volumes of mission critical and historical data stored in a variety of legacy mainframe databases (for this blog, we are focusing on Adabas). These government agencies combine a broad range human service programs, including employment assistance and job training, child and adult protection, child support enforcement, cash assistance, and services for the developmentally disabled, all of which constantly require accurate, up-to-date, and secure data.

To support the needs of clients, government agencies are generally broken into multiple divisions, including the division that Treehouse Software works with the most — Technology Services. The Technology Services divisions provide technical and systems services for the development, maintenance, and enhancement of automated business systems. They also ensure that the production and sub-production databases are running smoothly and efficiently by performing necessary maintenance of the databases.

Rapidly changing national health and economic conditions are making fast access to the most current information more important than ever for government agencies and the public.  As a result, a top priority for Technology Services divisions is modernizing critical data residing on long-standing mainframe databases. Unlocking the value of this important data can be difficult, because the data can be utilized by numerous interlinked and dependent programs that have been in place for many years, and sometimes decades.

Many Treehouse government customers are now looking for modernization solutions that allow their legacy mainframe environments to continue, while replicating data in real time on highly available Cloud-based platforms, such as AWS, Google Cloud, and Microsoft Azure. tcVISION from Treehouse Software allows a “data-first” approach, whereby immediate data replication to the Cloud helps government agencies begin strategies to meet spikes in demand for vital information, especially in times of crisis.

Just what is it about Adabas?

Adabas is a mainframe database that is still heavily used by government sites throughout the U.S. and the world. Having specialized in tools and services complementary to Adabas/Natural applications since 1982, Treehouse has successfully encountered and addressed many unique issues within the Adabas environment. This excerpt from a Treehouse technical document outlines three primary issues with Adabas/Natural that must be considered:

  1. Adabas has no concept of “transaction isolation”, in that a program may read a record that another program has updated, in its updated state, even though the update has not been committed.  This means that programmatically reading a live Adabas database—one that is available to update users—will almost inevitably lead to erroneous extraction of data.  Record modifications (updates, inserts and deletes) that are extracted, and subsequently backed out, will be represented incorrectly—or not at all—in the target. Because of this, at Treehouse we say “the only safe data source is a static data source”—not the live database.
  2. Many legacy Adabas applications make use of “record typing”, i.e., multiple logical tables stored in a single Adabas file.  Often, each must be extracted to a separate table in the target RDBMS.  The classic example is that of the “code-lookup file”.  Most shops have a single file containing state codes, employee codes, product-type codes, etc.  Records belonging to a given “code table” may be distinguished by the presence of a value in a particular index (descriptor or superdescriptor in ADABAS parlance), or by a range of specific values.  Thus, the extraction process must be able to dynamically assign data content from a given record to different target tables depending on the data content itself.
  3. Adabas is most often used in conjunction with Software AG’s Natural 4GL, and “conveniently” provides for unique datatypes (“D” and “T”) that appear to be merely packed-decimal integers on the surface, but that represent date or date-time values when interpreted using Software AG’s proprietary Natural-oriented algorithm. The most appropriate way to migrate such datatypes is to recognize them and map them to the corresponding native RDBMS datatype (e.g., Oracle DATE) in conjunction with a transformation that decodes the Natural value and formats it to match the target datatype.

For discovery and analysis of legacy source data structures, tcVISION’s modeling and mapping facilities view and capture logical Adabas structures, as documented in Software AG’s PREDICT data dictionary, as well as physical structures as described in Adabas Field Definition Tables (FDTs).  Note that PREDICT is a “passive” data dictionary—there is no requirement that the logical and physical representations agree, so it is necessary to scrutinize both to ensure that the source structures are accurately modeled.

Furthermore, tcVISION generates specification and implementation of appropriate mappings and transformations for converting Adabas datatypes and structures to corresponding RDBMS datatypes and structures, including automatic handling of the proprietary “D” and “T” source datatypes.

There are three ways tcVISION can access Adabas data:

  1. ETL – read the active database nucleus
  2. ETL – read datasets containing unloaded Adabas files created by the ADAULD utility
  3. CDC – read the active and archived PLOGs datasets

It is important to note that the schema, mappings and transformations that result from metadata import can be tailored to any specific requirements after the fact.  It is even possible to import an existing RDBMS schema and retrofit it, via drag-and-drop, to the source Adabas elements.

Using the tcVISION Control Board, a Windows GUI interface, the most complex transformations can be specified. Source fields can be combined into a single column, decomposed into separate columns, and be subject to calculations, database lookups, string operations and even programmatic manipulation. Furthermore, mapping rules can be implemented to specify that data content from a source Adabas record be mapped to one or more target RDBMS tables—each with its own different structure, as desired—based on the data content itself. Target tables can even be populated from more than one source file.

tcVISION supports complex replication scenarios…

tcVISION_Complex_Replication_Scenarios

tcVISION GUI Control Board functions as a central point of administration…

tcVISION_Control_Board

Automatically apply target schema within the Control Board…

tcVISION_Target_Schema

It is impossible to discuss all the features and capabilities of tcVISION within a high-level overview.  Given the maturity, wealth of functionality and relative low cost of tcVISION, as compared to the effort, complexity and risk entailed in a “Do-It-Yourself”, solution there is no reason why a legacy renewal project should run aground on data migration.

tcVISION’s minimal footprint on the mainframe…

Customers are very happy with tcVISION‘s “staged processing” methodology, where the only processing occurring on the mainframe was the capture of changes from Adabas PLOGs. The bulk of the processing occurs on the target platform, minimizing tcVISION’s footprint on the mainframe as seen in this diagram…

tcVISION_Staged_Processing

The user defines on which platform stage their processing should be done. Do as little as possible on the mainframe: Stage 0 – capture data and send data (internal format) to target, and process data up to Stage 3 in receiving environment.

Moving forward with tcVISION…

Treehouse Software has been helping local, state, and federal government agencies with Adabas and Natural in the areas of data migration, security, control, auditing, performance enhancement, etc. for decades. Over the years, Treehouse has expanded its capabilities to address new requirements for modernizing legacy mainframe Adabas databases on various Cloud platforms. By using Treehouse Software’s tcVISION Mainframe-to-Cloud data replication product, our government customers are able to immediately utilize some of the most advanced Cloud tools and services in the world.

tcVISION enables government agencies to synchronize mainframe Adabas data with various highly available and secure Cloud databases, data warehouses. etc.. Additionally, bi-directional, real-time data synchronization will enable changes on either platform to be reflected on the other platform (e.g., a change to a PostgreSQL table is reflected back on the mainframe database). This allows governments to modernize  applications on Cloud platforms without disrupting the existing critical work on the legacy system, and modern tools can now be used in the new environment, greatly enhancing agility.

Replicating mainframe data on the Cloud can happen within days during a tcVISION Proof of Concept (POC)…

tcVISION_Overall_Diagram_General_Cloud01

An online tcVISION POC is approximately 10 business days, with the customer providing use case and goals for the POC. A Treehouse Software consultant will assist in downloading and installing tcVISION and conducting a limited-scope implementation of a tcVISION application. This application uses customer data and executes on customer facilities, usually in a non-production environment. A document is provided beforehand that outlines the requirements, use cases, and agenda for the POC.

By the end of the 10-day POC, customers can begin replicating mainframe data to their Cloud target database.  It can happen that fast!

Further Reading…

_0_AWS_Logo

Treehouse Software is an AWS Technology Partner, and the AWS Partner Network published a blog about tcVISION, our Mainframe-to-Cloud data replication product, which describes how tcVISION allows legacy mainframe environments to continue, while replicating data on highly available and secure Cloud platforms:

https://aws.amazon.com/blogs/apn/real-time-mainframe-data-replication-to-aws-with-tcvision-from-treehouse-software/


__TSI_LOGO

Contact Treehouse Software for a tcVISION Demo Today…

Simply fill out our Product Demonstration Request Form and a Treehouse representative will be contacting you to set up a time for your requested demonstration.

Treehouse Software is Helping Higher Education Customers Modernize Long-standing Mainframe Data on the Cloud

by Joseph Brady, Director of Business Development / Cloud Alliance Lead at Treehouse Software, Inc.

___Higher_Ed_Data

The Business Issue

Many higher education institutions have large volumes of mission critical and historical data stored in legacy mainframe databases (Db2, Adabas, IMS, IDMS, Datacom, VSAM, etc.). The cost to maintain these databases is high and they lack the features required for modernizing the data architecture. Additionally, the data is utilized by an extensive number of interlinked programs dependent on this legacy data.

Colleges and universities are searching for a solution that allows them to unlock their mainframe data within a Cloud-based data store, such as Amazon Simple Storage Service (Amazon S3), where they can use a wide array of analytics and machine learning services for easy access to all relevant data, without compromising security or governance.

Once mainframe data is on AWS, an institution can innovate quickly by creating new functions with Cloud speed, such as mobile users via Amazon API Gateway, or voice devices such as Amazon Alexa.

Additionally, data security is one of the biggest challenges facing most higher education organizations. Beyond the certifications and best practices that are part of having data reside on the AWS Cloud platform, there are also many security features and services designed to help an organization stay compliant with industry best practices and regulations.

The Solution: Mainframe-to-Cloud Data Replication 

Treehouse Software recently helped a large university with a requirement for a solution that allows their legacy mainframe database to continue while replicating data in real time on AWS. By using Treehouse Software’s tcVISION Mainframe-to-Cloud data replication product, the university was able to immediately utilize some of the most advanced Cloud tools and services in the world.

___tcVISION_AWS_Overall_Architecture

tcVISION enables the university to synchronize mainframe data to Amazon RDS for PostgreSQL. Furthermore, bi-directional, real-time data synchronization will enable changes on either platform to be reflected on the other platform (e.g., a change to a PostgreSQL table is reflected back on the mainframe database). This allows the university to modernize the application on PostgreSQL without disrupting the existing critical work on the legacy system, and modern tools can now be used in the PostgreSQL environment, greatly enhancing business agility.

Moving Forward…

Having on-demand, Cloud-based services available can now help IT teams build secure environments for mission-critical applications for the University, freeing them to focus on student success and plan for growth or increased seasonal demand.

tcVISION provides the quality of service required by enterprise data workloads for security, availability, and scalability, and university staff and students can look forward to quickly and affordably accessing Cloud compute, storage, and application services.

Replicating mainframe data on the Cloud can happen within days during a tcVISION Proof of Concept (POC)…

tcVISION_Overall_Diagram_General_Cloud

An online tcVISION POC is approximately 5-10 business days, with the customer providing use case and goals for the POC. A Treehouse Software consultant will assist in downloading and installing tcVISION and conducting a limited-scope implementation of a tcVISION application. This application uses customer data and executes on customer facilities, usually in a non-production environment. A document is provided beforehand that outlines the requirements and agenda for the POC.

By the end of the 10-day POC, customers can begin replicating mainframe data to their Cloud target database.  It can happen that fast!

Further Reading…

_0_AWS_Logo

Treehouse Software is an AWS technology partner, and the AWS Partner Network published a blog about tcVISION, our Mainframe-to-Cloud data replication product, which describes how tcVISION allows legacy mainframe environments to continue, while replicating data on highly available and secure Cloud platforms:

https://aws.amazon.com/blogs/apn/real-time-mainframe-data-replication-to-aws-with-tcvision-from-treehouse-software/


__TSI_LOGO

Contact Treehouse Software for a tcVISION Demo Today…

Simply fill out our Product Demonstration Request Form and a Treehouse representative will be contacting you to set up a time for your requested demonstration.

Customers are discovering that they can quickly begin replicating their mainframe data on the Cloud during tcVISION Proof of Concepts (POCs)

by Joseph Brady, Director of Business Development / Cloud Alliance Lead at Treehouse Software, Inc.

___Data_Flow_Cloud

Modernizing long-standing mainframe systems has become a strategic imperative at many government, education, healthcare, financial, and retail organizations. As a result, these organizations are looking for solutions that allow their legacy environments to continue, while replicating data, in real time, on Cloud-based platforms, such as AWS, Microsoft Azure, Google Cloud, etc. This “data first” approach allows organizations to quickly take advantage of advanced Cloud technologies, such as big data analytics, artificial intelligence (AI), rapid global database deployments, high-level security, etc., while keeping the mainframe and Cloud sides synchronized.

With new IT modernization initiatives in the forefront, Treehouse Software is seeing a significant upswing in requests for online demonstrations and POCs of tcVISION, our Mainframe-to-Cloud data replication product.

You can start moving your mainframe data to the Cloud within days during a tcVISION POC…

tcVISION_Overall_Diagram_General_Cloud

An online tcVISION POC is approximately 5-10 business days, with the customer providing use case and goals for the POC. A Treehouse Software consultant will assist in downloading and installing tcVISION, and conduct a limited-scope implementation of a tcVISION application. This application uses customer data and executes on customer facilities, usually in a non-production environment. A document is provided beforehand that outlines the requirements and agenda for the POC.

By the end of the 10-day POC, customers can begin replicating mainframe data to their Cloud target database.  It can happen that fast!


__TSI_LOGO

Contact Treehouse Software for a tcVISION Demo Today…

Simply fill out our Product Demonstration Request Form and a Treehouse representative will be contacting you to set up a time for your requested demonstration.