Providing a High Availability Framework for Mainframe-to-AWS Data Replication

by Dan Vimont, Cloud Solutions Architect at Treehouse Software, Inc.

tcV_HA_on_AWS

Treehouse Software customers are using tcVISION to enable mission-critical mainframe-to-AWS data replication pipelines.  Some of these production pipelines are providing vital near-real-time synchronization between source and target, and thus can’t afford any significant downtime in the event of failure.  So it’s only natural that a number of our customers have been asking for advice in setting up a high availability configuration for their tcVISION components that run on AWS EC2 instances.  The High Availability Framework discussed here provides for a Failover EC2 instance to automatically pick up tcVISION processing should the Primary instance (running in another Availability Zone) go down.

The Core Components:  Primary Instance & Failover Instance

The core components of a tcVISION high availability framework consist of two EC2 instances running in different Availability Zones:  a Primary EC2 instance and a Failover EC2 instance.  Both identically-configured EC2 instances are attached to a shared working-storage file system (either an EFS or FSx volume), which allows the Failover instance to seamlessly and quickly pick up tcVISION processing should the Primary instance suddenly become unavailable.

HA1

Use a Step Function to Automate the Failover Process

In the event of failure of the Primary instance, the recommended framework calls for automatic triggering of a Step Function for reliable failover processing, with steps that include the following:

  • verify that the Primary instance is unavailable (The tcVISION service cannot be active on both instances simultaneously, so this verification is vital.)
  • redirect all network traffic from the Primary instance to the Failover instance (via Route 53)
  • start tcVISION processing on the Failover instance

HA2

When Ready, Use a Step Function to Automate the Restoration Process

After operations personnel have completed recovery of the Primary EC2 instance, another Step Function may be manually triggered to reliably transfer tcVISION processing back to the Primary instance.

HA3.jp

Many More Details are Available Upon Request to Treehouse Customers

Full details regarding our recommended High Availability Framework for tcVISION are available upon request to Treehouse customers.  AWS services utilized in the complete recommended framework include Step Functions, Lambda Functions, EventBridge rules, CloudWatch alarms, SNS topics, a Route 53 Private Hosted Zone, and more.  The following diagram is a partial visual inventory of the recommended framework components.

HA5

Interested in seeing a live, online demo of tcVISION?

Just fill out the Treehouse Software tcVISION Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.


__001_TSI_LOGO

Some are calling mainframes “dinosaurs”, but many of us see that as a good comparison!

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

____Cloud_Mainframe_Dinosaur

Since the dinosaur analogy has been used so much to describe mainframe computer systems in recent years, I would like to use this blog to take a look at the parallels of dinosaurs and mainframes as it relates to the current buzz about modernization on the Cloud.

Of course, dinosaurs and mainframes have been around for a long time and are extremely resilient and successful. I especially say “are” in relation to dinosaurs, because many are not extinct at all, and the fossil record shows that several types have adapted to the changing world by evolving into birds. Additionally, during the age of dinosaurs, they branched off into countless varieties during a span of about 165 million years – hardly a failed species. Also, like the dinosaurs, the mainframe has thrived and survived for over six decades and is continuing to adapt – albeit not nearly as long as the reign of the dinosaurs, but an impressive run, nonetheless.

And the mainframe isn’t finished yet! Mainframe systems are still very much in use, running major banking processes, healthcare systems, government IT services, and critical business operations of many Global 2000 companies. As a matter of fact, IBM has been reporting growth year after year, as the IBM Z platform continues to see important innovations, such as with Cloud-native development capabilities, as well as impressive improvements in processing power.

Looking up and moving forward…

____Cloud_Mainframe_Dinosaur04

As with the dinosaurs who did not fear looking to the clouds and taking wing to ensure survival, the new breed of mainframers envision bold and exciting possibilities in Cloud computing. Many see remarkable opportunities for business advantage by modernizing their mainframe environments. This modernization includes replicating mainframe data on Cloud platforms in order to quickly capitalize on the latest Cloud services, such as analytics, auto scaling, machine learning and artificial intelligence (AI), high availability, advanced security, etc., or to move data to a variety of newer Cloud databases, streaming services, container services, and much more. With the proper data replication technology and planning, all of this modernization can occur while keeping the legacy mainframe environment active as long as it is needed!

The IBM Z mainframe isn’t going anywhere, and with visionary and daring leadership, it can continue to evolve and adapt to whatever develops in the Cloud… and beyond.

Ready to move forward, adapt, and evolve? Treehouse Software is here to help!

Treehouse Software is your partner on your journey into future mainframe modernization plans. With our “data first” approach, we can help accelerate digital transformation and successfully leverage Cloud and Hybrid Cloud initiatives on the IBM Z platform, storing sensitive data on a private Cloud or local data center, and simultaneously leveraging leading technologies on a managed public Cloud.

Bidirectional_Data_Replication

Through an innovative changed data capture (CDC) technology, our tcVISION product tracks and captures changes occurring in any mainframe application data, and then publishes them to a variety of Cloud targets. The customer moves only the right data to the right place at the right time – as much, or as little as they want.

The tcVISION data replication solution has a modular design, which enables it to support mass data load from one source to one or more targets, as well as continuous data exchange processes in real-time via CDC. This modular architecture and the provided APIs gives customers unlimited future potential for continued evolution, and use of new and emerging technologies.


__TSI_LOGO

Want to see tcVISION in action?

You can schedule a live, online demonstration that shows tcVISION replicating data from the mainframe to a Cloud target database. Just fill out the Treehouse Software tcVISION Demonstration Request Form and a Treehouse representative will contact you to set up a time for your tcVISION Mainframe-to-Cloud data replication demonstration.

How to Synchronize Data in Real Time Between the Mainframe and AWS with Treehouse Software’s Enterprise CDC Tool

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

Bidirectional_Data_Replication

Many mainframe integration scenarios require continuous near-real-time replication of relational data to keep a copy of the data synched in the Cloud. Change Data Capture (CDC) is used for this near-real-time transactional replication by capturing change log activity to drive changes in the target dataset.

Just what is CDC anyway?

Simply put, and in relation to Mainframe-to-Cloud and open systems data replication, CDC is the use of processes to identify when data has been changed in a source system, so the replicated upstream or downstream (depending on how you look at it) target can be kept in sync with the changes.

In a recent AWS Architecture Blog, readers learn about integration using mainframe data to build Cloud native services with AWS, including transactional replication-based integration via CDC.

____AWS_Mainframe_CDC_Diagram

As mentioned in the blog, AWS Partner CDC Tools are available for connecting data center mainframes to the various data targets, and Treehouse Software’s tcVISION is one of those tools available in the AWS Marketplace.

tcVISION allows changes occurring in any mainframe application data to be tracked and captured, and then published to a variety of target AWS databases and applications. tcVISION provides an easy and fast approach for Hybrid Cloud projects, enabling real-time and bi-directional data replication between the hardware and AWS.

Example of Db2-to-AWS CDC using tcVISION Mainframe Manager:

tcVISION_Db2_To_AWS_CDC

tcVISION supports several CDC methods available, depending on each customer’s use case:

Bulk Transfer

  • Efficient transfer of entire databases
  • Analysis for data consistency (verification)
  • Initial load (ETL) and periodic mass data transfer
  • One-step data transfer

Log Processing

  • Transfer of changed data near-realtime or scheduled time frame
  • Reads both active logs and archived logs

Batch Compare

  • Comparison of data snapshots using checksums
  • Efficient transfer of changed data since last processing
  • Flexible processing options (SORT etc.)
  • Automatic creation of deltas by tcVISION

DBMS Extension

  • Real-time capture of changed data directly from the DBMS
  • Secure data storage even across DBMS restart
  • Flexible propagation methods

Interested in seeing a live, online demo of tcVISION CDC?

Just fill out the Treehouse Software tcVISION Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.


__001_TSI_LOGO

Should You Stay, or Should You Go? You Can Do Both by Incrementally Replicating Your Mainframe Data on the Cloud While Keeping Both Sides Synchronized

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

Stay_And_Go_Data_Replication

Many of Treehouse Software’s enterprise customers are not close to considering the retirement of their mainframe systems, but instead have long-term data replication projects, or want to indefinitely have their legacy systems co-exist with a new Cloud platform. These organizations are looking for solutions that allow their legacy mainframe environment to continue while replicating data – in real time and bi-directionally – to take advantage of the latest Cloud services, such as analytics, auto scaling, machine learning and artificial intelligence (AI), high availability, advanced security, etc., or move data to a variety of newer Cloud databases, streaming services, container services, and more.

The Transition Doesn’t Have to be a Sudden Big Bang

Much of an enterprise’s mission critical mainframe data is stored in legacy mainframe databases, and the cost to maintain these databases is high.  An added complication is that the data is utilized by many interlinked and dependent programs that have been in place for many years, and sometimes decades. Unlocking the value of this legacy data is also difficult due to many very different types of mainframe databases (e.g., Db2, Adabas, CA Datacom, CA IDMS, etc.).

Immediate data replication on the Cloud is enabling government, healthcare, supply chain, financial, and a variety of public service organizations to meet spikes in demand for vital information, especially in times of crisis. The globalization of markets, increase of data volumes, 24×7 operations, changing business conditions, and high demand for up-to-date information also requires new data transfer and exchange solutions for heterogeneous IT architectures.

The Data-First Solution

Treehouse Software is here to help enterprise mainframe customers accelerate digital transformation and successfully leverage Hybrid Cloud initiatives on the IBM Z platform, storing sensitive data on a private Cloud or local data center and simultaneously leveraging leading technologies on a managed public Cloud. Our tcVISION replication solution focuses on changed data capture (CDC) when transferring information between mainframe data sources and modern databases and applications. Through an innovative technology, changes occurring in any mainframe application data are tracked and captured, and then published to a variety of targets. The customer moves only the right data to the right place at the right time – as much, or as little as they want.

The tcVISION replication solution has a modular design, which enables it to support mass data load from one source to one or more targets, as well as continuous data exchange processes in realtime via CDC. This modular architecture and the provided APIs gives customers unlimited potential for growth and use of new technologies.

tcVISION allows bi-directional, real-time data synchronization of changes on either platform to be reflected on the other platform (e.g., a change to a PostgreSQL table is reflected back on mainframe). The customer can then modernize their application on the cloud, open systems, etc. without disrupting the existing critical work on the legacy system.

In the following example high level architecture diagram, bi-directional data replication between Db2 z/OS and AWS using tcVISION is shown:

___tcVISON_Bidirectional_Db2

tcVISION utilizes a Windows-based GUI Control Board, which is ideal for non-mainframe programmers.  While mainframe experts are required in the design/architecture phase and occasionally during implementation, the requirement for their involvement is limited. The tcVISION Control Board acts as a single point of administration, data modeling and mapping, script generation, and monitoring. Comprehensive monitoring and logging of all data movements ensure transparency across all data exchange processes. In the following example, the mainframe can be seen communicating to an Amazon EC2-based tcVISION replication manager. The tcVISION Control Board shows the user a graphical representation of this replication:

___tcVISION_Control_Board_AWS_Agentless

Additionally, tcVISION supports complex data replication scenarios between multiple data sources and targets, as seen here:

tcVISION_Complex_Replication_Scenarios

With tcVISION, data replication projects can be implemented within a few of months, depending on the complexity of the project.  This includes the proof of concept and design/architecture stages.  After these stages are complete, the customer can start the first production implementation sprint, immediately providing business value.  We suggest successive agile sprints to allow for incremental deployment of additional file replication, sprint by sprint.

Supported Sources and Targets

tcVISION supports a vast array of integration scenarios throughout the enterprise, providing easy and fast data replication for Mainframe-to-Cloud and Open Systems application modernization projects.


__TSI_LOGO

Contact Treehouse Software for a tcVISION Demo Today…

Just fill out the Treehouse Software tcVISION Demonstration Request Form and a Treehouse representative will contact you to set up a time for your tcVISION demonstration. This will be a live, on-line demonstration that shows tcVISION replicating data from the mainframe to a Cloud target database.

Treehouse Software Customer Case Study: A State Government Agency’s Real-time Data Synchronization Between IBM Mainframe Adabas and AWS

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

Mainframe_to_AWS_Graphic

Software AG’s Adabas is a mainframe database that is still heavily used by government sites throughout the U.S. and the world, and this blog focuses on a current Treehouse Software customer – a U.S. State Government Agency that uses Adabas on their mainframe system.

Business Issue

The Agency’s modernization team was looking for a Change Data Capture (CDC) technology solution that enables them to synchronize their mainframe Adabas data on AWS, particularly an Amazon RDS. As with most Treehouse customers, the State’s mainframe contains vital data that must always be highly available, so rather than attempting a complete migration from the mainframe, the modernization teams decided to implement a multi-year data replication plan. This allows the mainframe legacy teams to maintain existing critical applications, while the modernization team develops new applications on AWS.

After researching various technologies, the Agency discovered tcVISION on the AWS Parter Network Blog and contacted Treehouse Software to discuss their project and to see a demonstration of Mainframe-to-AWS data replication.

Addressing the Uniqueness of Adabas

Having specialized in tools and services complementary to Adabas/Natural applications since 1982, Treehouse Software has successfully encountered and addressed many unique scenarios within the Adabas environment. The Treehouse technical team documented three primary issues with Adabas/Natural that the Agency needed to consider when they began planning data replication on AWS:

  1. Adabas has no concept of “transaction isolation”, in that a program may read a record that another program has updated, in its updated state, even though the update has not been committed.  This means that programmatically reading a live Adabas database—one that is available to update users—will almost inevitably lead to erroneous extraction of data.  Record modifications (updates, inserts and deletes) that are extracted, and subsequently backed out, will be represented incorrectly—or not at all—in the target. Because of this, at Treehouse we say “the only safe data source is a static data source”—not the live database.
  2. Many legacy Adabas applications make use of “record typing”, i.e., multiple logical tables stored in a single Adabas file.  Often, each must be extracted to a separate table in the target RDBMS.  The classic example is that of the “code-lookup file”.  Most shops have a single file containing state codes, employee codes, product-type codes, etc.  Records belonging to a given “code table” may be distinguished by the presence of a value in a particular index (descriptor or superdescriptor in ADABAS parlance), or by a range of specific values.  Thus, the extraction process must be able to dynamically assign data content from a given record to different target tables depending on the data content itself.
  3. Adabas is most often used in conjunction with Software AG’s Natural 4GL, and “conveniently” provides for unique datatypes (“D” and “T”) that appear to be merely packed-decimal integers on the surface, but that represent date or date-time values when interpreted using Software AG’s proprietary Natural-oriented algorithm. The most appropriate way to migrate such datatypes is to recognize them and map them to the corresponding native RDBMS datatype (e.g., Oracle DATE) in conjunction with a transformation that decodes the Natural value and formats it to match the target datatype.

The tcVISION Technology Solution...

Adabas_To_AWS

After technical discussions and a successful proof of concept (POC) that proved out a set of use cases, all teams at the Agency determined that tcVISION real-time mainframe data replication capabilities were the perfect fit for meeting their goals.

tcVISION‘s modeling and mapping facilities are utilized to view and capture logical Adabas structures, as documented in Software AG’s PREDICT data dictionary, as well as physical structures as described in Adabas Field Definition Tables (FDTs).  Given that PREDICT is a “passive” data dictionary (there is no requirement that the logical and physical representations agree), it was necessary to scrutinize both to ensure that the source structures were accurately modeled.

Furthermore, tcVISION generates appropriate mappings and transformations for converting Adabas datatypes and structures to corresponding target datatypes and structures, including automatic handling of the proprietary “D” and “T” source datatypes.

The teams examined the three ways that tcVISION can access Adabas data:

  1. ETL – read the active database nucleus
  2. ETL – read datasets containing unloaded Adabas files created by the ADAULD utility
  3. CDC – read the active and archived PLOGs datasets

It was decided to access the data by reading the active and archived PLOGs datasets. The schema, mappings, and transformations from the metadata import were tailored to the customer’s specific requirements.  It is also now possible to import an existing RDBMS schema and retrofit it, via drag-and-drop in tcVISION, to the source Adabas elements.

Additionally, the Agency’s teams are very pleased with tcVISION‘s minimal usage of mainframe resources. The product’s “staged processing” methodology accomplishes this, whereby the only processing occurring on the mainframe is the capture of changes from Adabas PLOGs. The bulk of the processing occurs on the AWS side, minimizing tcVISION’s footprint on the mainframe as seen in this diagram:

tcVISION_Staged_Processing

The user defines on which platform stage their processing should be done. Do as little as possible on the mainframe: Stage 0 – capture data and send data (internal format) to target, and process data in Stages 1 – 3 in AWS.

Customer Outcome

All requirements were met by tcVISION, which led to a successful project implementation.


__001_TSI_LOGO
Contact Treehouse Software for a tcVISION Demo Today…

No matter where you want your mainframe data to go – the Cloud, open systems, or any LUW target – tcVISION from Treehouse Software is your answer.

Just fill out the Treehouse Software tcVISION Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.


Further reading:

Many more mainframe data migration and replication customer case studies can be read on the Treehouse Software Website.

Enterprise Mainframe Change Data Capture (CDC) to Apache Kafka with tcVISION and Confluent

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc. and Ram Dhakne, Solutions Engineer at Confluent

___Mainframe_To_Kafka_Confluent

This blog focuses on using Treehouse Software’s tcVISION to replicate data in real time between mainframes and Confluent, allowing for new use cases and truly setting data in motion.

Why mainframe modernization? Benefits and use cases

Mainframe data stores often hold large amounts of complex and critical data in proprietary legacy formats, making this data difficult to extract and incompatible with modern databases, data types, and data tools.

Enterprises are looking to take advantage of the latest cloud services, such as analytics, artificial intelligence (AI) and machine learning, scalable storage, security, high availability, etc., or move data to a variety of newer databases. Additionally, many customers want to modernize their application on a cloud or open systems platform without disrupting the existing critical work on the legacy system.

How tcVISION syncs legacy data for the cloud

tcVISION is a data replication software product that performs real-time synchronization of mainframe data sources and cloud and open systems, allowing critical mainframe data to be consumed by a variety of leading cloud services.

tcVISION supports many mainframe data sources for both online and offline scenarios. Data can be replicated from IBM Db2 z/OS, Db2 z/VSE, VSAM, IMS/DB, CA IDMS, CA Datacom, or Software AG ADABAS. tcVISION can replicate data to many targets including Confluent Platform, Apache Kafka®, AWS, Google Cloud, Microsoft Azure, PostgreSQL, Snowflake, etc. To learn more, see the complete list of supported tcVISION sources and targets.

tcvision-mainframe-to-confluent-cloud-data-replication-1536x1042

tcVISION focuses on CDC (change data capture) when transferring information between mainframe data sources and cloud and open systems databases and applications. Through innovative technology, changes occurring in any mainframe application data are tracked and captured, and then published to a variety of cloud and open systems targets.

tcVISION stores metadata in a relational database and the tcVISION manager components are administered by the tcVISION control board, a Windows GUI interface, which can be installed on premises or in the cloud. This allows tcVISION users to create metadata, create and control replication scripts, and control database interactions. tcVISION’s architecture is designed to minimize mainframe resource utilization.

Using the tcVISION control board, the most complex transformations can be specified, and it facilitates the mapping of the mainframe copybooks, redefines, data dictionaries, data catalogs, codepages, data type mapping, and more via the user-friendly interface. The repository editor allows users to control data transformations.

What is Confluent?

Confluent Cloud is a real-time data in motion platform that can be deployed in any public cloud, in any region of your choice. It comes with an SLA and uptime of 99.95%, and fully managed components like ZooKeeper, Kafka brokers, 120+ Kafka connectors, Schema Registry, and ksqlDB so you can leverage it on any cloud without having to worry about how it runs and scales.

Kafka Connect, Connect API, connectors, and tcVISION IBM Db2 connector

Kafka comes with three core APIs:

  • Kafka producer/Consumer API
  • Connect API
  • KStreams API

Kafka Connect is a tool for scalably and reliably streaming data between Kafka and other data systems. It makes it simple to quickly define connectors that move large data sets into and out of Kafka. Kafka Connect can ingest entire databases or collect metrics from all your application servers into Kafka topics, making the data available for stream processing with low latency. Kafka Connect connects APIs under the hood with fully managed connector support in Confluent Cloud.

Step-by-step guide on how to use tcVISION and Confluent

This example discusses the integration of tcVISION replication of data from Db2 to Confluent Cloud.

Set up tcVISION access to Confluent

Create an account with Confluent to make a Confluent user ID/password; the user ID is generally your email address. To sign on to Confluent, go to the Confluent Cloud login and enter your user ID:

Confluent Cloud welcome page

Then, enter your password:

Enter your password

When you log in, you’ll be in a Confluent environment called “default”:

Confluent environment called “default”

A Confluent environment is a type of container that holds clusters which in turn hold topics. If you are familiar with messaging systems, Confluent/Kafka will seem familiar. A cluster will need to be created to serve as a target for the data produced by tcVISION. The first attribute to be selected is the type of cluster. Confluent offers three types: Basic, Standard, and Dedicated. For the purposes of this demonstration, Basic will be used. A Basic cluster does not incur charges for simply existing, but does for data transmission and data storage.

Select "Basic cluster" and begin configuration

Select Begin configuration.

Select a cloud provider

Here, a cloud provider can be chosen—AWS, Google Cloud, or Microsoft Azure. For this example, AWS is used. Select Continue and the characteristics of the new cluster are displayed, which we’ve named “tcVISION_cluster_0”:

Cluster characteristics

After entering your payment information (not shown), you can click on the cluster name to launch the cluster overview.

Cluster overview

In order to use Confluent with tcVISION, the user must provide tcVISION with information about the cluster they intend to use. Specifically, the user must supply the hostname and port of the Confluent AWS virtual machine, and the credentials needed to access the cluster.

Confluent refers to the hostname and port as a bootstrap server. There can be multiple bootstrap servers for the purpose of load balancing, but a single server is used for this demonstration.

To find bootstrap server information, click Cluster Settings on the left-hand side:

Cluster settings

The bootstrap server will be listed under “Identification,” and includes both the AWS hostname and the port.

Credentials in Confluent consist of an API Key and an API Secret. These are generated for the cluster and take the place of the Confluent user ID and password used to log in. To generate a key/secret pair, click API Access on the left:

API Keys page

Followed by Create Key:

Select API Key scope

For this example, we use “Global Access” here, so click Next:

API Key and secret

Pay particular attention to the tip about saving the key and secret somewhere safe, because once this panel is exited, there is no way to display the secret again. A descriptive string for this key/secret pair can be filled in. The key or secret text to be copied can be selected, or use the convenient icons at the end of the field to copy. Once the key/secret has been safely stored, check the box that says it has been done, and click Save. You will return to the “API Keys” panel, and the key is now displayed:

API Key displayed

Set up Confluent and define the topic

The last thing to do is define a topic within the cluster. Confluent producers have the capability to define their own topics within a cluster, but this capability can be disabled by a Confluent configuration and is disabled in the configuration used here.

Go back to the cluster Overview:

Cluster Overview

On the left sidebar, click Topics:

Topics

Then Create Topic:

Create a topic

The topic name is filled in (“CONFLUENT_CLOUD_TOPIC1”), overriding the number of partitions from 6 to 1, since that is what the Confluent demo uses. Click Create with defaults:

Cloud topic

A topic is now available, which can be populated with Db2 data.

Set up tcVISION and run a bulk load of Db2 data

tcVISION’s control board is a Windows graphical user interface (GUI) that allows users to configure the replication stream between various database platforms, including the IBM mainframe and Confluent. Using the control board and built-in wizards, users can define the metadata and the mappings between the mainframe and target.

The following sequence of screens shows the steps required to create the tcVISION metadata and scripts for replicating mainframe Db2 z/OS data to Confluent.

Access the tcVISION control board:

tcVISION control board

Log on to Db2 z/OS:

Db2 z/OS

Create metadata that is specific to the input (Db2) and output (Kafka) and the replication definition. In this example, the Db2 table is mapped to the Confluent Cloud Kafka topic using JSON:

Import of structure definitions

The tcVISION metadata wizard asks for the information required for the replication of the mainframe database to Confluent Cloud. For Db2 z/OS, it asks for the mainframe Db2 subsystem:

Source type for structure definition import

Db2 subsystem

tcVISION presents the tables contained in the Db2 z/OS catalog on the mainframe. Select the schemas and associated tables for replication:

Select the schemas and associated tables for replication

Once the required tcVISION wizard-based screens are completed, the tool automatically defines the mappings between the source and target. tcVISION’s metadata import wizard creates a default mapping that handles data type conversion issues, such as EBCDIC to ASCII, Endianness conversion, codepages, redefines data types, and more:

Default mapping

tcVISION data scripts are created through wizards. Data scripts control the replication of data from the source (Db2 z/OS) to the target (Confluent Cloud Kafka JSON). tcVISION bulk load scripts are a type of data script that performs the initial load of the Kafka topic. The following script shows data being accessed directly from the mainframe Db2 z/OS database. Another alternative to reduce MIPS consumption is to read the data from a Db2 image copy.

Data script

Bulk load script running:

Bulk load script running

After execution of the bulk load script, replication statistics of the Db2 bulk load into the Confluent Cloud Kafka topic can be viewed:

Replication statistics of the Db2 bulk load

Now that the topic has been loaded with data from Db2, it can be displayed in Confluent. To do this, navigate to the topics panel again:

Notice that there are now statistics indicating that the tcVISION producer uploaded some data to the topic. On the horizontal menu, switch from “Overview” to “Messages” to display the messages (data records) that the tcVISION bulk load placed in the topic. The display can be filtered in various ways, but for this example, the default is used: “Jump to Offset,” which says “start displaying sequentially from this offset.” Here, an offset of 0 (start at the beginning) is specified, since we just want to verify that the Db2 data uploaded by tcVISION was actually delivered:

Messages (data records) from tcVISION bulk load

Run a change script in tcVISION to show the changes in Confluent

To capture ongoing changes to Db2 in real time, a Db2 z/OS CDC replication script is created.

This script captures the changes on the Db2 z/OS side and applies them into the repository where the output target is Confluent Cloud topic.

Replication script

Replication script

Target database Confluent Cloud topic

The CDC replication is initiated from the tcVISION control board. The tcVISION control board shows a graphical representation of the replication:

Graphical representation of the replication

The CDC replication is now actively capturing and replicating data changes whenever they occur on the Db2 z/OS side. You can test it by making a change in the Db2 z/OS table:

 
********************************* Top of Data **********************************
---------+---------+---------+---------+---------+---------+---------+---------+
UPDATE SXE1.TVKFKATB                                                    00010004
SET DEPT = '696969'                                                     00040029
WHERE PERS_ID = 5;                                                      00050004
---------+---------+---------+---------+---------+---------+---------+---------+
DSNE615I NUMBER OF ROWS AFFECTED IS 1                                           
DSNE616I STATEMENT EXECUTION WAS SUCCESSFUL, SQLCODE IS 0                       
---------+---------+---------+---------+---------+---------+---------+---------+
--COMMIT;                                                               00060019
---------+---------+---------+---------+---------+---------+---------+---------+
DSNE617I COMMIT PERFORMED, SQLCODE IS 0                                         
DSNE616I STATEMENT EXECUTION WAS SUCCESSFUL, SQLCODE IS 0                       
---------+---------+---------+---------+---------+---------+---------+---------+
DSNE601I SQL STATEMENTS ASSUMED TO BE BETWEEN COLUMNS 1 AND 72                  
DSNE620I NUMBER OF SQL STATEMENTS PROCESSED IS 1                                
DSNE621I NUMBER OF INPUT RECORDS READ IS 4                                      
DSNE622I NUMBER OF OUTPUT RECORDS WRITTEN IS 17                                 
******************************** Bottom of Data ********************************

This change is processed and replicated by tcVISION. The tcVISION control board shows the statistics highlighting that one update was performed:

Display of extended statistics

Checking in Confluent, the Db2 z/OS change has successfully been propagated to the Confluent Cloud topic:

Db2 z/OS change successfully propagated to Confluent Cloud topic

tcVISION and Confluent are better together

With tcVISION’s groundbreaking Db2 CDC connector and Confluent’s ability to serve as the multi-tenant data hub, this combination creates a very powerful solution to aggregate data from multiple sources and have data published into various Kafka topics. Sourcing events from any kind of Db2 via a connector into Confluent will set data in motion for the entire organization. Simplicity and agility are key elements of the tcVISION and Confluent “better together” story.


__001_TSI_LOGO

Video: tcVISION Demonstration…

In this video, we show a tcVISION overview, then a demonstration of replication of mainframe data on AWS RDS for PostgreSQL:

Contact Treehouse Software for a tcVISION Demo Today!

No matter where you want your mainframe data to go – the Cloud, open systems, or any LUW target – tcVISION from Treehouse Software is your answer.

Just fill out the Treehouse Software tcVISION Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.

Treehouse Software Customer Success: ETS uses tcVISION for Real-Time Synchronization Between their Mainframe IDMS Data and AWS RDS for PostgreSQL

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

ETS_Graphic

This blog focuses on a current Treehouse Software customer – ETS. Headquartered in Princeton, New Jersey, ETS is a private, nonprofit organization with approximately 3,000 employees devoted to educational measurement and research. ETS develops and administer a broad range of educational products and services for government agencies, academic institutions and corporations, including the TOEFL® and TOEIC® tests, the GRE® General and Subject Tests, and the Praxis® assessments. At nonprofit ETS, our belief in the life-changing power of learning is at the root of everything we do — it’s behind the tools we develop to move learning forward, the research that inspires educational progress and the commitment we make to enable opportunity for learners everywhere. We’re with you on the journey to what’s possible.

Business Background

ETS products and services are available to institutions, businesses, organizations and governments in more than 180 countries around the world. The top industries served by ETS are K–12 Education, Higher Education, English-language Learning, Career Development, and Consulting Services.

Business Issue

Most of ETS’s high volume critical application data is stored on an IBM mainframe in IDMS databases.  The technology is very old, therefore it is difficult to recruit and retain qualified technical personnel to maintain applications.  ETS is moving to Cloud-based computing which will allow them to retire the mainframe environments and modernize the applications.  The data is used and shared across several applications.  ETS required a solution that would allow them to continue, uninterrupted, daily operations on their mainframe while replicating data to their AWS Cloud platform, where they could develop modern application features.  This solution enables ETS to maintain demanding daily processing while they modernize and develop innovative Cloud solutions to meet and exceed customer requirements.

The Technology Solution

ETS_Diagram

Treehouse Software and the ETS team developed a rigid testing plan to implement tcVISION and performed a Proof of Concept to measure the effectiveness of the data replication, considering the high volumes of data changes on the source databases.  We collaborated on architecture requirements and installation steps.  There were many considerations associated with this process, including monitoring, alarming, configuration options, high availability, measuring the impact to existing mainframe database performance, restart capability, and security.  Concurrently, a team of subject matter experts worked on data mappings and translation of database designs from the IDMS network databases to AWS PostgreSQL relational databases.  The goal was to be able to replicate two very large IBM mainframe IDMS databases real-time on two Cloud-based PostgreSQL databases. Implementation was done in phases, starting with one non-production database being replicated to the Cloud.  High-volume testing was performed on the source database to simulate peak processing, replicating millions of transactions to the target PostgreSQL databases.  Many technical challenges were encountered and resolved with outstanding technical assistance from the Treehouse Software support team.  Once in production, the tcVISION product was able to deliver real-time data to the Cloud platform with no interruptions to the customer’s daily processing. The customer was then able to develop modern application features and functions in the Cloud to achieve independence from the legacy mainframe systems.  Using new Cloud-based capabilities enabled the customer to be more agile with meeting new requirements.


__001_TSI_LOGO

Video: tcVISION Demonstration…

In this video, we show a tcVISION overview, then a demonstration of replication of mainframe data on AWS RDS for PostgreSQL:

Contact Treehouse Software for a tcVISION Demo Today!

No matter where you want your mainframe data to go – the Cloud, open systems, or any LUW target – tcVISION from Treehouse Software is your answer.

Just fill out the Treehouse Software tcVISION Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.

What are the Benefits of Replicating Mainframe Data on Cloud or Hybrid Cloud Systems?

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software

Enterprise customers with mainframe systems have begun their movement of data to the Cloud or hybrid Cloud (a mixed computing, storage, and services environment made up of on-premises infrastructure, private Cloud services, and public Cloud) to benefit from new and powerful technologies that deliver significant business benefits and competitive advantage. Compared to the number of mainframe shops that are in the planning stages of their Cloud projects, existing adopters’ numbers are still relatively small.

Today, it is easier than ever for customers to take advantage of cutting edge, Cloud-based technologies, changing the way they manage, deploy, and distribute mission-critical data currently residing on mainframe systems. During the planning phase of a Cloud or hybrid Cloud modernization strategy, some benefits that are quickly discovered include:

Trade Capital Expense for Variable Expense – Instead of having to invest heavily in data centers and servers before customers know how they are going to use them, they pay only when they consume computing resources, and pay only for how much they consume.

Global Deployments – Cloud platforms span many geographic regions globally. Enterprises can easily deploy applications in multiple regions around the world with just a few clicks. This means there can be lower latency and a better experience for customers at minimal cost.

Economies to Scale – By using Cloud computing, customers can achieve a lower variable cost than they can get on their own, because usage from hundreds of thousands of customers is aggregated in the Cloud. Providers such as AWS, Google Cloud, etc. can achieve higher economies of scale, which translates into lower pay as-you-go prices.

Scale of Services – Cloud-based products offer a broad set of global services including compute, storage, databases, analytics, machine learning and AI, networking, mobile, developer tools, management tools, IoT, security, and enterprise applications. These services help organizations move faster, lower IT costs, and scale.

World Class Security – All major Cloud platforms offer advanced and strict security that complies with the most stringent government and private sector requirements.

Extreme High Availability (HA) – Major Cloud platforms span many geographic regions around the world.  By designing services and applications to be redundant across regions, HA is enhanced far beyond a single on-premises data center.

Testing at Scale – Cloud servers and services can be created and charged on demand for a specific amount of time.  This allows customers to create temporary large-scale test environments prior to deployment that are not practical for on-premises environments.  Large scale testing reduces deployment risks and helps to provide a better customer experience.

Auto Scaling and Serverless Deployments – Major Cloud platforms have many serverless and autoscaling options available, allowing for scalable computing capacity as required.  Customers pay only for the compute time they consume – there is no charge when the code is not running. Another example is the ability for a Cloud database to automatically start up, shut down, and scale capacity up or down based on the application’s needs.

Customer Agility and Innovation – In a Cloud computing environment, new IT resources are only a click away, which means that customers reduce the time to make those resources available to developers from weeks to just minutes. This results in a dramatic increase in agility for the organization, since the cost and time it takes to experiment and develop is significantly lower.

Many companies who haven’t started their modernization journeys yet are looking for tools that allow their legacy mainframe environments to continue, while replicating data – in real time – on a variety of Cloud and open systems platforms. Treehouse Software is the worldwide distributor of tcVISION, a software tool that provides an easy and fast approach for Cloud and hybrid Cloud projects, enabling bi-directional data replication between the hardware source and many targets, including (mainframe): Db2 z/OS, Db2 z/VSE, Adabas, VSAM, IMS/DB, CA IDMS, CA DATACOM, etc. and (Cloud and open systems): AWS, Google Cloud, Microsoft Azure, Kafka, PostgreSQL, etc..

If your enterprise is planning on a Mainframe-to-Cloud data modernization project, we would welcome the opportunity to help get you moving immediately with an online demonstration of tcVISION. Contact Treehouse Software for a tcVISION demonstration today!

The Mainframe-to-Hybrid Cloud Wave has Arrived, and Treehouse Software is Helping Customers Begin the Ride!

by Joseph Brady, Director of Business Development / AWS and Cloud Alliance Lead at Treehouse Software and Andy Jones, Certified AWS Solutions Architect at Treehouse Software

There are many pioneering organizations with mainframe systems that have already begun their movement to the Cloud, and are now taking advantage of the new and powerful technologies delivering significant business benefits and competitive advantage, including:

Trade Capital Expense for Variable Expense – Instead of having to invest heavily in data centers and servers before you know how you’re going to use them, you can pay only when you consume computing resources, and pay only for how much you consume. https://aws.amazon.com/pricing/

Global Deployments – The AWS Cloud spans 22 geographic regions globally. Enterprises can easily deploy applications in multiple regions around the world with just a few clicks. This means you can provide lower latency and a better experience for your customers at minimal cost. https://aws.amazon.com/about-aws/global-infrastructure/regions_az/

Economies to Scale – By using Cloud computing, you can achieve a lower variable cost than you can get on your own, because usage from hundreds of thousands of customers is aggregated in the Cloud, providers such as AWS can achieve higher economies of scale, which translates into lower pay as-you-go prices. https://aws.amazon.com/economics/

Scale of Services – Amazon Web Services offers a broad set of global Cloud-based products including compute, storage, databases, analytics, networking, mobile, developer tools, management tools, IoT, security, and enterprise applications. These services help organizations move faster, lower IT costs, and scale. https://aws.amazon.com/products/

World Class Security – AWS security compliance is second to none and complies with the most stringent government and private sector requirements. https://aws.amazon.com/compliance/programs/

Extreme High Availability (HA) – The AWS Cloud spans 69 Availability Zones within 22 geographic Regions around the world https://aws.amazon.com/about-aws/global-infrastructure/regions_az/.  By designing your services and applications to be redundant across AWS availability zones or regions, HA is enhanced far beyond a single on premises data center. https://aws.amazon.com/marketplace/solutions/infrastructure-software/high-availability 

Testing at Scale – AWS servers and services can be created and charged on demand for a specific amount of time.  This allows customer to create temporary large-scale test environments prior to deployment that are not practical for on premises environments.  Large scale testing reduces deployment risks and helps to provide a better customer experience. https://aws.amazon.com/  

Auto Scaling and Serverless Deployments – AWS has many serverless and autoscaling options available, allowing for scalable computing capacity as required.  For example, AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume – there is no charge when your code is not running. Another example is Amazon Aurora Serverless, which is an on-demand, auto-scaling configuration for Amazon Aurora (MySQL-compatible edition), where the database will automatically start up, shut down, and scale capacity up or down based on your application’s needs. https://aws.amazon.com/serverless/

Customer Agility and Innovation – In a Cloud computing environment, new IT resources are only a click away, which means that you reduce the time to make those resources available to your developers from weeks to just minutes. This results in a dramatic increase in agility for the organization, since the cost and time it takes to experiment and develop is significantly lower. https://aws.amazon.com/architecture/

Infrastructure as Code – AWS CloudFormation provides a common language for you to describe and provision all the infrastructure resources in your Cloud environment. CloudFormation allows you to use programming languages or a simple text file to model and provision, in an automated and secure manner, all the resources needed for your applications across all regions and accounts. This gives you a single source of truth for your AWS resources. https://aws.amazon.com/cloudformation/ 

However, compared to the number of mainframe shops in enterprises that are just beginning to plan their moves to the Cloud, existing adopters’ numbers are still relatively small.

As an example of the current boom in overall Cloud growth, the worldwide public sector Cloud market will be growing to a staggering $331B by 2022 according to Gartner. By the end of 2019, more than 30% of technology providers’ new software investments will shift from Cloud-first to Cloud-only, further reducing license-based software spending and increasing subscription-based Cloud revenue. (Source: Forbes) 

Treehouse Software is a Trusted Partner on Your Mainframe-to-Cloud Journey

Treehouse Software is a well-established company serving mainframe customers since 1982. We are currently developing a strong presence in the emerging Cloud market space related to mainframe data migration, primarily through our partnership with Amazon Web Services (AWS).  AWS is aware that most large enterprises use mainframe systems that are housing vast amounts of data encompassing historical, customer, logistics, etc., and they have helped us bring our tcVISION solution to the AWS Marketplace. tcVISION provides real-time replication between a variety of mainframe and non-mainframe sources, including (Mainframe): VSAM, IMS, Db2, CA Datacom, Adabas, CA IDMS, and Flat Files; and (Non-mainframe): AWS RDS databases, AWS Aurora, AWS S3, AWS Kinesis,  PostgreSQL, MySQL, Kafka, MongoDB, Hadoop, Oracle, Microsoft SQL Server, IBM Db2 LUW and Db2 BLU, IBM Informix, SAP Hana, and many more. 

AWS sales and technical leaders within various verticals (GovCloud, Nonprofit, K12/Higher Ed, Automotive, etc.) are also beginning to engage with Treehouse Software to learn more about our unique skills and solution that can help them tap into this potential goldmine of massive amounts of legacy data that needs to be moved to AWS. 

Treehouse Software Helps Customers Begin Moving Their Mainframe Data to the Cloud Immediately

Treehouse Software specializes in providing data replication for enterprise customers who want a fully developed and automated way to move data from their mainframe systems to the Cloud. Treehouse Software’s tcVISION is a low risk option that allows customers to immediately begin moving data to the Cloud while they work on the sometimes massive complexity of application migration. Our experience has shown that projects can become stalled while the application side is being figured out. For example, Treehouse Software recently became involved in a project with a government agency that was facing the complexity of a “big bang” migration, which is slowing the project. We are now presenting them with our tcVISION data replication solution option, where they can replicate data to AWS while maintaining their current environment for modernization and migration of their applications. 

Additionally, Treehouse Software’s decades of experience developing software and working in the IBM mainframe environment, in addition to selling and supporting a comprehensive automated data replication product, is making us a desirable partner for AWS and many Cloud migration companies.  

AWS recently published a blog about tcVISION, our Mainframe-to-Cloud data replication product: https://aws.amazon.com/blogs/apn/real-time-mainframe-data-replication-to-aws-with-tcvision-from-treehouse-software/ 

Additionally, here is a blog about Treehouse Software’s extensive mainframe experience: https://treehousesoftware.wordpress.com/2019/09/12/treehouse-softwares-differentiator-weve-been-helping-enterprise-mainframe-sites-since-1982/

If your enterprise is planning on riding the wave with a Mainframe-to-Cloud migration project, we would welcome the opportunity to help get you moving immediately with an online presentation and demonstration of our tcVISION data replication solution. Contact Treehouse Software today!

Treehouse Software’s Mainframe-to-AWS Data Replication Solution is Featured on the AWS Partner Network Blog

“We are very excited about the APN blog showcasing tcVISION’s Mainframe-to-AWS capabilities to such a substantial and relevant audience. The AWS partner and customer base not only has access to our solution via the tcVISION AMI in the AWS Marketplace, they can now use the tcVISION blog as a valuable educational and marketing tool!”

– Joseph Brady, Director of Business Development and AWS Alliance Lead at Treehouse Software

READ THE BLOG HERE


__TSI_LOGO

Since 1982, Treehouse Software has been serving enterprises worldwide with industry-leading software products and outstanding technical support. Today, Treehouse Software is a global leader in providing data migration, replication, and integration solutions for the most complex and demanding heterogeneous environments, as well as feature-rich, accelerated-ROI offerings for information delivery, business intelligence and analytics, and application modernization.