Mainframe-to-Cloud Data Replication with tcVISION: Recommendations for Roadmapping Your Deployment on a Cloud Environment

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software

Mainframe_To_Cloud_Roadmap

Careful planning must occur for a Mainframe-to-Cloud data modernization project, including how a customer’s desired Cloud environment will look. This blog serves as a general guide for organizations planning to replicate their mainframe data on Cloud platforms using Treehouse Software‘s tcVISION.

A successful move to the Cloud requires a number of post-migration considerations and solutions in order to modernize an application on the Cloud.  Some examples of these considerations and solutions include: 

Personnel Resource Considerations

Staffing for Mainframe-to-Cloud data replication projects depends on the scale and requirements of your replication project (e.g., bi-directional data replication projects will require more staffing).  

Most customers deploy a data replication product with Windows and Linux knowledgeable staff at varying levels of seniority.  For the architecture and setup tasks, we recommend senior technical staff to deal with complex requirements around the mainframe, Cloud architecture, networking, security, complex data requirements, and high availability.  Less senior staff are effective for the more repeatable deployment tasks such as mapping new database/file deployments.  Business staff and system staff are rarely required but can be necessary for more complex deployment tasks.  For example, bi-directional replication requires matching keys on both platforms and their input might be required.  Other activities would be PII consideration, specifics of data transformation and data verification requirements.

An example of staffing for a very large deployment might be one very part-time project manager, a part-time mainframe DBA/systems programmer, 1-2 staff to setup and deployment the environment and an additional 1-2 staff to manage the existing replication processes.

Environment Considerations

As part of the architecture planning, your team needs to decide how many tiers of deployment are needed for your replication project.  Much like with applications, you may want a Dev, QA, and Prod tier.  For each of these tiers, you will need to decide the level of separation.  For example, you might combine Dev and QA, but not Prod.  Many customers will keep production as a distinct environment.  Each environment will have its own set of resources, including mainframe managers (possibly on separate LPARs), Could VMs (e.g., EC2) for replication processing, and for managed Cloud RDBMSs (such as AWS RDS).  

After the required QA testing, changes are deployed to the production environment.  Object promotion test procedures should be detailed and documented, allowing for less experience personnel to work in some testing tasks.  Adherence to details, processes, and extended testing is most import when deploying bi-directional replication, due to the high impact of errors and difficult remediation.

Rollout Planning

A data replication product is typically deployed using Agile methods with sprints.  This allows for incrementally realized business value.  The first phase is typically a planning/architecture phase during which the technical architecture and deployment process are defined.   Files for replication are deployed in groups during sprint planning.  Initial sprint deployments might be low value file replications to shield the business from any interruptions due to process issues.  Once the team is satisfied that the process is effective, replication is working correctly, and data is verified on the source and targets, wide scale deployments can start.  The number of files to deploy in a sprint will depend on the customer’s requirements.  An example would be to deploy 20 mainframe files per 2–3-week sprint.  Technical personnel and business users need to work together to determine which files and deployment order will have the greatest business benefit.

Security

For security, both on-premises and to the major Cloud environments, there are several considerations:

  • Data will be replicated between a source and target. The data security for PII data must be considered.  In addition, rules such as HIPPA, FIPS, etc. will govern specific security requirements.
  • The path of the data must be considered, whether it is a private path, or if the data transverses the internet. For example, when going from on-premises to the Cloud the major Cloud providers have a VPN option which encrypts data going over the internet.  More secure options are also available, such as AWS Direct Connect and Azure ExpressRoute.  With these options, the on-premises network is connected directly to the Cloud provider edge location via a telecom provider, and the data goes over a private route rather than the internet.
  • Additionally, Cloud services such as S3, Azure Blob Storage, and GCP buckets default to route service connections over the internet. Creating a private end point (e.g., AWS PrivateLink) allows for a private network connection within the Cloud provider’s network.  Private connections that do not traverse the Internet provide better security and privacy.
  • Protecting data at rest is important for both the source and target environments. The modern Z/OS mainframe has advanced pervasive and encryption capabilities: https://www.redbooks.ibm.com/redbooks/pdfs/sg248410.pdf.  The major Cloud providers all provide extensive at-rest encryption capabilities.  Turning on encryption for Cloud Storage and databases is often just a parameter setting and the Cloud provider takes care of the encryption, keys, and certificates automatically.    
  • Protecting data in transit is equally important. There are often multiple transit points to encrypt and protect.  First, is the transit from the mainframe to on-premises to the Cloud VM instance.  A mainframe data replication product should provide protection employing TLS 1.2 to utilize keys and certificates on both the mainframe and Cloud.  Second is from the Cloud VM to the Cloud target database or service.  Encryption may be less important since often these services are in a private environment.  However, encryption can be achieved as required.

High Availability

  • During CDC processing, high availability must be maintained in the Cloud environment. The data replication product should keep track of processing position.  The first can be a Restart file, which keeps track of mainframe log position, target processing position, and uncommitted transactions.  The second can be a container stored on Linux or Windows to store committed unprocessed transactions.  Both need to be on highly available storage with a preference for storage across Availability Zones (AZs), such as Elastic File System (Amazon EFS) or Windows File Server (FSx).
  • The Amazon EC2 instance (or other Cloud instance) can be part of an Auto Scaling Group spread across AZs with minimum and maximum of one Amazon EC2 instance.
  • Upon failure, the replacement Amazon EC2 instance of the replication product’s administrator function is launched and communicates its IP address to the product’s mainframe administrator function. The mainframe then starts communication with the replacement Amazon EC2 instance.
  • Once the Amazon EC2 instance is restarted, it continues processing at the next logical restart point, using a combination of the LUW and Restart files.
  • For production workloads, Treehouse Software recommends turning on Multi-AZ target and metadata databases.

Scalable Storage

  • With scalable storage provided on most Cloud platforms, the customer pays only for what is used. The data replication product should require file-based storage for its files that can grow in size if target processing stops for an unexpected reason.  For example, Amazon EFS, and Amazon FSx provide a serverless elastic file system that lets the customer share file data without provisioning or managing storage.

Analytics

  • All top Cloud platform providers give customers the broadest and deepest portfolio of purpose-built analytics services optimized for all unique analytics use cases. Cloud analytics services allow customers to analyze data on demand, and helps streamline the business intelligence process of gathering, integrating, analyzing, and presenting insights to enhance business decision making.
  • A data replication product should replicate data to several data sources that can easily be captured by various Cloud based analytics services. For example, mainframe database data can be replicated to the various Cloud ‘buckets’ in JSON, CSV, or AVRO format, which allows for consumption by the various Cloud analytic services.  Bucket types include AWS S3, Azure BLOB Data, Azure Data Lake Storage, and GCP Cloud storage.  Several other Cloud analytics type services also support targets including Kafka, Elasticsearch, HADOOP, and AWS Kinesis.
  • Kafka has become a common target and can serve as a central data repository. Most customers target Kafka using JSON formatted replicated mainframe data.  Kafka can be installed on-premises, or using a managed Kafka service, such as the Confluent Cloud, AWS Managed Kafka, or the Azure Event Hub.

Monitoring

  • Monitoring is a critical part of any data replication process. There are several levels of monitoring at various points in a data replication project.  For example, each node of the replication including the mainframe, network communication, Cloud VM instances (such as EC2) and the target Cloud database service all can require a level of monitoring.  The monitoring process will also be different in development or QA vs. a full production deployment.
  • A data replication product should also have its own monitoring features. One important area to measure is performance and it is important to determine where any performance bottleneck is located.  Sometimes it could be the mainframe process, the network, the transformation computation process, or the target database.  A performance monitor helps to detect where the bottleneck is occurring and then the customer can drill down into specifics.  For example, if the bottleneck is the input data, areas to examine are the mainframe replication product component performance, or the network connection.  The next step is to monitor the area where the bottleneck is occurring using the data replication product’s statistics, mainframe monitoring tools, or Cloud monitoring such as AWS CloudWatch.
  • A data replication product should also allow the customer to monitor processing functions during the replication process. The data replication product should also have extensive logs and traces that allow for detailed monitoring of the data replication process and produce detailed replication statistics that include a numeric breakdown of processing statistics by table, type of operation (insert, update delete), and where these operations occurred (mainframe, or target database). 
  • CloudWatch collects monitoring and operational data in the form of logs, metrics, and events, providing customers with a unified view of AWS resources, applications, and services that run on AWS, and on-premises servers. You can use CloudWatch to set high resolution alarms, visualize logs and metrics side by side, take automated actions, troubleshoot issues, discover insights to optimize your applications, and ensure they are running smoothly.
  • Some customers are satisfied with a basic monitoring that polls every five minutes, while others need more detailed monitoring and can choose polls that occur every minute.
  • CloudWatch allows customers to record metrics for EC2 and other Amazon Cloud Services and display them in a graph on a monitoring dashboard. This provides visual notifications of what is going on, such as CPU per server, query time, number of transactions, and network usage.
  • Given the dynamic nature of AWS resources, proactive measures including the dynamic re-sizing of infrastructure resources can be automatically initiated. Amazon CloudWatch alarms can be sent to the customer, such as a warning that CPU usage is too high, and as a result, an auto scale trigger can be set up to launch another EC2 instance to address the load. Additionally, customers can set alarms to recover, reboot, or shut down EC2 instances if something out of the ordinary happens.

Disaster Recovery

  • IT disasters such as data center failures, or cyber attacks can not only disrupt business, but also cause data loss, and impact revenue. Most Cloud platforms offer disaster recovery solutions that minimize downtime and data loss by providing extremely fast recovery of physical, virtual, and Cloud-based servers.
  • A disaster recovery solution must continuously replicate machines (including operating system, system state configuration, databases, applications, and files) into a low-cost staging area in a target Cloud account and preferred region.
  • Unlike snapshot-based solutions that update target locations at distinct, infrequent intervals, a Cloud based disaster recovery solution should provide continuous and asynchronous replication.
  • Consult with your Cloud platform provider to make sure you are adhering to their respective best practices.
  • Example: https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/introduction.html

Artificial Intelligence and Machine Learning

  • Many organizations lack the internal resources to support AI and machine learning initiatives, but fortunately the leading Cloud platforms offer broad sets of machine learning services that put machine learning in the hands of every developer and data scientist. For example, AWS offers SageMaker, GCP has AI Platform, and Microsoft Azure provides Azure AI.
  • Applications that are good candidates for AI or ML are those that need to determine and assign meaning to patterns (e.g., systems used in factories that govern product quality using image recognition and automation, or fraud detection programs in financial organizations that examine transaction data and patterns).

The list goes on…

  • Treehouse Software and our Cloud platform and migration partners can advise and assist customers in designing their roadmaps into the future, taking advantage of the most advanced technologies in the world.
  • Successful customer goals are top priority for all of us, and we can continue to work with our customers on a consulting basis even after they are in production.

Of course, each project will have unique environments, goals, and desired use cases. It is important that specific use cases are determined and documented prior to the start of a project and a tcVISION POC. This planning will allow the Treehouse Software team and the customer develop a more accurate project timeline, have the required resources available, and realize a successful project. 

Your Mainframe-to-Cloud Data Migration Partner…

Treehouse Software is a global technology company and Technology Partner with AWS, Google Cloud, and Microsoft. The company assists organizations with migrating critical workloads of mainframe data to the Cloud.

Further reading on tcVISION from AWS, Google Cloud, and Confluent:

More About tcVISION from Treehouse Software…

__Plans_To_Reality

tcVISION supports a vast array of integration scenarios throughout the enterprise, providing easy and fast data migration for mainframe application modernization projects. This innovative technology offers comprehensive abilities to identify and capture changes occurring in mainframe and relational databases, then publish the required information to an impressive variety of targets, both Cloud and on-premises.

tcVISION acquires data in bulk or via CDC methods from virtually any IBM mainframe data source (Software AG Adabas, IBM Db2, IBM VSAM, CA IDMS, CA Datacom, and sequential files), and transform and deliver to a wide array of Cloud and Open Systems targets, including AWS, Google Cloud, Microsoft Azure, Confluent, Kafka, PostgreSQL, MongoDB, etc. In addition, tcVISION can extract and replicate data from a variety of non-mainframe sources, including Adabas LUW, Oracle Database, Microsoft SQL Server, IBM Db2 LUW and Db2 BLU, IBM Informix, and PostgreSQL.


__TSI_LOGO

Contact Treehouse Software for a tcVISION Demo Today…

Simply fill out our tcVISION Demonstration Request Form and a Treehouse representative will be contacting you to set up a time for your requested demonstration.

Enterprise Mainframe Change Data Capture (CDC) to Apache Kafka with tcVISION and Confluent

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc. and Ram Dhakne, Solutions Engineer at Confluent

___Mainframe_To_Kafka_Confluent

This blog focuses on using Treehouse Software’s tcVISION to replicate data in real time between mainframes and Confluent, allowing for new use cases and truly setting data in motion.

Why mainframe modernization? Benefits and use cases

Mainframe data stores often hold large amounts of complex and critical data in proprietary legacy formats, making this data difficult to extract and incompatible with modern databases, data types, and data tools.

Enterprises are looking to take advantage of the latest cloud services, such as analytics, artificial intelligence (AI) and machine learning, scalable storage, security, high availability, etc., or move data to a variety of newer databases. Additionally, many customers want to modernize their application on a cloud or open systems platform without disrupting the existing critical work on the legacy system.

How tcVISION syncs legacy data for the cloud

tcVISION is a data replication software product that performs real-time synchronization of mainframe data sources and cloud and open systems, allowing critical mainframe data to be consumed by a variety of leading cloud services.

tcVISION supports many mainframe data sources for both online and offline scenarios. Data can be replicated from IBM Db2 z/OS, Db2 z/VSE, VSAM, IMS/DB, CA IDMS, CA Datacom, or Software AG ADABAS. tcVISION can replicate data to many targets including Confluent Platform, Apache Kafka®, AWS, Google Cloud, Microsoft Azure, PostgreSQL, Snowflake, etc. To learn more, see the complete list of supported tcVISION sources and targets.

tcvision-mainframe-to-confluent-cloud-data-replication-1536x1042

tcVISION focuses on CDC (change data capture) when transferring information between mainframe data sources and cloud and open systems databases and applications. Through innovative technology, changes occurring in any mainframe application data are tracked and captured, and then published to a variety of cloud and open systems targets.

tcVISION stores metadata in a relational database and the tcVISION manager components are administered by the tcVISION control board, a Windows GUI interface, which can be installed on premises or in the cloud. This allows tcVISION users to create metadata, create and control replication scripts, and control database interactions. tcVISION’s architecture is designed to minimize mainframe resource utilization.

Using the tcVISION control board, the most complex transformations can be specified, and it facilitates the mapping of the mainframe copybooks, redefines, data dictionaries, data catalogs, codepages, data type mapping, and more via the user-friendly interface. The repository editor allows users to control data transformations.

What is Confluent?

Confluent Cloud is a real-time data in motion platform that can be deployed in any public cloud, in any region of your choice. It comes with an SLA and uptime of 99.95%, and fully managed components like ZooKeeper, Kafka brokers, 120+ Kafka connectors, Schema Registry, and ksqlDB so you can leverage it on any cloud without having to worry about how it runs and scales.

Kafka Connect, Connect API, connectors, and tcVISION IBM Db2 connector

Kafka comes with three core APIs:

  • Kafka producer/Consumer API
  • Connect API
  • KStreams API

Kafka Connect is a tool for scalably and reliably streaming data between Kafka and other data systems. It makes it simple to quickly define connectors that move large data sets into and out of Kafka. Kafka Connect can ingest entire databases or collect metrics from all your application servers into Kafka topics, making the data available for stream processing with low latency. Kafka Connect connects APIs under the hood with fully managed connector support in Confluent Cloud.

Step-by-step guide on how to use tcVISION and Confluent

This example discusses the integration of tcVISION replication of data from Db2 to Confluent Cloud.

Set up tcVISION access to Confluent

Create an account with Confluent to make a Confluent user ID/password; the user ID is generally your email address. To sign on to Confluent, go to the Confluent Cloud login and enter your user ID:

Confluent Cloud welcome page

Then, enter your password:

Enter your password

When you log in, you’ll be in a Confluent environment called “default”:

Confluent environment called “default”

A Confluent environment is a type of container that holds clusters which in turn hold topics. If you are familiar with messaging systems, Confluent/Kafka will seem familiar. A cluster will need to be created to serve as a target for the data produced by tcVISION. The first attribute to be selected is the type of cluster. Confluent offers three types: Basic, Standard, and Dedicated. For the purposes of this demonstration, Basic will be used. A Basic cluster does not incur charges for simply existing, but does for data transmission and data storage.

Select "Basic cluster" and begin configuration

Select Begin configuration.

Select a cloud provider

Here, a cloud provider can be chosen—AWS, Google Cloud, or Microsoft Azure. For this example, AWS is used. Select Continue and the characteristics of the new cluster are displayed, which we’ve named “tcVISION_cluster_0”:

Cluster characteristics

After entering your payment information (not shown), you can click on the cluster name to launch the cluster overview.

Cluster overview

In order to use Confluent with tcVISION, the user must provide tcVISION with information about the cluster they intend to use. Specifically, the user must supply the hostname and port of the Confluent AWS virtual machine, and the credentials needed to access the cluster.

Confluent refers to the hostname and port as a bootstrap server. There can be multiple bootstrap servers for the purpose of load balancing, but a single server is used for this demonstration.

To find bootstrap server information, click Cluster Settings on the left-hand side:

Cluster settings

The bootstrap server will be listed under “Identification,” and includes both the AWS hostname and the port.

Credentials in Confluent consist of an API Key and an API Secret. These are generated for the cluster and take the place of the Confluent user ID and password used to log in. To generate a key/secret pair, click API Access on the left:

API Keys page

Followed by Create Key:

Select API Key scope

For this example, we use “Global Access” here, so click Next:

API Key and secret

Pay particular attention to the tip about saving the key and secret somewhere safe, because once this panel is exited, there is no way to display the secret again. A descriptive string for this key/secret pair can be filled in. The key or secret text to be copied can be selected, or use the convenient icons at the end of the field to copy. Once the key/secret has been safely stored, check the box that says it has been done, and click Save. You will return to the “API Keys” panel, and the key is now displayed:

API Key displayed

Set up Confluent and define the topic

The last thing to do is define a topic within the cluster. Confluent producers have the capability to define their own topics within a cluster, but this capability can be disabled by a Confluent configuration and is disabled in the configuration used here.

Go back to the cluster Overview:

Cluster Overview

On the left sidebar, click Topics:

Topics

Then Create Topic:

Create a topic

The topic name is filled in (“CONFLUENT_CLOUD_TOPIC1”), overriding the number of partitions from 6 to 1, since that is what the Confluent demo uses. Click Create with defaults:

Cloud topic

A topic is now available, which can be populated with Db2 data.

Set up tcVISION and run a bulk load of Db2 data

tcVISION’s control board is a Windows graphical user interface (GUI) that allows users to configure the replication stream between various database platforms, including the IBM mainframe and Confluent. Using the control board and built-in wizards, users can define the metadata and the mappings between the mainframe and target.

The following sequence of screens shows the steps required to create the tcVISION metadata and scripts for replicating mainframe Db2 z/OS data to Confluent.

Access the tcVISION control board:

tcVISION control board

Log on to Db2 z/OS:

Db2 z/OS

Create metadata that is specific to the input (Db2) and output (Kafka) and the replication definition. In this example, the Db2 table is mapped to the Confluent Cloud Kafka topic using JSON:

Import of structure definitions

The tcVISION metadata wizard asks for the information required for the replication of the mainframe database to Confluent Cloud. For Db2 z/OS, it asks for the mainframe Db2 subsystem:

Source type for structure definition import

Db2 subsystem

tcVISION presents the tables contained in the Db2 z/OS catalog on the mainframe. Select the schemas and associated tables for replication:

Select the schemas and associated tables for replication

Once the required tcVISION wizard-based screens are completed, the tool automatically defines the mappings between the source and target. tcVISION’s metadata import wizard creates a default mapping that handles data type conversion issues, such as EBCDIC to ASCII, Endianness conversion, codepages, redefines data types, and more:

Default mapping

tcVISION data scripts are created through wizards. Data scripts control the replication of data from the source (Db2 z/OS) to the target (Confluent Cloud Kafka JSON). tcVISION bulk load scripts are a type of data script that performs the initial load of the Kafka topic. The following script shows data being accessed directly from the mainframe Db2 z/OS database. Another alternative to reduce MIPS consumption is to read the data from a Db2 image copy.

Data script

Bulk load script running:

Bulk load script running

After execution of the bulk load script, replication statistics of the Db2 bulk load into the Confluent Cloud Kafka topic can be viewed:

Replication statistics of the Db2 bulk load

Now that the topic has been loaded with data from Db2, it can be displayed in Confluent. To do this, navigate to the topics panel again:

Notice that there are now statistics indicating that the tcVISION producer uploaded some data to the topic. On the horizontal menu, switch from “Overview” to “Messages” to display the messages (data records) that the tcVISION bulk load placed in the topic. The display can be filtered in various ways, but for this example, the default is used: “Jump to Offset,” which says “start displaying sequentially from this offset.” Here, an offset of 0 (start at the beginning) is specified, since we just want to verify that the Db2 data uploaded by tcVISION was actually delivered:

Messages (data records) from tcVISION bulk load

Run a change script in tcVISION to show the changes in Confluent

To capture ongoing changes to Db2 in real time, a Db2 z/OS CDC replication script is created.

This script captures the changes on the Db2 z/OS side and applies them into the repository where the output target is Confluent Cloud topic.

Replication script

Replication script

Target database Confluent Cloud topic

The CDC replication is initiated from the tcVISION control board. The tcVISION control board shows a graphical representation of the replication:

Graphical representation of the replication

The CDC replication is now actively capturing and replicating data changes whenever they occur on the Db2 z/OS side. You can test it by making a change in the Db2 z/OS table:

 
********************************* Top of Data **********************************
---------+---------+---------+---------+---------+---------+---------+---------+
UPDATE SXE1.TVKFKATB                                                    00010004
SET DEPT = '696969'                                                     00040029
WHERE PERS_ID = 5;                                                      00050004
---------+---------+---------+---------+---------+---------+---------+---------+
DSNE615I NUMBER OF ROWS AFFECTED IS 1                                           
DSNE616I STATEMENT EXECUTION WAS SUCCESSFUL, SQLCODE IS 0                       
---------+---------+---------+---------+---------+---------+---------+---------+
--COMMIT;                                                               00060019
---------+---------+---------+---------+---------+---------+---------+---------+
DSNE617I COMMIT PERFORMED, SQLCODE IS 0                                         
DSNE616I STATEMENT EXECUTION WAS SUCCESSFUL, SQLCODE IS 0                       
---------+---------+---------+---------+---------+---------+---------+---------+
DSNE601I SQL STATEMENTS ASSUMED TO BE BETWEEN COLUMNS 1 AND 72                  
DSNE620I NUMBER OF SQL STATEMENTS PROCESSED IS 1                                
DSNE621I NUMBER OF INPUT RECORDS READ IS 4                                      
DSNE622I NUMBER OF OUTPUT RECORDS WRITTEN IS 17                                 
******************************** Bottom of Data ********************************

This change is processed and replicated by tcVISION. The tcVISION control board shows the statistics highlighting that one update was performed:

Display of extended statistics

Checking in Confluent, the Db2 z/OS change has successfully been propagated to the Confluent Cloud topic:

Db2 z/OS change successfully propagated to Confluent Cloud topic

tcVISION and Confluent are better together

With tcVISION’s groundbreaking Db2 CDC connector and Confluent’s ability to serve as the multi-tenant data hub, this combination creates a very powerful solution to aggregate data from multiple sources and have data published into various Kafka topics. Sourcing events from any kind of Db2 via a connector into Confluent will set data in motion for the entire organization. Simplicity and agility are key elements of the tcVISION and Confluent “better together” story.


__001_TSI_LOGO

Video: tcVISION Demonstration…

In this video, we show a tcVISION overview, then a demonstration of replication of mainframe data on AWS RDS for PostgreSQL:

Contact Treehouse Software for a tcVISION Demo Today!

No matter where you want your mainframe data to go – the Cloud, open systems, or any LUW target – tcVISION from Treehouse Software is your answer.

Just fill out the Treehouse Software tcVISION Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.

Video: Mainframe-to-Azure Data Replication with tcVISION from Treehouse Software

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

Mainframe_To_Azure

Treehouse Software was recently invited by Microsoft Azure Mainframe Modernization technical teams to do a presentation and demonstration of tcVISION, our innovative Mainframe-to-Cloud data replication software product.

In this video, we show an overview of the product, then demonstrate replication of mainframe data on Azure SQL:

Click Here to View the Video


__001_TSI_LOGO

Contact Treehouse Software Today for a tcVISION Demonstration…

No matter where you want your mainframe data to go – the Cloud, Open Systems, or any LUW target – tcVISION from Treehouse Software is your answer.

For more information, please contact customer sales at +1.724.759.7070, email us at sales@treehouse.com, or fill out the Treehouse Software Product Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.

How to Replicate Mainframe Data on Azure SQL with tcVISION from Treehouse Software

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

tcVISION allows enterprise customers to replicate data between mainframe, Cloud, or Hybrid Cloud while maintaining their legacy environments.

We are currently working with a government site to architect bi-directional mainframe data replication on Azure SQL.  One of the customer’s requirements is for tcVISION to provide real-time data synchronization of changes on either platform reflected on the other platform (e.g., a change to an Azure SQL table is reflected back on mainframe). This way, the customer can modernize their application on the Azure Cloud without disrupting the existing critical work on their legacy system.

tcVISION_Azure_Architecture

VIDEO: See how tcVISION easily connects mainframe systems to Azure SQL…

The tcVISION solution focuses on changed data capture (CDC) when transferring information between mainframe data sources and modern databases and applications. Through an innovative technology, changes occurring in any mainframe application data are tracked and captured, and then published to a variety of targets.

Azure SQL is a supported target in tcVISION, and in this instructional video, tcVISION is shown synchronizing data in real-time between Db2 on z/OS and Azure SQL:


__tsi_logo_400x200

Contact Treehouse Software Today…

No matter where you want your mainframe data to go – the Cloud, Open Systems, or any LUW target – tcVISION from Treehouse Software is your answer.

Just fill out the Treehouse Software Product Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.

Video: Mainframe-to-AWS Data Replication with tcVISION from Treehouse Software

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

___Video_Mainframe_To_Cloud

Treehouse Software was recently invited by AWS mainframe modernization technical teams to do a presentation and demonstration of tcVISION, our innovative Mainframe-to-Cloud data replication software product.

In this video, Chris Rudolph, Treehouse Software’s tcVISION Product Manager shows an overview of the product, then demonstrates replication of mainframe data on AWS RDS for PostgreSQL:


__tsi_logo_400x200

Contact Treehouse Software Today for a tcVISION Demonstration…

No matter where you want your mainframe data to go – the Cloud, Open Systems, or any LUW target – tcVISION from Treehouse Software is your answer.

If interested in more information, please contact customer sales at +1.724.759.7070, email us at sales@treehouse.com, or fill out the Treehouse Software Product Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.

Mainframe VSAM Change Data Capture (CDC) to Cloud and Open Systems with tcVISION from Treehouse Software

by Joseph Brady, Director of Business Development and AWS and Cloud Alliance Leader at Treehouse Software, Inc.

tcVISION_Mainframe_VSAM

Treehouse Software is the worldwide distributor of tcVISION, the innovative software product that allows immediate data replication between an impressive array of Mainframe sources and Cloud and Open Systems targets. This blog focuses on tcVISION‘s support of VSAM mainframe data sources (batch and CICS on z/OS, and CICS on z/OS and z/VSE).

tcVISION performs VSAM Change Data Capture (CDC) either via its own “DBMS-Extensions”, or via IBM’s CICS VR product. tcVISION has separate DBMS-Extensions to capture changes from CICS (using the CICS External Interface) and batch (via a JCL wrapper). All captured changes, regardless of whether they are performed by tcVISION or CICS VR are written to the z/OS Logstream on the mainframe. tcVISION then reads the Logstream and transfers the transactions to a tcVISION server running in the Cloud or on-prem, which is responsible for queueing, transforming, and applying the captured changes to the specified target.

Additionally, when planning VSAM CDC there are a number of operational items to consider, such as volume of batch transactions, data changes that occur during periods of time while the VSAM file is offline, etc.

In this instructional video, tcVISION is shown capturing changes from VSAM on z/OS and writing them to SQL Server on Windows:

 


__tsi_logo_400x200

Contact Treehouse Software Today for a tcVISION Demonstration…

No matter where you want your mainframe data to go – the Cloud, Open Systems, or any LUW target – tcVISION from Treehouse Software is your answer.

Just fill out the Treehouse Software Product Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.

How to Replicate Mainframe Data to a Big Data Environment via Kafka with tcVISION

by Joseph Brady, Director of Business Development and AWS and Cloud Alliance Leader at Treehouse Software, Inc.

tcVISION from Treehouse Software allows enterprise customers to replicate data between mainframe, Cloud, or Hybrid Cloud while maintaining their legacy environments, and one of the more popular targets for mainframe modernization that we have been seeing is Apache Kafka®.

tcVISION_Mainframe_To_Kafka

What is Kafka? 

Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. A data pipeline processes and moves data from one system to another, and a streaming application is an application that consumes streams of data.

Kafka is reliable, stable, flexible, robust, and scales well with numerous consumers, working seamlessly with most popular data warehouses and data lakes like Hadoop, Redshift, S3, BigQuery, Azure, etc. Kafka can also be used for real-time analytics, as well as to process real-time streams to collect Big Data.

See how tcVISION easily connects mainframe systems to Kafka…

Kafka handles massive volumes of data and remains responsive, making Kafka a preferred platform when the volume of the data at the mainframe level –> BIG.

Kafka is a supported target in tcVISION, and in this instructional video, tcVISION is shown synchronizing data in real-time from Db2 on z/OS via Kafka to a Big Data environment:

Additional Reading: Treehouse Software is a Confluent technology partner and we recently co-authored a blog entitled, “Enterprise Mainframe Change Data Capture (CDC) to Apache Kafka with tcVISION and Confluent”.


__tsi_logo_400x200

Contact Treehouse Software Today…

No matter where you want your mainframe data to go – the Cloud, Open Systems, or any LUW target – tcVISION from Treehouse Software is your answer.

Just fill out the Treehouse Software Product Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.

New Faces at Treehouse Software

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

__TSI_LOGO

Treehouse Software is growing and on the move! We are proud to have many staff members who have been here for 20+ years, and we have recently brought on several experienced business, mainframe, and Cloud experts. Meet our newest team members:

John Szakach, Chief Operating Officer

John joined Treehouse as a Business Strategy Consultant and in 2021 was promoted to Chief Operating Officer. While new to Treehouse, John brings over 40 years of relevant work experience to the organization and is an AWS Certified Cloud Practitioner as well as a Certified Project Management Professional. John has held a variety of management roles in different industries including VP of Organizational Effectiveness, VP of Quality Assurance, and VP of New Product Development. He has also held positions as Director of Flight Standards and Quality Control, and Director of Operations. In addition to over 51 years of total flight experience, including 20 years as a pilot for United Airlines, he has received numerous awards including the United Airlines Captain of the Year and the FAA Master Pilot Award, the FAA’s highest award for safety and compliance. John has a Bachelor’s Degree in aviation management.

Dan Miley, Product Support

Dan is a software engineer with deep experience and understanding of IBM Assembler, COBOL, JCL, IDMS, SAP ECC. He has worked with some of the world’s largest organizations, including president/consultant of his own company for over 10 years. Dan has already been instrumental in landing some major mainframe-to-Cloud data modernization customers for Treehouse Software.

Sasha Efron, Senior Technical Representative

Sasha is a mainframe technical specialist and DBA with over 25 years experience in in systems analysis, design, development, enhancement, testing, implementation and maintenance in insurance and banking systems with specialization in Software AG and IBM Mainframe technologies. He also has been involved in legacy modernization projects for several worldwide companies.

Joseph Rogan, Senior Technical Representative

Joseph is a Senior Technology Leader with 30+ years experience working in multiple industries, including transportation (specifically rail), logistics, education, financial services (banking, re-insurance, and trading systems), commercial insurance, and state government. His core competencies include database design and implementation, OLTP, OLAP, and data warehouse design, project planning, and project management. Joseph is also a highly trusted, conceptual, business partner and leader with excellent presentation, negotiating, management, mentoring, and strategic planning skills.

Daniel Vimont, Senior Technical Representative

Daniel brings 30+ years experience in multiple computer languages, databases, frameworks, and distributed processing for mainframe, Cloud, and open systems. He is very familiar with the principles of ETL and CDC in mainframe data transformation and migration. Dan is a Certified AWS Cloud Practitioner and has experience in designing and developing AWS/SDK (boto3) framework for on-premises invocation/monitoring of AWS services. Additionally, Dan’s versatile background as a data and software engineer, educator, and business advisor is a valuable asset to Treehouse’s vision of being a close partner in our customers’ planning and modernization efforts.

Treehouse Software Experts are Our Best Assets

management-team

When asked by prospective customers, “What are your primary differentiators?”, we immediately point to our people who have decades worth of experience in helping mainframe customers with innovative tools, services, and training. Our extensive experience, deep knowledge, and wide-ranging capabilities in mainframe technologies make the company a valued partner for third-party solution providers and a trusted advisor to customers.

We are fortunate to have a staff with a wealth of knowledge and skills that span not only Mainframe, but Cloud, LUW, and Open Systems technologies. Treehouse Software‘s technicians have installed products and trained end-users in some of the largest mainframe sites around the world, and our highly-rated 24X7 technical support is second to none.

The Treehouse Team Approach

Treehouse Software’s expert staff has proven its ability to work effectively as part of a larger team to meet clients’ complex business goals. AWS, Google, Microsoft, IBM, Oracle, Deloitte, Accenture, Confluent, and other large vendors have selected our expertise, technology, services, and training for their mainframe data modernization practices.


__tsi_logo_400x200

Contact Treehouse Software Today…

Treehouse Software has been helping enterprises mainframe customers since 1982, and in recent years, we have been developing a strong presence in the Cloud market space relating to mainframe data replication and modernization. As a result, Treehouse Software is currently working with technical and sales leaders from all popular Cloud platform companies and major systems integrators to take advantage of our deep mainframe skills and our tcVISION Mainframe-to-Cloud data replication solution.

No matter where you want your mainframe data to go – the Cloud, Open Systems, or any LUW target –Treehouse Software is here to help. Contact us to discuss your needs.

Replicating Mainframe Data on Cloud-based Relational Databases

by Joseph Brady, Director of Business Development / AWS and Cloud Alliance Leader at Treehouse Software, Inc.

Right_Data_Right_Place_Time02

Treehouse Software has been helping enterprises mainframe customers since 1982, and in recent years, we have been developing a strong presence in the emerging Cloud market space relating to mainframe data replication and modernization. As a result, Treehouse Software is currently working with technical and sales leaders from all popular Cloud platform companies and major systems integrators to take advantage of our deep mainframe skills and our tcVISION Mainframe-to-Cloud data replication solution.

The Choice is Yours…

tcVISION provides the means for customers to easily replicate relevant data between most mainframe data sources (IBM Db2, IBM VSAM, IBM IMS/DB, Software AG Adabas, CA IDMS, CA Datacom, or even sequential files) and the most popular Cloud platforms, including AWS, Google Cloud, Microsoft Azure, Confluent Cloud, and Oracle Cloud

Today, customers are finding it easier than ever to set up, operate, and scale relational databases in the Cloud. Here is a quick look at some Cloud relational database systems that tcVISION supports…

Amazon Aurora relational database, a MySQL and PostgreSQL-compatible relational database built for the Cloud that combines the performance and availability of traditional enterprise databases with the simplicity and cost-effectiveness of open source databases:

Google Cloud SQL, a fully managed relational database service for MySQL, PostgreSQL, and SQL server. You can connect with nearly any application, anywhere in the world. Cloud SQL automates backups, replication, and failover to ensure your database is reliable, highly available, and flexible to your performance needs:

Azure SQL Database is an intelligent, scalable, relational database service built for the Cloud. Optimize performance and durability with automated, AI-driven features that are always up to date. Focus on building new applications without worrying about storage size or resource management with serverless compute and Hyperscale storage options that automatically scale resources on demand:

How Does tcVISION Work?

tcVISION focuses on changed data capture (CDC) when transferring information between mainframe data sources and Cloud-based databases and applications. Through an innovative technology, changes occurring in any mainframe application data are tracked and captured, and then published to the targets.

tcVISION allows bi-directional, real-time data synchronization of changes on either platform to be reflected on the other platform (e.g., a change to a Cloud PostgreSQL table is reflected back on mainframe). The customer can then modernize their application on the cloud, open systems, etc. without disrupting the existing critical work on the legacy system.

Here is a high-level walkthrough of tcVISION mainframe data replication on Cloud and open systems…


__tsi_logo_400x200

Contact Treehouse Software Today…

No matter where you want your mainframe data to go – the Cloud, Open Systems, or any LUW target – tcVISION from Treehouse Software is your answer.

Just fill out the Treehouse Software Product Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.

Enterprises with Vast Amounts of Data on Mainframe Systems are Perfectly Positioned to Take Advantage of Advanced Cloud-based Machine Learning and AI Technology

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

AI_02

Machine learning and artificial intelligence (AI) are not only revolutionizing the computing and business worlds, but the value proposition for Treehouse Software’s customer base who are replicating large amounts of mainframe data on the Cloud can be profound. Machine learning is proving to be a powerful force in improving customer experience, optimizing business operations, and accelerating innovation. This powerful, predictive technology helps improve customer engagement and conversion by creating personalized web experiences that are based on individual customer preferences and behaviors. Intelligent machine learning search services also boost business productivity and customer satisfaction by delivering accurate and useful information faster from various information sources across the organization. Additionally, business forecasting machine learning and AI tools can accurately predict demand and streamline supply-demand decisions by combining historical time series data with additional variables, such as product features, pricing, and holiday demand.

Treehouse Software is partnered with AWS, Google Cloud, and Microsoft, and has been in the mainframe market space since 1982. Our tcVISION product helps customers replicate their mainframe data, in real-time and bi-directionally, between a vast array of source databases and Cloud technologies. As soon as their mainframe data is made available within Cloud-based data stores, customers can use the wide variety of available machine learning and AI technologies.

Machine Learning and AI technologies are available at your fingertips… 

The dominant Cloud platforms are making incredible leaps in creating the most sophisticated AI tools available today — all literally at one’s fingertips. Never before has such powerful and useful technology been so easily available to so many, bringing the deepest set of machine learning services and supporting Cloud infrastructures instantly into the hands of developers, data scientists, etc. The following are a few examples of today’s most popular Cloud-based tools…

__AWS_On_White

Some examples of leading machine learning and AI product available on AWS:

  • Amazon SageMaker – A fully managed service that provides every developer and data scientist with the ability to build, train, and deploy ML models at scale.
  • AWS pre-trained AI Services – Provide ready-made intelligence for your applications and workflows.
  • More information on AWS AI and Machine Learning products

GoogleCloudPartner

Some examples of leading machine learning and AI product available on Google Cloud:

Microsoft

Some examples of leading machine learning and AI product available on Microsoft Azure:


The Journey Begins with Treehouse Software

__tsi_logo_400x200

Treehouse Software is here to help enterprise mainframe customers accelerate digital transformation and successfully leverage Hybrid Cloud initiatives on the IBM Z platform, storing sensitive data on a private cloud or local data center and simultaneously leveraging robust machine learning and AI resources on a managed public cloud.

tcVISION supports a vast array of integration scenarios throughout the enterprise, providing easy and fast data replication for mainframe application modernization projects. This innovative technology offers comprehensive abilities to identify and capture changes occurring in mainframe and relational databases, then publish the required information to an impressive variety of targets, both Cloud and on-premises.

tcVISION_Overall_Diagram_Cloud_OS

Just fill out the Treehouse Software Product Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.


Further reading: tcVISION Mainframe-to-AWS data replication is featured on the AWS Partner Network Blog…

The AWS Partner blog talks about tcVISION’s Mainframe-to-AWS data replication capabilities, including a technical overview, security, high availability, scalability, and a step-by-step example of the creation of tcVISION metadata and scripts for replicating mainframe Db2 z/OS data on Amazon Aurora. Read the blog here: AWS Partner Network (APN) Blog: Real-Time Mainframe Data Replication to AWS with tcVISION from Treehouse Software.