Considerations for Planning Bi-Directional Mainframe Data Replication with tcVISION

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

Data_Modrnization

Many medium-to-large size enterprises use mainframe systems that are housing vast amounts of mission-critical data encompassing historical, customer, logistics, etc. information.  Each mainframe site is unique and can have decades worth of customizations requiring innovative approaches to establishing data replication on Cloud and open systems platforms. Fortunately for these customers, Treehouse Software has been in the mainframe software market since 1982, bringing deep experience in mainframe, Cloud, and open systems technologies, as well as delivering the tcVISION mainframe data replication product. Today, Treehouse Software is helping many enterprise mainframe customers accelerate digital transformation and successfully leverage Hybrid Cloud initiatives on the IBM Z platform, storing sensitive data on a private Cloud or local data center and simultaneously leveraging leading technologies on a managed public Cloud.

Treehouse Software’s tcVISION solution focuses on changed data capture (CDC) when transferring information between mainframe data sources and Cloud and open systems-based databases and applications. Changes occurring in the mainframe application data are then tracked and captured, and published to a variety of targets. Additionally, tcVISION supports bi-directional data replication, where changes on either platform are reflected on the other platform (e.g., a change to a PostgreSQL table in the Cloud is reflected back on mainframe), allowing the customer to modernize their application on the Cloud or open systems without disrupting the existing critical work on the legacy system. tcVISION’s bi-directional replication writes directly to the mainframe database, thereby bypassing all mainframe business logic, so this architecture requires careful planning, as well as thorough and repeated testing.

Plan carefully…

The following section offers some real-world customer examples, as well as considerations and recommendations when planning bi-directional replication for any mainframe/RDBMS environments. Bi-directional replication by its nature is a very complicated undertaking, so it is necessary that customers are fully educated in all environments, software, and processes before attempting to write data back to a mainframe database. It is always recommended that customers use a minimally effective measure of bi-directional replication required to accomplish their goal — and no more. An overblown project with unnecessary bi-directional data replication invites undue complexity and delays.

Real-world customer examples…

Treehouse Software has many customers performing bi-directional data replication, and each scenario is vastly different from the others, even if some have the same sources and targets as each other.  For example, some customers utilize a Master/Master, collision-heavy proposition, while others use uni-directional one way, then “flip a switch” uni-directional the other way. Another example is a customer who has a “grand circle,” where data hits multiple applications before it finally makes its way back to an RDBMS staging database that tcVISION replicates to the mainframe.

Example of a Treehouse customer’s bi-directional data replication environment using tcVISION:

tcVISION_Adabas_To_AWS_RDS

There are many planning and implementation stages that go into a successful mainframe replication environment, and performance testing is a vital part of a successful project.  For example, customers should do performance tests on how long it takes tcVISION to read a database log, transfer data, process data, etc.  During testing at one of our reference customer sites we found a significant difference in how long it took for their test and prod LPARs to transmit data to the Cloud, based on whether the mainframe TCP/IP stack used a 32-bit or 128-bit setting.

At another site, where we are helping a large government agency perform bi-directional replication on mainframe data, their original goal was for a significant percentage of mainframe objects to have bi-directional replication. It was determined that it would be impossible to extract business logic from the existing mainframe application for usage in the downstream application. Therefore, they have decided to use a middleware product to perform the “write-back” to the mainframe database.  Given the complexity of the mainframe application, this has proven the safest way for them to proceed.

Because of the variety of customer scenarios as described above, before any site can attempt bi-directional data replication, it is crucial that they have a well-tested uni-directional process with operational controls in place for a significant time period.  “Operational controls” means processes to restart scripts, evaluation of failed transactions, orchestration of mainframe/non-mainframe DBMS changes, etc.

Please contact Treehouse Software to discuss your Mainframe-to-Cloud and Open Systems modernization plans. We can help put in place a roadmap to modernization success.


__TSI_LOGO

Contact Treehouse Software Today for a tcVISION Demo…

No matter where you want your mainframe data to go – the Cloud, open systems, or any LUW target – tcVISION from Treehouse Software is your answer.

_0_Treehouse_tcV_Cloud_OpenSystems

Just fill out the Treehouse Software tcVISION Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.


Mainframe-to-Cloud Data Replication with tcVISION: Recommendations for Roadmapping Your Deployment on a Cloud Environment

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software

Mainframe_To_Cloud_Roadmap

Careful planning must occur for a Mainframe-to-Cloud data modernization project, including how a customer’s desired Cloud environment will look. This blog serves as a general guide for organizations planning to replicate their mainframe data on Cloud platforms using Treehouse Software‘s tcVISION.

A successful move to the Cloud requires a number of post-migration considerations and solutions in order to modernize an application on the Cloud.  Some examples of these considerations and solutions include: 

Personnel Resource Considerations

Staffing for Mainframe-to-Cloud data replication projects depends on the scale and requirements of your replication project (e.g., bi-directional data replication projects will require more staffing).  

Most customers deploy a data replication product with Windows and Linux knowledgeable staff at varying levels of seniority.  For the architecture and setup tasks, we recommend senior technical staff to deal with complex requirements around the mainframe, Cloud architecture, networking, security, complex data requirements, and high availability.  Less senior staff are effective for the more repeatable deployment tasks such as mapping new database/file deployments.  Business staff and system staff are rarely required but can be necessary for more complex deployment tasks.  For example, bi-directional replication requires matching keys on both platforms and their input might be required.  Other activities would be PII consideration, specifics of data transformation and data verification requirements.

An example of staffing for a very large deployment might be one very part-time project manager, a part-time mainframe DBA/systems programmer, 1-2 staff to setup and deployment the environment and an additional 1-2 staff to manage the existing replication processes.

Environment Considerations

As part of the architecture planning, your team needs to decide how many tiers of deployment are needed for your replication project.  Much like with applications, you may want a Dev, QA, and Prod tier.  For each of these tiers, you will need to decide the level of separation.  For example, you might combine Dev and QA, but not Prod.  Many customers will keep production as a distinct environment.  Each environment will have its own set of resources, including mainframe managers (possibly on separate LPARs), Could VMs (e.g., EC2) for replication processing, and for managed Cloud RDBMSs (such as AWS RDS).  

After the required QA testing, changes are deployed to the production environment.  Object promotion test procedures should be detailed and documented, allowing for less experience personnel to work in some testing tasks.  Adherence to details, processes, and extended testing is most import when deploying bi-directional replication, due to the high impact of errors and difficult remediation.

Rollout Planning

A data replication product is typically deployed using Agile methods with sprints.  This allows for incrementally realized business value.  The first phase is typically a planning/architecture phase during which the technical architecture and deployment process are defined.   Files for replication are deployed in groups during sprint planning.  Initial sprint deployments might be low value file replications to shield the business from any interruptions due to process issues.  Once the team is satisfied that the process is effective, replication is working correctly, and data is verified on the source and targets, wide scale deployments can start.  The number of files to deploy in a sprint will depend on the customer’s requirements.  An example would be to deploy 20 mainframe files per 2–3-week sprint.  Technical personnel and business users need to work together to determine which files and deployment order will have the greatest business benefit.

Security

For security, both on-premises and to the major Cloud environments, there are several considerations:

  • Data will be replicated between a source and target. The data security for PII data must be considered.  In addition, rules such as HIPPA, FIPS, etc. will govern specific security requirements.
  • The path of the data must be considered, whether it is a private path, or if the data transverses the internet. For example, when going from on-premises to the Cloud the major Cloud providers have a VPN option which encrypts data going over the internet.  More secure options are also available, such as AWS Direct Connect and Azure ExpressRoute.  With these options, the on-premises network is connected directly to the Cloud provider edge location via a telecom provider, and the data goes over a private route rather than the internet.
  • Additionally, Cloud services such as S3, Azure Blob Storage, and GCP buckets default to route service connections over the internet. Creating a private end point (e.g., AWS PrivateLink) allows for a private network connection within the Cloud provider’s network.  Private connections that do not traverse the Internet provide better security and privacy.
  • Protecting data at rest is important for both the source and target environments. The modern Z/OS mainframe has advanced pervasive and encryption capabilities: https://www.redbooks.ibm.com/redbooks/pdfs/sg248410.pdf.  The major Cloud providers all provide extensive at-rest encryption capabilities.  Turning on encryption for Cloud Storage and databases is often just a parameter setting and the Cloud provider takes care of the encryption, keys, and certificates automatically.    
  • Protecting data in transit is equally important. There are often multiple transit points to encrypt and protect.  First, is the transit from the mainframe to on-premises to the Cloud VM instance.  A mainframe data replication product should provide protection employing TLS 1.2 to utilize keys and certificates on both the mainframe and Cloud.  Second is from the Cloud VM to the Cloud target database or service.  Encryption may be less important since often these services are in a private environment.  However, encryption can be achieved as required.

High Availability

  • During CDC processing, high availability must be maintained in the Cloud environment. The data replication product should keep track of processing position.  The first can be a Restart file, which keeps track of mainframe log position, target processing position, and uncommitted transactions.  The second can be a container stored on Linux or Windows to store committed unprocessed transactions.  Both need to be on highly available storage with a preference for storage across Availability Zones (AZs), such as Elastic File System (Amazon EFS) or Windows File Server (FSx).
  • The Amazon EC2 instance (or other Cloud instance) can be part of an Auto Scaling Group spread across AZs with minimum and maximum of one Amazon EC2 instance.
  • Upon failure, the replacement Amazon EC2 instance of the replication product’s administrator function is launched and communicates its IP address to the product’s mainframe administrator function. The mainframe then starts communication with the replacement Amazon EC2 instance.
  • Once the Amazon EC2 instance is restarted, it continues processing at the next logical restart point, using a combination of the LUW and Restart files.
  • For production workloads, Treehouse Software recommends turning on Multi-AZ target and metadata databases.

Scalable Storage

  • With scalable storage provided on most Cloud platforms, the customer pays only for what is used. The data replication product should require file-based storage for its files that can grow in size if target processing stops for an unexpected reason.  For example, Amazon EFS, and Amazon FSx provide a serverless elastic file system that lets the customer share file data without provisioning or managing storage.

Analytics

  • All top Cloud platform providers give customers the broadest and deepest portfolio of purpose-built analytics services optimized for all unique analytics use cases. Cloud analytics services allow customers to analyze data on demand, and helps streamline the business intelligence process of gathering, integrating, analyzing, and presenting insights to enhance business decision making.
  • A data replication product should replicate data to several data sources that can easily be captured by various Cloud based analytics services. For example, mainframe database data can be replicated to the various Cloud ‘buckets’ in JSON, CSV, or AVRO format, which allows for consumption by the various Cloud analytic services.  Bucket types include AWS S3, Azure BLOB Data, Azure Data Lake Storage, and GCP Cloud storage.  Several other Cloud analytics type services also support targets including Kafka, Elasticsearch, HADOOP, and AWS Kinesis.
  • Kafka has become a common target and can serve as a central data repository. Most customers target Kafka using JSON formatted replicated mainframe data.  Kafka can be installed on-premises, or using a managed Kafka service, such as the Confluent Cloud, AWS Managed Kafka, or the Azure Event Hub.

Monitoring

  • Monitoring is a critical part of any data replication process. There are several levels of monitoring at various points in a data replication project.  For example, each node of the replication including the mainframe, network communication, Cloud VM instances (such as EC2) and the target Cloud database service all can require a level of monitoring.  The monitoring process will also be different in development or QA vs. a full production deployment.
  • A data replication product should also have its own monitoring features. One important area to measure is performance and it is important to determine where any performance bottleneck is located.  Sometimes it could be the mainframe process, the network, the transformation computation process, or the target database.  A performance monitor helps to detect where the bottleneck is occurring and then the customer can drill down into specifics.  For example, if the bottleneck is the input data, areas to examine are the mainframe replication product component performance, or the network connection.  The next step is to monitor the area where the bottleneck is occurring using the data replication product’s statistics, mainframe monitoring tools, or Cloud monitoring such as AWS CloudWatch.
  • A data replication product should also allow the customer to monitor processing functions during the replication process. The data replication product should also have extensive logs and traces that allow for detailed monitoring of the data replication process and produce detailed replication statistics that include a numeric breakdown of processing statistics by table, type of operation (insert, update delete), and where these operations occurred (mainframe, or target database). 
  • CloudWatch collects monitoring and operational data in the form of logs, metrics, and events, providing customers with a unified view of AWS resources, applications, and services that run on AWS, and on-premises servers. You can use CloudWatch to set high resolution alarms, visualize logs and metrics side by side, take automated actions, troubleshoot issues, discover insights to optimize your applications, and ensure they are running smoothly.
  • Some customers are satisfied with a basic monitoring that polls every five minutes, while others need more detailed monitoring and can choose polls that occur every minute.
  • CloudWatch allows customers to record metrics for EC2 and other Amazon Cloud Services and display them in a graph on a monitoring dashboard. This provides visual notifications of what is going on, such as CPU per server, query time, number of transactions, and network usage.
  • Given the dynamic nature of AWS resources, proactive measures including the dynamic re-sizing of infrastructure resources can be automatically initiated. Amazon CloudWatch alarms can be sent to the customer, such as a warning that CPU usage is too high, and as a result, an auto scale trigger can be set up to launch another EC2 instance to address the load. Additionally, customers can set alarms to recover, reboot, or shut down EC2 instances if something out of the ordinary happens.

Disaster Recovery

  • IT disasters such as data center failures, or cyber attacks can not only disrupt business, but also cause data loss, and impact revenue. Most Cloud platforms offer disaster recovery solutions that minimize downtime and data loss by providing extremely fast recovery of physical, virtual, and Cloud-based servers.
  • A disaster recovery solution must continuously replicate machines (including operating system, system state configuration, databases, applications, and files) into a low-cost staging area in a target Cloud account and preferred region.
  • Unlike snapshot-based solutions that update target locations at distinct, infrequent intervals, a Cloud based disaster recovery solution should provide continuous and asynchronous replication.
  • Consult with your Cloud platform provider to make sure you are adhering to their respective best practices.
  • Example: https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/introduction.html

Artificial Intelligence and Machine Learning

  • Many organizations lack the internal resources to support AI and machine learning initiatives, but fortunately the leading Cloud platforms offer broad sets of machine learning services that put machine learning in the hands of every developer and data scientist. For example, AWS offers SageMaker, GCP has AI Platform, and Microsoft Azure provides Azure AI.
  • Applications that are good candidates for AI or ML are those that need to determine and assign meaning to patterns (e.g., systems used in factories that govern product quality using image recognition and automation, or fraud detection programs in financial organizations that examine transaction data and patterns).

The list goes on…

  • Treehouse Software and our Cloud platform and migration partners can advise and assist customers in designing their roadmaps into the future, taking advantage of the most advanced technologies in the world.
  • Successful customer goals are top priority for all of us, and we can continue to work with our customers on a consulting basis even after they are in production.

Of course, each project will have unique environments, goals, and desired use cases. It is important that specific use cases are determined and documented prior to the start of a project and a tcVISION POC. This planning will allow the Treehouse Software team and the customer develop a more accurate project timeline, have the required resources available, and realize a successful project. 

Your Mainframe-to-Cloud Data Migration Partner…

Treehouse Software is a global technology company and Technology Partner with AWS, Google Cloud, and Microsoft. The company assists organizations with migrating critical workloads of mainframe data to the Cloud.

Further reading on tcVISION from AWS, Google Cloud, and Confluent:

More About tcVISION from Treehouse Software…

__Plans_To_Reality

tcVISION supports a vast array of integration scenarios throughout the enterprise, providing easy and fast data migration for mainframe application modernization projects. This innovative technology offers comprehensive abilities to identify and capture changes occurring in mainframe and relational databases, then publish the required information to an impressive variety of targets, both Cloud and on-premises.

tcVISION acquires data in bulk or via CDC methods from virtually any IBM mainframe data source (Software AG Adabas, IBM Db2, IBM VSAM, CA IDMS, CA Datacom, and sequential files), and transform and deliver to a wide array of Cloud and Open Systems targets, including AWS, Google Cloud, Microsoft Azure, Confluent, Kafka, PostgreSQL, MongoDB, etc. In addition, tcVISION can extract and replicate data from a variety of non-mainframe sources, including Adabas LUW, Oracle Database, Microsoft SQL Server, IBM Db2 LUW and Db2 BLU, IBM Informix, and PostgreSQL.


__TSI_LOGO

Contact Treehouse Software for a tcVISION Demo Today…

Simply fill out our tcVISION Demonstration Request Form and a Treehouse representative will be contacting you to set up a time for your requested demonstration.

Video: Mainframe-to-Azure Data Replication with tcVISION from Treehouse Software

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

Mainframe_To_Azure

Treehouse Software was recently invited by Microsoft Azure Mainframe Modernization technical teams to do a presentation and demonstration of tcVISION, our innovative Mainframe-to-Cloud data replication software product.

In this video, we show an overview of the product, then demonstrate replication of mainframe data on Azure SQL:

Click Here to View the Video


__001_TSI_LOGO

Contact Treehouse Software Today for a tcVISION Demonstration…

No matter where you want your mainframe data to go – the Cloud, Open Systems, or any LUW target – tcVISION from Treehouse Software is your answer.

For more information, please contact customer sales at +1.724.759.7070, email us at sales@treehouse.com, or fill out the Treehouse Software Product Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.

Video: Mainframe-to-AWS Data Replication with tcVISION from Treehouse Software

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

___Video_Mainframe_To_Cloud

Treehouse Software was recently invited by AWS mainframe modernization technical teams to do a presentation and demonstration of tcVISION, our innovative Mainframe-to-Cloud data replication software product.

In this video, Chris Rudolph, Treehouse Software’s tcVISION Product Manager shows an overview of the product, then demonstrates replication of mainframe data on AWS RDS for PostgreSQL:


__tsi_logo_400x200

Contact Treehouse Software Today for a tcVISION Demonstration…

No matter where you want your mainframe data to go – the Cloud, Open Systems, or any LUW target – tcVISION from Treehouse Software is your answer.

If interested in more information, please contact customer sales at +1.724.759.7070, email us at sales@treehouse.com, or fill out the Treehouse Software Product Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.

Mainframe VSAM Change Data Capture (CDC) to Cloud and Open Systems with tcVISION from Treehouse Software

by Joseph Brady, Director of Business Development and AWS and Cloud Alliance Leader at Treehouse Software, Inc.

tcVISION_Mainframe_VSAM

Treehouse Software is the worldwide distributor of tcVISION, the innovative software product that allows immediate data replication between an impressive array of Mainframe sources and Cloud and Open Systems targets. This blog focuses on tcVISION‘s support of VSAM mainframe data sources (batch and CICS on z/OS, and CICS on z/OS and z/VSE).

tcVISION performs VSAM Change Data Capture (CDC) either via its own “DBMS-Extensions”, or via IBM’s CICS VR product. tcVISION has separate DBMS-Extensions to capture changes from CICS (using the CICS External Interface) and batch (via a JCL wrapper). All captured changes, regardless of whether they are performed by tcVISION or CICS VR are written to the z/OS Logstream on the mainframe. tcVISION then reads the Logstream and transfers the transactions to a tcVISION server running in the Cloud or on-prem, which is responsible for queueing, transforming, and applying the captured changes to the specified target.

Additionally, when planning VSAM CDC there are a number of operational items to consider, such as volume of batch transactions, data changes that occur during periods of time while the VSAM file is offline, etc.

In this instructional video, tcVISION is shown capturing changes from VSAM on z/OS and writing them to SQL Server on Windows:

 


__tsi_logo_400x200

Contact Treehouse Software Today for a tcVISION Demonstration…

No matter where you want your mainframe data to go – the Cloud, Open Systems, or any LUW target – tcVISION from Treehouse Software is your answer.

Just fill out the Treehouse Software Product Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.

Why is High Availability so Important for Mainframe Data Modernization on the Cloud?

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

Many customers embarking on Mainframe-to-Cloud data replication projects with Treehouse Software are looking at high availability (HA) as a key consideration in the planning process. All of the major Cloud platforms have robust HA infrastructures that keep businesses running without downtime or human intervention when a zone or instance becomes unavailable. HA basic principles are essentially the same across all Cloud platforms.

In this blog, our example shows how the AWS Global Infrastructure and HA is architected with Treehouse Software’s tcVISION real-time mainframe data replication product. A well planned HA architecture ensures that systems are always functioning and accessible, with deployments located in various Availability Zones (AZs) worldwide.

The following example describes tcVISION‘s HA Architecture on AWS. During tcVISION ’s Change Data Capture (CDC) processing for mainframe data replication on the Cloud, HA must be maintained. The Amazon Elastic Compute Cloud (Amazon EC2), which contains the tcVISION Agent, is part of an Auto Scaling Group that is spread across AZs with Amazon EC2 instance(s).

tcVISION and AWS overall architecture…

___tcVISION_AWS_Overall_Architecture

Upon failure, the replacement Amazon EC2 instance tcVISION Agent is launched and communicates its IP address to the mainframe tcVISION Agent. The mainframe tcVISION Agent then starts communication with the replacement Amazon EC2 tcVISION Agent.

Once the Amazon EC2 tcVISION Agent is restarted, it continues processing at its next logical restart point, using a combination of the LUW and Restart files. LUW files contain committed data transactions not yet applied to the target database. Restart files contain a pointer to the last captured and committed transaction and queued uncommitted CDC data. Both file types are stored on a highly available data store, such as Amazon Elastic File System (EFS).

tcVISION and AWS HA architecture…

___tcVISION_AWS_HA_Architecture

For production workloads, Treehouse Software recommends turning on Multi-AZ target and metadata databases.

To keep all the dynamic data in an HA architecture, tcVISION uses EFS, which provides a simple, scalable, fully managed elastic file system for use with AWS Cloud services and on-premises resources. It is built to scale on-demand to petabytes without disrupting applications, growing and shrinking automatically as you add and remove files, eliminating the need to provision and manage capacity to accommodate growth.

More information on AWS HA


Treehouse Software helps enterprises immediately start synchronizing their mainframe data on the Cloud, Hybrid Cloud, and Open Systems to take advantage of the most advanced, scalable, secure, and highly available technologies in the world with tcVISION

tcVISION supports a vast array of integration scenarios throughout the enterprise, providing easy and fast data replication for mainframe application modernization projects and enabling bi-directional data replication between mainframe, Cloud, Open Systems, Linux, Unix, and Windows platforms.

tcV_Arch01


__TSI_LOGO

Contact Treehouse Software for a Demo Today…

Just fill out the tcVISION Product Demonstration Request Form and a Treehouse representative will contact you to set up a time for your tcVISION demonstration. This will be a live, on-line demonstration that shows tcVISION replicating data from the mainframe to a Cloud target database.

Starting a Mainframe Data Replication Project? Consider Your Use Cases Carefully

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software

___Data_Center_To_Targets_Overview

Planning for Real-time and Bi-directional Mainframe Data Replication with tcVISION

A customer’s mainframe data may be utilized by many interlinked and dependent programs that have been in place for several years, and sometimes, decades, so unlocking the value of this legacy data can be difficult. Therefore, careful planning must occur for a mainframe data modernization project, beginning with identifying uses cases for the project and a Proof of Concept (POC) of data replication software, such as tcVISION, the Mainframe-to-Hybrid Cloud and Open Systems data replication product from Treehouse Software.

This blog serves as a general guide for organizations planning to replicate their mainframe data on Cloud and/or Open Systems platforms using tcVISION.

Questions and Considerations for Your Use Cases

A general principle should be to prove out the data replication technology (tcVISION) by using identified use cases. Listed below are some examples of questions and use cases to assist customers in planning and experiencing successful Mainframe-to-Cloud and/or Open Systems data replication projects:

  • What is/are the mainframe source database(s)? Obvious, yes, but the software solution vendor and outside consultants need this information.
  • What are the critical issues you need to test? Are there any areas that you believe that would be challenging for the vendor? Examples would be specific transformations, data types (e.g., BLOB), required CDC SLAs, field/column changes, specific security requirements, data volume requirements, etc. Document all of the critical test items.
  • Select the minimal set of files (generally 3-10) with representative conditions that will enable you to test all of your critical items. If there are multiple source databases, ensure specific test use cases are defined for each source.
  • What are the target databases? Will you be replicating to a Cloud database manager, such as AWS RDS? Are there additional requirements to replicate to S3, Azure BLOB, GCP Cloud Storage, Kinesis, and Kafka? What needs to be tested?
  • What are your bulk or initial load requirements? tcVISION can load data directly from mainframe databases, mainframe unloads, or image copies. What are the data volumes? Do you have sufficient bandwidth between your mainframe manager and on-premises or Cloud VM to handle your volume requirements?
  • What are your Change Data Capture (CDC) requirements?
  • Do you have plans for bi-directional replication in the future. What are your specific requirements? Since bi-directional replication can be complex and greatly lengthen a modernization project, the customer generally will perform one bi-directional use case for conceptual proof. Will this suffice for your organization?
  • What are your specific high availability requirements? Can they be handled by a technical discussion, or is a specific use case required?
  • What are your general security requirements for data at rest and data in transit? Do you have any specific security regulations to follow, such as HIPPA or FIPS? What are your PII / data masking requirements?
  • What are your schema requirements? For example, tcVISION creates a default schema based on your input mainframe data. Major changes to the default schema usually require a staging database.
  • Do you have staff available to perform the required tasks for the project? For example, for the length of a tcVISION POC you will need part-time staff, 2-4 hours per day. A part-time mainframe administrator will generally require 2-8 elapsed hours. Other staff will include Windows/Linux/Cloud administrators. 2-4 hours of project management may also be required.
  • Are business data transformations required? tcVISION handles minor transformation via point and click (e.g., date format transformations). Major transformations can require C++ or product scripting.
  • Are there any triggers or stored procedures? tcVISION performs CDC replication processing using a database that utilizes these database features.

Of course, each project will have unique environments, goals, and desired use cases. It is important that specific use cases are determined and documented prior to the start of a project and a tcVISION POC. This planning will allow the Treehouse Software team and the customer develop a more accurate project timeline, have the required resources available, and realize a successful project. 

More About tcVISION from Treehouse Software…

__Plans_To_Reality

tcVISION supports a vast array of integration scenarios throughout the enterprise, providing easy and fast data migration for mainframe application modernization projects. This innovative technology offers comprehensive abilities to identify and capture changes occurring in mainframe and relational databases, then publish the required information to an impressive variety of targets, both Cloud and on-premises.

tcVISION acquires data in bulk or via CDC methods from virtually any IBM mainframe data source (Software AG Adabas, IBM Db2, IBM VSAM, IBM IMS/DB, CA IDMS, CA Datacom, and sequential files), and transform and deliver to a wide array of Cloud and Open Systems targets, including AWS, Google Cloud, Microsoft Azure, Confluent, Kafka, PostgreSQL, MongoDB, etc. In addition, tcVISION can extract and replicate data from a variety of non-mainframe sources, including Adabas LUW, Oracle Database, Microsoft SQL Server, IBM Db2 LUW and Db2 BLU, IBM Informix, and PostgreSQL.


__TSI_LOGO

Contact Treehouse Software for a tcVISION Demo Today…

Simply fill out our Product Demonstration Request Form and a Treehouse representative will be contacting you to set up a time for your requested demonstration.

What’s Your Mix?

Real-time and Bi-directional Data Replication Between Mainframes and Virtually Any Target

Cloud004_Swirl

tcVISION from Treehouse Software — Enterprise ETL and Real-Time Data Replication Through Change Data Capture (CDC)

Planning a data replication project between Mainframe, Cloud, and Open Systems platforms? tcVISION supports a vast array of integration scenarios throughout the enterprise, providing easy and fast data migration for mainframe application modernization projects. This innovative technology offers comprehensive abilities to identify and capture changes occurring in mainframe and relational databases, then publish the required information to an impressive variety of targets, both Cloud and on-premises.

tcVISION acquires data in bulk or via CDC methods from virtually any IBM mainframe data source (Software AG Adabas, IBM Db2, IBM VSAM, IBM IMS/DB, CA IDMS, CA Datacom, and sequential files), and transform and deliver to a wide array of Cloud and Open Systems targets, including AWS, Google Cloud, Microsoft Azure, Confluent, Kafka, PostgreSQL, MongoDB, etc. In addition, tcVISION can extract and replicate data from a variety of non-mainframe sources, including Adabas LUW, Oracle Database, Microsoft SQL Server, IBM Db2 LUW and Db2 BLU, IBM Informix and PostgreSQL.

Whatever your mix, Treehouse Software has got it covered with tcVISION.


__TSI_LOGO

Moving the right data to the right place at the right time

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.

Quickly Begin Replicating Mainframe Data on Cloud and Open Systems During a tcVISION Proof of Concept.

by Joseph Brady, Director of Business Development / Cloud Alliance Leader at Treehouse Software, Inc.

___Start_Moving_Now_Graphic

Customers can start moving mainframe data within days during a tcVISION POC…

An online tcVISION Proof of Concept (POC) is approximately 10 business days, with the customer providing a representative subset of data, use cases, and goals for the POC. A Treehouse Software consultant will assist in downloading and installing tcVISION, and conduct a limited-scope implementation of a tcVISION application. This application uses customer data and executes on customer facilities, in a non-production environment. A document is provided beforehand that outlines the requirements and agenda for the POC.

By the end of the POC, customers can begin replicating mainframe data to their Cloud or Open Systems target database.  It can happen that fast!.

About tcVISION

More Cloud, Open Systems, and Systems Integration partners are recommending tcVISION, Treehouse Software’s Mainframe-to-Cloud data replication product for modernization projects. tcVISION focuses on changed data capture (CDC) when transferring information between mainframe data sources and Cloud and Open System databases and applications. Through an innovative technology, changes occurring in any mainframe application data are tracked and captured, and then published to a variety of RDBMS and other targets.

tcVISION_Overall_Diagram_Cloud_OS

Further reading…

Treehouse Software is an AWS, Google Cloud, and Microsoft Technology Partner, and the AWS Partner Network published a blog about tcVISION, which describes how tcVISION allows legacy mainframe environments to continue, while replicating data on highly available and secure Cloud platforms.


__TSI_LOGO

Contact Treehouse Software for a tcVISION Demo Today…

Simply fill out our Product Demonstration Request Form and a Treehouse representative will be contacting you to set up a time for your requested demonstration.

Treehouse Software is Helping Government Agencies with Mainframe Adabas Data Take Advantage of Cutting Edge Cloud-based Technologies

by Joseph Brady, Director of Business Development / Cloud Alliance Lead at Treehouse Software, Inc.

___Gov_IT_Cloud

Many government agencies have large volumes of mission critical and historical data stored in a variety of legacy mainframe databases (for this blog, we are focusing on Adabas). These government agencies combine a broad range human service programs, including employment assistance and job training, child and adult protection, child support enforcement, cash assistance, and services for the developmentally disabled, all of which constantly require accurate, up-to-date, and secure data.

To support the needs of clients, government agencies are generally broken into multiple divisions, including the division that Treehouse Software works with the most — Technology Services. The Technology Services divisions provide technical and systems services for the development, maintenance, and enhancement of automated business systems. They also ensure that the production and sub-production databases are running smoothly and efficiently by performing necessary maintenance of the databases.

Rapidly changing national health and economic conditions are making fast access to the most current information more important than ever for government agencies and the public.  As a result, a top priority for Technology Services divisions is modernizing critical data residing on long-standing mainframe databases. Unlocking the value of this important data can be difficult, because the data can be utilized by numerous interlinked and dependent programs that have been in place for many years, and sometimes decades.

Many Treehouse government customers are now looking for modernization solutions that allow their legacy mainframe environments to continue, while replicating data in real time on highly available Cloud-based platforms, such as AWS, Google Cloud, and Microsoft Azure. tcVISION from Treehouse Software allows a “data-first” approach, whereby immediate data replication to the Cloud helps government agencies begin strategies to meet spikes in demand for vital information, especially in times of crisis.

Just what is it about Adabas?

Adabas is a mainframe database that is still heavily used by government sites throughout the U.S. and the world. Having specialized in tools and services complementary to Adabas/Natural applications since 1982, Treehouse has successfully encountered and addressed many unique issues within the Adabas environment. This excerpt from a Treehouse technical document outlines three primary issues with Adabas/Natural that must be considered:

  1. Adabas has no concept of “transaction isolation”, in that a program may read a record that another program has updated, in its updated state, even though the update has not been committed.  This means that programmatically reading a live Adabas database—one that is available to update users—will almost inevitably lead to erroneous extraction of data.  Record modifications (updates, inserts and deletes) that are extracted, and subsequently backed out, will be represented incorrectly—or not at all—in the target. Because of this, at Treehouse we say “the only safe data source is a static data source”—not the live database.
  2. Many legacy Adabas applications make use of “record typing”, i.e., multiple logical tables stored in a single Adabas file.  Often, each must be extracted to a separate table in the target RDBMS.  The classic example is that of the “code-lookup file”.  Most shops have a single file containing state codes, employee codes, product-type codes, etc.  Records belonging to a given “code table” may be distinguished by the presence of a value in a particular index (descriptor or superdescriptor in ADABAS parlance), or by a range of specific values.  Thus, the extraction process must be able to dynamically assign data content from a given record to different target tables depending on the data content itself.
  3. Adabas is most often used in conjunction with Software AG’s Natural 4GL, and “conveniently” provides for unique datatypes (“D” and “T”) that appear to be merely packed-decimal integers on the surface, but that represent date or date-time values when interpreted using Software AG’s proprietary Natural-oriented algorithm. The most appropriate way to migrate such datatypes is to recognize them and map them to the corresponding native RDBMS datatype (e.g., Oracle DATE) in conjunction with a transformation that decodes the Natural value and formats it to match the target datatype.

For discovery and analysis of legacy source data structures, tcVISION’s modeling and mapping facilities view and capture logical Adabas structures, as documented in Software AG’s PREDICT data dictionary, as well as physical structures as described in Adabas Field Definition Tables (FDTs).  Note that PREDICT is a “passive” data dictionary—there is no requirement that the logical and physical representations agree, so it is necessary to scrutinize both to ensure that the source structures are accurately modeled.

Furthermore, tcVISION generates specification and implementation of appropriate mappings and transformations for converting Adabas datatypes and structures to corresponding RDBMS datatypes and structures, including automatic handling of the proprietary “D” and “T” source datatypes.

There are three ways tcVISION can access Adabas data:

  1. ETL – read the active database nucleus
  2. ETL – read datasets containing unloaded Adabas files created by the ADAULD utility
  3. CDC – read the active and archived PLOGs datasets

It is important to note that the schema, mappings and transformations that result from metadata import can be tailored to any specific requirements after the fact.  It is even possible to import an existing RDBMS schema and retrofit it, via drag-and-drop, to the source Adabas elements.

Using the tcVISION Control Board, a Windows GUI interface, the most complex transformations can be specified. Source fields can be combined into a single column, decomposed into separate columns, and be subject to calculations, database lookups, string operations and even programmatic manipulation. Furthermore, mapping rules can be implemented to specify that data content from a source Adabas record be mapped to one or more target RDBMS tables—each with its own different structure, as desired—based on the data content itself. Target tables can even be populated from more than one source file.

tcVISION supports complex replication scenarios…

tcVISION_Complex_Replication_Scenarios

tcVISION GUI Control Board functions as a central point of administration…

tcVISION_Control_Board

Automatically apply target schema within the Control Board…

tcVISION_Target_Schema

It is impossible to discuss all the features and capabilities of tcVISION within a high-level overview.  Given the maturity, wealth of functionality and relative low cost of tcVISION, as compared to the effort, complexity and risk entailed in a “Do-It-Yourself”, solution there is no reason why a legacy renewal project should run aground on data migration.

tcVISION’s minimal footprint on the mainframe…

Customers are very happy with tcVISION‘s “staged processing” methodology, where the only processing occurring on the mainframe was the capture of changes from Adabas PLOGs. The bulk of the processing occurs on the target platform, minimizing tcVISION’s footprint on the mainframe as seen in this diagram…

tcVISION_Staged_Processing

The user defines on which platform stage their processing should be done. Do as little as possible on the mainframe: Stage 0 – capture data and send data (internal format) to target, and process data up to Stage 3 in receiving environment.

Moving forward with tcVISION…

Treehouse Software has been helping local, state, and federal government agencies with Adabas and Natural in the areas of data migration, security, control, auditing, performance enhancement, etc. for decades. Over the years, Treehouse has expanded its capabilities to address new requirements for modernizing legacy mainframe Adabas databases on various Cloud platforms. By using Treehouse Software’s tcVISION Mainframe-to-Cloud data replication product, our government customers are able to immediately utilize some of the most advanced Cloud tools and services in the world.

tcVISION enables government agencies to synchronize mainframe Adabas data with various highly available and secure Cloud databases, data warehouses. etc.. Additionally, bi-directional, real-time data synchronization will enable changes on either platform to be reflected on the other platform (e.g., a change to a PostgreSQL table is reflected back on the mainframe database). This allows governments to modernize  applications on Cloud platforms without disrupting the existing critical work on the legacy system, and modern tools can now be used in the new environment, greatly enhancing agility.

Replicating mainframe data on the Cloud can happen within days during a tcVISION Proof of Concept (POC)…

tcVISION_Overall_Diagram_General_Cloud01

An online tcVISION POC is approximately 10 business days, with the customer providing use case and goals for the POC. A Treehouse Software consultant will assist in downloading and installing tcVISION and conducting a limited-scope implementation of a tcVISION application. This application uses customer data and executes on customer facilities, usually in a non-production environment. A document is provided beforehand that outlines the requirements, use cases, and agenda for the POC.

By the end of the 10-day POC, customers can begin replicating mainframe data to their Cloud target database.  It can happen that fast!

Further Reading…

_0_AWS_Logo

Treehouse Software is an AWS Technology Partner, and the AWS Partner Network published a blog about tcVISION, our Mainframe-to-Cloud data replication product, which describes how tcVISION allows legacy mainframe environments to continue, while replicating data on highly available and secure Cloud platforms:

https://aws.amazon.com/blogs/apn/real-time-mainframe-data-replication-to-aws-with-tcvision-from-treehouse-software/


__TSI_LOGO

Contact Treehouse Software for a tcVISION Demo Today…

Simply fill out our Product Demonstration Request Form and a Treehouse representative will be contacting you to set up a time for your requested demonstration.