Treehouse Software Customers are Looking Upwards to Mainframe-to-Cloud Data Replication

The search is on for a mature, easy-to-implement Extract Transform and Load (ETL) solution for migrating mission critical data to the cloud.

Mainframe_to_Cloud

Treehouse Software’s tcVISION supports a vast array of integration scenarios throughout the enterprise, providing easy and fast data migration for mainframe application modernization projects, and enabling data replication between mainframe, Linux, Unix and Windows platforms. This innovative technology offers comprehensive abilities to identify and capture changes occurring in mainframe and relational databases, then publish the required information to an impressive variety of targets, both on-premise and cloud-based.

Mainframe-to-Cloud Case Use Example…

BAWAG P.S.K. is one of the largest banks in Austria, with more than 1.6 million private and business customers and is a well-known brand in the country. Their business strategy is oriented towards low risk and high efficiency.

BAWAG was looking to reduce the load on their IBM mainframe and as a result, reduce costs. The project involved offloading data from their core database system to a less expensive system, in real-time, and to provide read access from that system to the new infrastructure. The primary motivator for this data migration was the constantly increasing CPU costs on the mainframe caused by the growing transaction load of online banking, mobile banking, and the use of self service devices.

BAWAG ultimately migrated their online banking application to the cloud using tcVISION. Realtime Event-handling, Realtime Analytics, Realtime Fraud Prevention are only a few of the use cases that the bank’s solution currently covers.

BAWAG_Diagram

The bank decided to use tcVISION to migrate z/OS DB2 data into a Hadoop data lake (a storage repository that holds raw data in its native format). 20 Million transactions were made within 15 minutes.

Cost Reductions Seen Immediately

BAWAG is now seeing a 35-40 percent reduction of the MIPS consumption for online processing during business hours. After hours, consumption is less, because it is mainly batch processing on the mainframe. Currently, a volume of approximately 30 GB changed data (uncompressed) is replicated from DB2 per day.

In addition to the primary usage scenario, BAWAG can also cover additional use cases. This includes real-time-event handling and stream processing, analytics based upon real-time data as well as the possibility to report and analyze structured and unstructured data with excellent performance. The system can be inexpensively operated on Commodity Hardware and has no scalability limitations. Compared to the savings, the costs of replication (CPU consumption) of tcVISION are now very low.

Additionally, BAWAG plans to extend the use of tcVISION in the future, including implementation of real-time replication from ORACLE into the data lake.


Find out more about tcVISION — Enterprise ETL and Real-Time Data Replication Through Change Data Capture

tcVISION supports a vast array of integration scenarios throughout the enterprise, providing easy and fast data migration for mainframe application modernization projects and enabling bi-directional data replication between mainframe, Linux, Unix and Windows platforms. This innovative technology offers comprehensive abilities to identify and capture changes occurring in mainframe and relational databases, then publish the required information to an impressive variety of targets, both on-premise and Cloud-based.

_0_tcvision_connection_overview

tcVISION acquires data in bulk or via change data capture methods, including in real time, from virtually any IBM mainframe data source (Software AG Adabas, IBM DB2, IBM VSAM, IBM IMS/DB, CA IDMS, CA Datacom, even sequential files), and transform and deliver to virtually any target. In addition, the same product can extract and replicate data from a variety of non-mainframe sources, including Adabas LUW, Oracle Database, Microsoft SQL Server, IBM DB2 LUW and DB2 BLU, IBM Informix and PostgreSQL.


__TSI_LOGO

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.

TREETIP: tcVISION now supports IAM Files – a solution of Innovation Data Processing

IAM

“The Innovation Access Method (IAM) is a transparent alternative to VSAM KSDS, ESDS, and (as a cost option) to AIX and RRDS files. The savings achieved over VSAM can vary, but IAM typically uses 40-80% less I/Os, 20-70% less CPU and 30-70% less DASD. Savings in batch elapsed times and online response times can range from 20-80%.“ (INNOVATION Data Processing)

Users of IAM files now have the possibility of replicating data into any DBMS and NoSQL DBMS systems in the distributed systems environment as well as into Cloud and Hadoop platforms in real-time with tcVISION. Thus, the latest technology can easily be used for Analytics and Business Intelligence. tcVISION also supports bidirectional synchronization between IAM files and DBMS in the distributed environment. Therefore data between Oracle, SQL Server, DB2 LUW, Informix, Adabas LUW can be synchronized in real-time without programming effort. The transparent integration of IAM files in a heterogeneous system environment can be easily and quickly implemented.

Find out more about tcVISION — Enterprise ETL and Real-Time Data Replication Through Change Data Capture

tcVISION provides easy and fast data migration for mainframe application modernization projects and enables bi-directional data replication between mainframe, Linux, Unix and Windows platforms.

_0_tcVISION_Simple_Diagram

tcVISION acquires data in bulk or via change data capture methods, including in real time, from virtually any IBM mainframe data source (Software AG Adabas, IBM DB2, IBM VSAM, IBM IMS/DB, CA IDMS, CA Datacom, even sequential files), and transform and deliver to virtually any target. In addition, the same product can extract and replicate data from a variety of non-mainframe sources, including Adabas LUW, Oracle Database, Microsoft SQL Server, IBM DB2 LUW and DB2 BLU, IBM Informix and PostgreSQL.


__TSI_LOGO

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.

Treehouse Proof of Concept: Bi-directional Replication Between Adabas and SQL Server

Chris Rudolph and Kevin Heimbaugh, Senior Technical Representatives for Treehouse Software, visited a customer site (a large retail and distribution company) to perform a five-day proof of concept (POC) of tcVISION with bi-directional replication between Software AG’s Adabas and Microsoft SQL Server.

Chris and Kevin initially met with the customer team, consisting of the DBA, Applications Manager, and a technical applications person. The agenda for the week was set to:

  • Import metadata from several Adabas files
  • Bulk load the Adabas data into SQL Server
  • Set up replication from Adabas to SQL Server
  • Add the bi-directional replication back to Adabas

Additionally, there were a few other items the customer wanted the Treehouse team to address, including support for date formats; timestamps for bi-directional replication to avoid update conflicts; using Predict views to define multiple SQL Server tables; and support for MUs and PEs. Chris noted that everything on the customer’s list is easily supported, and there are several options for the update scenarios that can be used.

_0_tcVISION_Adabas_To_SQLServer

After the tcVISION components were installed, the POC began by using tcVISION’s Control Board to define a metadata repository database in SQL Server. Once that was set, the teams moved on to import the first Adabas file’s metadata using tcVISON’s Metadata Import Wizard. As part of this process tcVISION generated Adabas to SQL Server schemas and field-to-column links as well as created target tables in SQL Server. Bulk Transfer scripts were created using a wizard to read the Adabas file on the mainframe, and load the data into SQL Server using the SQL Server bulk loader. Chris created a control script to show how tcVISION can concurrently bulk transfer multiple Adabas files into SQL Server This required increasing the tcVISION Manager’s VSE partition size to successfully test multiple load scripts executing in parallel.

The teams moved on to define the real-time change data capture (CDC) scripts necessary to process the Adabas PLOG. The tcVISION scripts use a two-phase approach to queue captured Adabas transaction on the open platform, then transform and apply the transactions to SQL Server. The scripts were set up to automatically generate detailed logs to track the PLOG transactions captured, SQL statements successfully applied to SQL Server, failed SQL statements, and informational items such as auto-corrected data and transactions rejected due to processing rules.

Now that several tables were defined and loaded, the bi-directional process was set up. SQL Server CDC was enabled for each table to be replicated. The team made a change within SQL Server and verified that the change show up in the SQL Server CDC tables. The SQL Server-to-Adabas mappings were defined in the tcVISION metadata repository, including the “back update check” to ensure only non-tcVISION transactions are captured, and the scripts on both Windows and mainframe were defined to create the LUWs from the SQL Server CDC and apply the changes to Adabas.

CDC from SQL Server to Adabas was successfully tested. Chris then showed the ability to create Journal replication where each change can be captured by replication type. The team spent time creating a few more mappings so multiple file / table updates could be tested, in addition to doing updates while the scripts were stopped to simulate a lost connection. This included setting up a new script to process copied PLOG datasets created by the ADARES utility.

The team defined the remainder of their Adabas files to the metadata repository. Some were set them up for bi-directional replication, and others were setup for unidirectional replication and Journal replication. Everything work as expected at the wrap-up meeting where the team provided a live demonstration to management of tcVISION and the items accomplished. The final tcVISION presentation and demo went very well, and everyone was pleased with the progress made during the week.


Find out more about tcVISION — Enterprise ETL and Real-Time Data Replication Through Change Data Capture

tcVISION provides easy and fast data migration for mainframe application modernization projects and enables bi-directional data replication between mainframe, Linux, Unix and Windows platforms.

_0_tcvision_connection_overview

tcVISION acquires data in bulk or via change data capture methods, including in real time, from virtually any IBM mainframe data source (Software AG Adabas, IBM DB2, IBM VSAM, IBM IMS/DB, CA IDMS, CA Datacom, even sequential files), and transform and deliver to virtually any target. In addition, the same product can extract and replicate data from a variety of non-mainframe sources, including Adabas LUW, Oracle Database, Microsoft SQL Server, IBM DB2 LUW and DB2 BLU, IBM Informix and PostgreSQL.


__tsi_logo_400x200

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.