Treehouse Proof of Concept: Bi-directional Replication Between Adabas and SQL Server

Chris Rudolph and Kevin Heimbaugh, Senior Technical Representatives for Treehouse Software, visited a customer site (a large retail and distribution company) to perform a five-day proof of concept (POC) of tcVISION with bi-directional replication between Software AG’s Adabas and Microsoft SQL Server.

Chris and Kevin initially met with the customer team, consisting of the DBA, Applications Manager, and a technical applications person. The agenda for the week was set to:

  • Import metadata from several Adabas files
  • Bulk load the Adabas data into SQL Server
  • Set up replication from Adabas to SQL Server
  • Add the bi-directional replication back to Adabas

Additionally, there were a few other items the customer wanted the Treehouse team to address, including support for date formats; timestamps for bi-directional replication to avoid update conflicts; using Predict views to define multiple SQL Server tables; and support for MUs and PEs. Chris noted that everything on the customer’s list is easily supported, and there are several options for the update scenarios that can be used.

_0_tcVISION_Adabas_To_SQLServer

After the tcVISION components were installed, the POC began by using tcVISION’s Control Board to define a metadata repository database in SQL Server. Once that was set, the teams moved on to import the first Adabas file’s metadata using tcVISON’s Metadata Import Wizard. As part of this process tcVISION generated Adabas to SQL Server schemas and field-to-column links as well as created target tables in SQL Server. Bulk Transfer scripts were created using a wizard to read the Adabas file on the mainframe, and load the data into SQL Server using the SQL Server bulk loader. Chris created a control script to show how tcVISION can concurrently bulk transfer multiple Adabas files into SQL Server This required increasing the tcVISION Manager’s VSE partition size to successfully test multiple load scripts executing in parallel.

The teams moved on to define the real-time change data capture (CDC) scripts necessary to process the Adabas PLOG. The tcVISION scripts use a two-phase approach to queue captured Adabas transaction on the open platform, then transform and apply the transactions to SQL Server. The scripts were set up to automatically generate detailed logs to track the PLOG transactions captured, SQL statements successfully applied to SQL Server, failed SQL statements, and informational items such as auto-corrected data and transactions rejected due to processing rules.

Now that several tables were defined and loaded, the bi-directional process was set up. SQL Server CDC was enabled for each table to be replicated. The team made a change within SQL Server and verified that the change show up in the SQL Server CDC tables. The SQL Server-to-Adabas mappings were defined in the tcVISION metadata repository, including the “back update check” to ensure only non-tcVISION transactions are captured, and the scripts on both Windows and mainframe were defined to create the LUWs from the SQL Server CDC and apply the changes to Adabas.

CDC from SQL Server to Adabas was successfully tested. Chris then showed the ability to create Journal replication where each change can be captured by replication type. The team spent time creating a few more mappings so multiple file / table updates could be tested, in addition to doing updates while the scripts were stopped to simulate a lost connection. This included setting up a new script to process copied PLOG datasets created by the ADARES utility.

The team defined the remainder of their Adabas files to the metadata repository. Some were set them up for bi-directional replication, and others were setup for unidirectional replication and Journal replication. Everything work as expected at the wrap-up meeting where the team provided a live demonstration to management of tcVISION and the items accomplished. The final tcVISION presentation and demo went very well, and everyone was pleased with the progress made during the week.


Find out more about tcVISION — Enterprise ETL and Real-Time Data Replication Through Change Data Capture

tcVISION provides easy and fast data migration for mainframe application modernization projects and enables bi-directional data replication between mainframe, Linux, Unix and Windows platforms.

_0_tcvision_connection_overview

tcVISION acquires data in bulk or via change data capture methods, including in real time, from virtually any IBM mainframe data source (Software AG Adabas, IBM DB2, IBM VSAM, IBM IMS/DB, CA IDMS, CA Datacom, even sequential files), and transform and deliver to virtually any target. In addition, the same product can extract and replicate data from a variety of non-mainframe sources, including Adabas LUW, Oracle Database, Microsoft SQL Server, IBM DB2 LUW and DB2 BLU, IBM Informix and PostgreSQL.


__tsi_logo_400x200

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.

TREETIP: tcVISION Allows for Surprisingly Innovative Uses

tcVISONv6

Treehouse Professional Services consultants help Adabas / Natural customers in a variety of ways, including DBA services, performance tuning, change management implementation and training, and data replication planning and training. Our experience and long history of service to the Adabas / Natural community helps us create innovative solutions for our customers’ challenges.

Recently Treehouse Senior Technical Representative Chris Rudolph assisted a customer with a tricky data replication problem. The customer uses tcVISION to perform bi-directional replication between Adabas and an RDBMS during the phase-in of a new application. Unfortunately, the new application incorrectly updated certain columns in the RDBMS, which were then replicated to Adabas. The customer attempted to address the issue by running a series of ADASEL reports against the Adabas PLOG and manually checking for “bad” transactions, which was a very time consuming process that pulled the Adabas DBA away from her normal duties.

Chris explained that tcVISION could expedite the process by replicating all transactions for the Adabas file to a journal table capturing the “before” and “after” values of the problematic columns. The developers working on the new application could then identify invalid values, correct the application, and patch the data themselves. This also allowed the Adabas DBA to return to their normal duties.

The journal table now includes columns to display the “before” and “after” values of the corrupted column, Adabas transaction time, end transaction time, operation and Adabas userid. The customer’s developers immediately recognized immense value from being able to query the journal table to find bad data, patch the data, prove that corruption is no longer taking place, and verify that all corrupted instances of the data have been patched. Journal tables have been added for all replicated Adabas files, and the developers now rely on the journal tables for all of their data patches.


Find out more about tcVISION — Enterprise ETL and Real-Time Data Replication Through Change Data Capture

tcVISION provides easy and fast data migration for mainframe application modernization projects and enables bi-directional data replication between mainframe, Linux, Unix and Windows platforms.

_0_tcVISION_Simple_Diagram

tcVISION acquires data in bulk or via change data capture methods, including in real time, from virtually any IBM mainframe data source (Software AG Adabas, IBM DB2, IBM VSAM, IBM IMS/DB, CA IDMS, CA Datacom, even sequential files), and transform and deliver to virtually any target. In addition, the same product can extract and replicate data from a variety of non-mainframe sources, including Adabas LUW, Oracle Database, Microsoft SQL Server, IBM DB2 LUW and DB2 BLU, IBM Informix and PostgreSQL.


__TSI_LOGO

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.

PRODUCT SPOTLIGHT: tcACCESS

Transparent integration of mainframe data sources and mainframe programs into LUW applications

__tcACCESS_Cubes

tcACCESS is a comprehensive software solution that enables two-way integration between IBM mainframe systems and client/server, Web and SOA technologies — without the need for mainframe knowledge or programming effort. tcACCESS is a proven platform that facilitates SQL-based integration of mainframe data sources and programs into LUW applications using industry standards such as SQL, ODBC, JDBC, and .NET. SQL queries to access mainframe data can be easily created using drag and drop techniques — no programming required. Read More


__TSI_LOGO

Subscribe to Treehouse Software’s Monthly eBulletin

TREETIP: tcVISION’s Enhanced Web Statistics Functionality

by Joseph Brady, Manager of Marketing and Technical Documentation at Treehouse Software, Inc.

tcVISIONv6_Logo

With version 6 of tcVISION (Treehouse Software’s product for transferring information between mainframe data sources and LUW databases and applications), a newly enhanced web statistics functionality and web server have been introduced. Any standard web browser can access this server (Internet Explorer, Firefox, Opera, Chrome, Safari, etc.).

This valuable feature enables users to oversee data from the tcVISION Manager Monitor, and statistical and operational information from the tcVISION Manager network. The Manager Monitor displays diagrams in a large, scalable window that shows “running” transfers and “not running” processes, and a diagram that shows statistical data.

The following screen shot shows how the tcVISION web statistics can be displayed to the user…

tcVISION_Web_Stats001

An easy-to-use menu allows complete control of, and access to, all processes within the tcVISION Manager Monitor…

_0_tcVISION_Webstats002

  • “Scripts” shows running processes, completed processes, and not running processes.
  • “Server messages” shows the server messages.
  • “Manager network” shows all Managers of connected Manager Networks.
  • “Statistics” shows a selection of Managers and corresponding scripts from the monitoring database. A selection list with different diagram types is available and a button to save chosen diagrams and their selected settings.
  • “Profiles” shows the profiles of the monitoring database.
  • “Manager” shows the name, the version, the revision and the operating system of the observing Managers.

Find out more about tcVISION — Enterprise ETL and Real-Time Data Replication Through Change Data Capture

tcVISION provides easy and fast data migration for mainframe application modernization projects and enables bi-directional data replication between mainframe, Linux, Unix and Windows platforms.

_0_tcVISION_Simple_Diagram

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.

Need Some Help? Look to Treehouse Professional Services for Adabas / Natural

Since 1982, Treehouse Software has been a leader in helping Adabas/Natural customers with our outstanding software and services, for both mainframe and LUW (Linux/Unix/Windows) platforms.

With Treehouse Professional Services, you can benefit from our decades of experience in Adabas/Natural optimization, systems security, control, auditing, performance enhancement and application development and support.

Give us a call today, and tell us about your current Adabas/Natural situation, and we can discuss how Treehouse Professional Services can ease your company’s financial undertow.

_0_Consulting_Services_Lifesaver002

You can’t afford to have your mission-critical Adabas/Natural systems compromised, even if resources have been limited by budgetary constraints or expertise shortages.

_0_Consulting_Services002

Our expert consultants can provide Adabas/Natural performance analysis, environment tuning and optimization; development, training and implementation services involving any Adabas/Natural-related technology; product installation, upgrades and training; data replication and integration; application rehosting and modernization; and DBA services.

Instantly benefit from Treehouse Software’s 30+ years Software AG experience with our On-Site and Remote DBA Services

The Treehouse On-site and Remote DBA Services offer a proven, cost-effective alternative to full-time in-house systems programmers, system administrators and DBAs. Using our team of experts and monitoring techniques, Treehouse offers world-class, on-site and remote system support and DBA service at affordable prices. Treehouse Remote DBA Services offer various levels of system monitoring and administration. Our team will monitor critical factors within your Software AG environment, apply preventive maintenance for your systems and databases and keep your software up-to-date. We even provide z/OS systems programming services! We are strongly committed to our customers’ success and we deliver the highest quality DBA and support solutions.

Boost your Adabas/Natural team’s abilities today!

Visit the Treehouse Software website for more information on Treehouse Professional Services, or contact us today to discuss your needs.

Request a Free, Online Demo of Treehouse Software’s Mainframe Real-Time and Bidirectional Data Replication / Integration Products

Did you know that Treehouse Software offers online demonstrations of the most complete and flexible portfolio of solutions available anywhere for real-time, bidirectional data replication and integration between mainframe and LUW data sources?

You can see how Treehouse Software’s popular tcACCESS and tcVISION products efficiently and cost-effectively use ETL, CDC, SQL, XML, and SOA technologies for data replication / integration, in an interactive demonstration with our skilled technical experts.

Integrate mainframe data and applications with LUW data sources…

tcACCESS is a comprehensive software solution that enables two-way integration between IBM mainframe systems and client/server, Web and SOA technologies — without the need for mainframe knowledge or programming effort. A proven platform that facilitates SQL-based integration of mainframe data sources and programs into LUW applications, tcACCESS uses industry standards such as SQL, ODBC, JDBC, and .NET. SQL queries to access mainframe data can be easily created using drag and drop techniques — no programming required.

tcACCESS_Diagram01

tcACCESS is a modular software solution. It consists of a base system that can either be implemented as a CICS transaction or as a VTAM application. The base system provides its own communication modules. The heart of the system is the tcACCESS SQL Engine which allows access to mainframe data sources using SQL statements. tcACCESS offers Listener components on the mainframe and on the client, as well as scheduling and security functions. Batch processors automate the information exchange processes between distributed applications.

Enable ETL and bi-directional data replication between mainframe and LUW platforms…

tcVISION allows the exchange of data between heterogeneous databases, from legacy non-relational mainframe sources to standard RDBMSs, in batch or real-time, via CDC (change data capture). With tcVISION, complex replication scenarios can be implemented with ease–including bi-directional “master/master” replication requirements.

tcVISION_simple

tcVISION considerably simplifies mainframe data exchange processes. The structure of the existing mainframe data is analyzed by tcVISION processors, then automatically and natively mapped to the target. The data mapping information is presented in a user-friendly and transparent format – even for users with no mainframe knowledge.

See for yourself, right at your desk…

DemoRequest

Tell us about your challenges. If you have a project where our mainframe data replication and integration products could be of assistance, our skilled sales and technical staff would be happy to set up a free, online demo. Simply fill out our short Treehouse Software Demo Request Form.

tcVISION TREETIP: Navigating The Repository Editor

When modeling and mapping a new metadata structure, tcVISION’s powerful Repository Editor is used to define and modify Input and Output Objects. The rich features within the Editor allow users to:

  • import new objects
  • copy or create new objects
  • import metadata for input and/or output objects
  • define target DBMSs
  • specify input and/or output objects for import
  • define DMBS-specific options
  • define target object names
  • replace/supersede existing object metadata

To assist new and even experienced tcVISION users, we’ve put together the following screen shots to give a quick overview of the Repository Editor’s functionality buttons / icons used for setting up modeling and mapping of sources and targets.

Here is an example of what the tcVISION Repository Editor looks like:

_0_tcVISION_RepositoryEditor01

Repository Editor icon definitions: Input objects

_0_tcVISION_RepositoryEditor02Input

Repository Editor icon definitions: Output objects

_0_tcVISION_RepositoryEditor02output

Repository editor right click menu options

_0_tcVISION_RepositoryEditor03OutputOptions

Repository editor extended output target options

_0_tcVISION_RepositoryEditor03RightClick


Enterprise ETL and Real-Time Data Replication Through Change Data Capture

tcVISION_Architecture

tcVISION provides real-time data replication through change data capture, and allows easy and fast data migration for mainframe application modernization projects. Enterprises looking for a product that enables bi-directional heterogeneous data replication between mainframe, Linux, Unix, and Windows platforms need look no further than to tcVISION from Treehouse Software.

To learn more about tcVISION, or to request a demonstration, contact Treehouse Software today.