TREETIP: tcVISION Allows for Surprisingly Innovative Uses

tcVISONv6

Treehouse Professional Services consultants help Adabas / Natural customers in a variety of ways, including DBA services, performance tuning, change management implementation and training, and data replication planning and training. Our experience and long history of service to the Adabas / Natural community helps us create innovative solutions for our customers’ challenges.

Recently Treehouse Senior Technical Representative Chris Rudolph assisted a customer with a tricky data replication problem. The customer uses tcVISION to perform bi-directional replication between Adabas and an RDBMS during the phase-in of a new application. Unfortunately, the new application incorrectly updated certain columns in the RDBMS, which were then replicated to Adabas. The customer attempted to address the issue by running a series of ADASEL reports against the Adabas PLOG and manually checking for “bad” transactions, which was a very time consuming process that pulled the Adabas DBA away from her normal duties.

Chris explained that tcVISION could expedite the process by replicating all transactions for the Adabas file to a journal table capturing the “before” and “after” values of the problematic columns. The developers working on the new application could then identify invalid values, correct the application, and patch the data themselves. This also allowed the Adabas DBA to return to their normal duties.

The journal table now includes columns to display the “before” and “after” values of the corrupted column, Adabas transaction time, end transaction time, operation and Adabas userid. The customer’s developers immediately recognized immense value from being able to query the journal table to find bad data, patch the data, prove that corruption is no longer taking place, and verify that all corrupted instances of the data have been patched. Journal tables have been added for all replicated Adabas files, and the developers now rely on the journal tables for all of their data patches.


Find out more about tcVISION — Enterprise ETL and Real-Time Data Replication Through Change Data Capture

tcVISION provides easy and fast data migration for mainframe application modernization projects and enables bi-directional data replication between mainframe, Linux, Unix and Windows platforms.

_0_tcVISION_Simple_Diagram

tcVISION acquires data in bulk or via change data capture methods, including in real time, from virtually any IBM mainframe data source (Software AG Adabas, IBM DB2, IBM VSAM, IBM IMS/DB, CA IDMS, CA Datacom, even sequential files), and transform and deliver to virtually any target. In addition, the same product can extract and replicate data from a variety of non-mainframe sources, including Adabas LUW, Oracle Database, Microsoft SQL Server, IBM DB2 LUW and DB2 BLU, IBM Informix and PostgreSQL.


__TSI_LOGO

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.

PRODUCT SPOTLIGHT: tcVISION v6 Overview and Updates

tcVISONv6

Several exciting new features and updates are now in tcVISION v6, including new output targets Adabas LUW, DB2 BLU, EXASOL, Hadoop,and MongoDB. Additionally, new input sources include z/OS VSAM Logstream (CICS and Coupling Facility / Shared VSAM),z/OS VSAM Batch Extension, z/OS DBMS to Logstream, CA IDMS v17, CA Datacom CDC, IMS Active Log and SMF data.

Another feature recently announced is the tcVISION “Direct Loader” for BULK_LOAD processing. The function does not require output to a sequential file, and the loader utility for the target DBMS is called via API with data passed directly. Direct loader supports PostGreSQL, Microsoft SQL Server and DB2 LUW / DB2 BLU. The advantage of using the Direct Loader is the elimination of disk access in writing and reading the sequential loader data file. File output is still supported (e.g., where loader data is to be distributed to other machines).

Finally, as mentioned in a previous Treehouse Blog, with tcVISION v6 comes the newly enhanced web statistics functionality and web server. Any standard web browser can access this server (Internet Explorer, Firefox, Opera, Chrome, Safari, etc.)

This valuable feature enables users to view data from the tcVISION Manager Monitor, and statistical and operational information from the tcVISION Manager network.


Find out more about tcVISION — Enterprise ETL and Real-Time Data Replication Through Change Data Capture

tcVISION provides easy and fast data migration for mainframe application modernization projects and enables bi-directional data replication between mainframe, Linux, Unix and Windows platforms.

_0_tcVISION_Simple_Diagram

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.

Mainframe CDC from Treehouse Software

The globalization of markets, increase of data volumes, and high demand for up-to-date information require new data transfer and exchange solutions for heterogeneous IT architectures, and as many customers have discovered, Treehouse Software has the right product (or combination of products) to meet any conceivable mainframe data migration, replication, or integration requirement. To meet many of these needs, Treehouse Software’s proven and mature tcVISION product moves data – as little as possible – as much as necessary. tcVISION is an innovative software solution that processes changed data in real time, in intervals, or event based.

ChangeDataCapture

The tcVISION solution focuses on changed data capture (CDC) when transferring information between mainframe data sources and LUW databases and applications. Changes occurring in any mainframe application data are tracked and captured, and then published to a variety of RDBMS and other targets.

_0_tcVISION_Simple_Diagram

tcVISION enables bidirectional replication for DB2, Oracle, and SQL Server running on Linux/Unix/Windows, and synchronizes each data source, first by doing a bulk load from source(s) to target and then by replicating only changes— only committed changes—from source(s) to target. So there can never be ambiguity as to whether a query against the target database involves uncommitted data.

Read some tcVISON customer success stories here.


Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.

 

tcVISION TREETIP: Navigating The Repository Editor

When modeling and mapping a new metadata structure, tcVISION’s powerful Repository Editor is used to define and modify Input and Output Objects. The rich features within the Editor allow users to:

  • import new objects
  • copy or create new objects
  • import metadata for input and/or output objects
  • define target DBMSs
  • specify input and/or output objects for import
  • define DMBS-specific options
  • define target object names
  • replace/supersede existing object metadata

To assist new and even experienced tcVISION users, we’ve put together the following screen shots to give a quick overview of the Repository Editor’s functionality buttons / icons used for setting up modeling and mapping of sources and targets.

Here is an example of what the tcVISION Repository Editor looks like:

_0_tcVISION_RepositoryEditor01

Repository Editor icon definitions: Input objects

_0_tcVISION_RepositoryEditor02Input

Repository Editor icon definitions: Output objects

_0_tcVISION_RepositoryEditor02output

Repository editor right click menu options

_0_tcVISION_RepositoryEditor03OutputOptions

Repository editor extended output target options

_0_tcVISION_RepositoryEditor03RightClick


Enterprise ETL and Real-Time Data Replication Through Change Data Capture

tcVISION_Architecture

tcVISION provides real-time data replication through change data capture, and allows easy and fast data migration for mainframe application modernization projects. Enterprises looking for a product that enables bi-directional heterogeneous data replication between mainframe, Linux, Unix, and Windows platforms need look no further than to tcVISION from Treehouse Software.

To learn more about tcVISION, or to request a demonstration, contact Treehouse Software today.

L10n in Heterogeneous Data Replication

by Wayne Lashley, Chief Business Development Officer for Treehouse Software

Most software vendors whose product markets extend beyond their own home country are familiar with the concepts of “i18n” and “L10n”, which are numeronyms for “internationalization” and “localization” respectively. i18n is the process of making a software product capable of adaptation to different languages and cultures, while L10n is the specific adaptation process for a given local market.

These terms take on special significance in the context of data replication software products—such as Treehouse’s DPSync, which provides real-time replication of mainframe ADABAS data to relational database (RDBMS) targets like DB2, Microsoft SQL Server and Oracle on various platforms. The very purpose of these products is to take data from a source and apply appropriate L10n to make it usable at the target, which is generally dissimilar in various aspects of the technical environment.

Perhaps the simplest form of L10n, having nothing to do with language or locale, is to transform database-specific field/column datatypes. Alphanumeric (A) fields in ADABAS are often mapped to CHAR or VARCHAR datatypes in an RDBMS, which are conceptually quite similar. Packed (P) fields may be expressed in an RDBMS as NUMBER, INTEGER, NUMERIC, DECIMAL, etc., depending on the vendor implementation and desired usages.

When it comes to Binary (B) format, things get tricky.  An array of bits in an ADABAS field can’t usually be mapped directly to a binary representation in an RDBMS column, due to the differences in the way data are represented between the platforms.

Decades ago, when I was earning my stripes as a novice mainframe programmer, the rules seemed simple: 8 bits made up a byte, and characters were expressed in single bytes encoded in EBCDIC.

(True story: During a university Assembler class many years ago, one of my classmates was muttering to himself, and the professor queried him about the subject of the “conversation”. The student replied “Just practicing my EBCDIC, sir!”)

Later on, I learned about that ASCII column of the “CODE TRANSLATION TABLE” in my indispensable System/370 Reference Summary GX20-1850-3, and I realized there was a whole world of computers beyond mainframes.

Image

But in fact things can be much more complex than simply EBCDIC and ASCII. L10n of data has to take into account the multitude of code pages and conventions that customers may use—and the customizations and exceptions to these.

Our European Technical Representative, Hans-Peter Will, has had to become somewhat of an expert in this over the past few years as he has worked with various customers in the Middle East on DPSync implementations.

Take the case of the way the Arabic language is handled in the context of applications at one site. Arabic is normally read right-to-left. But depending on system configuration, Arabic characters in a given field may be stored either left-to-right or right-to-left. Certain characters are represented in one byte, others in two. The cursive appearance of certain characters must be altered if they appear in the middle of a word rather than on an end. And in certain of this customer’s applications, the same screen display may show both Arabic and English. Even on screens where all of the words are in Arabic, and displayed right-to-left, there may be embedded numbers (e.g., telephone numbers) that need to be displayed left-to-right.

Now take all these complexities and factor in different database management systems (ADABAS vs. Oracle) running on different platforms (mainframe vs. Unix), each of which have their own configuration settings that affect the way characters are stored and displayed. Add to that the question of endianness (big-endian vs. little-endian) of the processing architecture.

The first time that Hans-Peter visited the customer in question, Treehouse software engineers had to figure out how to handle all these issues to ensure that ADABAS data would be replicated accurately and appropriately for use in Oracle-based applications. Fortunately, the combination of great product maturity (DPSync and its key underlying components tRelational/DPS having been battle-tested at countless sites over many years) and product extensibility (the ability to plug in complex custom transformations) enabled DPSync to be readily configured to accomplish the task at hand.

Having learned from that initial experience, Hans-Peter is now on familiar ground when assisting new Arabic-language sites implementing DPSync. Recently he was back in the Middle East visiting one of these new customers, and only hours after product installation he was able to confirm the accuracy of the SQL Server representation of data materialized (initially loaded via what is commonly called ETL, Extract-Transform-Load) from ADABAS using DPSync. The customer was also impressed with the speed of the process, both in terms of configuring the materialization (taking advantage of the tRelational schema auto-generation feature) and executing it (using an ADASAV backup as source, avoiding any workload on ADABAS). That customer is now in production with real-time ADABAS-to-SQL Server replication.

What’s your L10n challenge? Contact Treehouse and learn how DPSync and our other products are able to meet it.

TREEHOUSE CUSTOMER UPDATE:

 

 

Image

by Chris Rudolph, Senior Technical Representative for Treehouse Software and Joseph Brady, Marketing and Documentation Manager for Treehouse Software

This is a follow-up to our recent Treehouse Software Blog entry “Treehouse Software is Setting Sights on Many New Data Replication Projects”, in which we described a typical customer visit to implement data replication.

Treehouse representatives were on-site at a state government agency to configure tcVISION and set up bulk transfer and change data capture, as well as train the State employees on using and managing tcVISION. In a subsequent discussion with our contacts at the site, they reported that their deadline for delivering a reporting database in Microsoft SQL Server replicated from 63 ADABAS files has been met. They also happily noted that by using tcVISION, the bulk transfer of 60 million ADABAS records into SQL Server completed in only 20 minutes.

We are very pleased to have yet another satisfied customer benefitting from one of Treehouse Software’s mature and proven enterprise software solutions.

tcVISION provides real-time data replication through change data capture, and allows easy and fast data migration for mainframe application modernization projects. Enterprises looking for a product that enables bi-directional heterogeneous data replication between mainframe, Linux, Unix, and Windows platforms need look no further than to tcVISION from Treehouse Software.

To learn more about tcVISION, or to request a demonstration, contact Treehouse Software today.

Treehouse Software is Setting Sights on Many New Data Replication Projects

Image

by Chris Rudolph, Senior Technical Representative for Treehouse Software and Joseph Brady, Marketing and Documentation Manager for Treehouse Software

Setting up a New Customer with tcVISION…

Treehouse Software representatives are busy these days, visiting customers and getting them started with our tcVISION product for complex data replication between mainframe, Linux, Unix, and Windows platforms.

As an example of a typical customer visit to implement data replication, we recently were on-site at a state government agency to configure tcVISION and set up bulk transfer and change data capture as well as train the State employees on using and managing tcVISION. Prior to our arrival, the customer had installed tcVISION and verified that its components were working correctly. After our three days on-site, the users were able to import their own ADABAS files, run scripts to perform bulk loads, and replicate transactions on the PLOG from ADABAS to SQL Server.

On the first day, we held a project kickoff meeting to discuss tcVISION’s architecture, the State’s ADABAS/NATURAL environment, and identified which ADABAS files should be replicated in the initial phase. We also presented a walkthrough of the product that covered metadata import, bulk transfer, and batch and real-time replication.

The group agreed that, for the purposes of the project at hand, real-time replication was not necessary immediately. It is possible that real-time replication and bi-directional replication may be needed in the future. Currently, monthly NATURAL extracts are used to transfer ADABAS data to SQL Server. The data is loaded into VARCHAR columns in the RDBMS, and then the data is copied to another set of tables. This process includes formatting dates, times, and numbers, but mostly ignores MUs and PEs (repeating fields in ADABAS). We showed how tcVISION can bulk transfer and replicate to a brand-new RDBMS schema created by tcVISION, as well as to an existing schema.

The setup process continued on the second day, and included creating two SQL Server databases for tcVISION, defining the connection to SQL Server, creating the tcVISION repository tables in one of the SQL Server databases, importing metadata from an ADABAS file into tcVISION, and creating the tables in SQL Server corresponding to that ADABAS file.

Next, we focused on populating these first tables. To do this, we ran the tcVISION bulk transfer script and then a control script to automatically execute the SQL Server bcp utility once the tcVISION Bulk Transfer completed.

We then imported another ADABAS file’s metadata into tcVISION. This second file contained several MUs, some of which we normalized, and others of which we denormalized on the parent table. After discussing the target structure, a bulk transfer on the ADABAS file was run. The ADABAS file contained over 100 fields and some 800,000 records. The bulk transfer ran in about 22 minutes, and the bcp into SQL Server ran for approximately two minutes. The customer’s existing NATURAL extract process takes a little under an hour for the same file, so tcVISION was able to perform the bulk transfer in about a quarter of the time.

As a last activity for the day, we demonstrated to the customer how to import an existing SQL Server table into tcVISION, and then map the table to an ADABAS file.

Image

The tcVISION “Control Board” is the user’s MS Windows GUI Interface, providing a single and central point of administration, featuring the ability to view and manage transaction logs from other systems; built-in Wizards for mapping and modeling metadata; script generation for bulk transfer and data replication; and monitoring of replication status, script output, and DBMS extensions.

For our final activities on the third day, we ran a few more bulk transfers, and the users focused on importing ADABAS metadata and creating tables in SQL Server. During this time, we set up a new control script that allowed bulk transfer of multiple ADABAS files, instead of having to process each ADABAS file with a different script. The control script reads a list of ADABAS files, and then runs a bulk transfer script for each ADABAS file. The tcVISION manager uses a class setting to limit the number of bulk transfers running in parallel.

We then verified that the control script had submitted the processing script for each ADABAS file, and we demonstrated how to use tcVISION’s scheduler to automatically submit the control script. Once scheduling was working, we then turned our attention to processing the PLOG for change data capture.

A processing script was set up to read the PLOG on the mainframe, and then update the newly-populated SQL Server tables. We explained how some JCL could be added to the to the PLOG archive copy job to automatically process the PLOG whenever it flips.

The users were very pleased with the progress made during the three-day visit, and are well on their way to having tcVISION automatically handle and simplify their ADABAS replication efforts.

To learn more, or to request a tcVISION demonstration, contact Treehouse Software today.