by Joseph Brady, Director of Business Development / Cloud Alliance Lead at Treehouse Software, Inc.
Many government agencies have large volumes of mission critical and historical data stored in a variety of legacy mainframe databases (for this blog, we are focusing on Adabas). These government agencies combine a broad range human service programs, including employment assistance and job training, child and adult protection, child support enforcement, cash assistance, and services for the developmentally disabled, all of which constantly require accurate, up-to-date, and secure data.
To support the needs of clients, government agencies are generally broken into multiple divisions, including the division that Treehouse Software works with the most — Technology Services. The Technology Services divisions provide technical and systems services for the development, maintenance, and enhancement of automated business systems. They also ensure that the production and sub-production databases are running smoothly and efficiently by performing necessary maintenance of the databases.
Rapidly changing national health and economic conditions are making fast access to the most current information more important than ever for government agencies and the public. As a result, a top priority for Technology Services divisions is modernizing critical data residing on long-standing mainframe databases. Unlocking the value of this important data can be difficult, because the data can be utilized by numerous interlinked and dependent programs that have been in place for many years, and sometimes decades.
Many Treehouse government customers are now looking for modernization solutions that allow their legacy mainframe environments to continue, while replicating data in real time on highly available Cloud-based platforms, such as AWS, Google Cloud, and Microsoft Azure. tcVISION from Treehouse Software allows a “data-first” approach, whereby immediate data replication to the Cloud helps government agencies begin strategies to meet spikes in demand for vital information, especially in times of crisis.
Just what is it about Adabas?
Adabas is a mainframe database that is still heavily used by government sites throughout the U.S. and the world. Having specialized in tools and services complementary to Adabas/Natural applications since 1982, Treehouse has successfully encountered and addressed many unique issues within the Adabas environment. This excerpt from a Treehouse technical document outlines three primary issues with Adabas/Natural that must be considered:
- Adabas has no concept of “transaction isolation”, in that a program may read a record that another program has updated, in its updated state, even though the update has not been committed. This means that programmatically reading a live Adabas database—one that is available to update users—will almost inevitably lead to erroneous extraction of data. Record modifications (updates, inserts and deletes) that are extracted, and subsequently backed out, will be represented incorrectly—or not at all—in the target. Because of this, at Treehouse we say “the only safe data source is a static data source”—not the live database.
- Many legacy Adabas applications make use of “record typing”, i.e., multiple logical tables stored in a single Adabas file. Often, each must be extracted to a separate table in the target RDBMS. The classic example is that of the “code-lookup file”. Most shops have a single file containing state codes, employee codes, product-type codes, etc. Records belonging to a given “code table” may be distinguished by the presence of a value in a particular index (descriptor or superdescriptor in ADABAS parlance), or by a range of specific values. Thus, the extraction process must be able to dynamically assign data content from a given record to different target tables depending on the data content itself.
- Adabas is most often used in conjunction with Software AG’s Natural 4GL, and “conveniently” provides for unique datatypes (“D” and “T”) that appear to be merely packed-decimal integers on the surface, but that represent date or date-time values when interpreted using Software AG’s proprietary Natural-oriented algorithm. The most appropriate way to migrate such datatypes is to recognize them and map them to the corresponding native RDBMS datatype (e.g., Oracle DATE) in conjunction with a transformation that decodes the Natural value and formats it to match the target datatype.
For discovery and analysis of legacy source data structures, tcVISION’s modeling and mapping facilities view and capture logical Adabas structures, as documented in Software AG’s PREDICT data dictionary, as well as physical structures as described in Adabas Field Definition Tables (FDTs). Note that PREDICT is a “passive” data dictionary—there is no requirement that the logical and physical representations agree, so it is necessary to scrutinize both to ensure that the source structures are accurately modeled.
Furthermore, tcVISION generates specification and implementation of appropriate mappings and transformations for converting Adabas datatypes and structures to corresponding RDBMS datatypes and structures, including automatic handling of the proprietary “D” and “T” source datatypes.
There are three ways tcVISION can access Adabas data:
- ETL – read the active database nucleus
- ETL – read datasets containing unloaded Adabas files created by the ADAULD utility
- CDC – read the active and archived PLOGs datasets
It is important to note that the schema, mappings and transformations that result from metadata import can be tailored to any specific requirements after the fact. It is even possible to import an existing RDBMS schema and retrofit it, via drag-and-drop, to the source Adabas elements.
Using the tcVISION Control Board, a Windows GUI interface, the most complex transformations can be specified. Source fields can be combined into a single column, decomposed into separate columns, and be subject to calculations, database lookups, string operations and even programmatic manipulation. Furthermore, mapping rules can be implemented to specify that data content from a source Adabas record be mapped to one or more target RDBMS tables—each with its own different structure, as desired—based on the data content itself. Target tables can even be populated from more than one source file.
tcVISION supports complex replication scenarios…
tcVISION GUI Control Board functions as a central point of administration…
Automatically apply target schema within the Control Board…
It is impossible to discuss all the features and capabilities of tcVISION within a high-level overview. Given the maturity, wealth of functionality and relative low cost of tcVISION, as compared to the effort, complexity and risk entailed in a “Do-It-Yourself”, solution there is no reason why a legacy renewal project should run aground on data migration.
tcVISION’s minimal footprint on the mainframe…
Customers are very happy with tcVISION‘s “staged processing” methodology, where the only processing occurring on the mainframe was the capture of changes from Adabas PLOGs. The bulk of the processing occurs on the target platform, minimizing tcVISION’s footprint on the mainframe as seen in this diagram…
The user defines on which platform stage their processing should be done. Do as little as possible on the mainframe: Stage 0 – capture data and send data (internal format) to target, and process data up to Stage 3 in receiving environment.
Moving forward with tcVISION…
Treehouse Software has been helping local, state, and federal government agencies with Adabas and Natural in the areas of data migration, security, control, auditing, performance enhancement, etc. for decades. Over the years, Treehouse has expanded its capabilities to address new requirements for modernizing legacy mainframe Adabas databases on various Cloud platforms. By using Treehouse Software’s tcVISION Mainframe-to-Cloud data replication product, our government customers are able to immediately utilize some of the most advanced Cloud tools and services in the world.
tcVISION enables government agencies to synchronize mainframe Adabas data with various highly available and secure Cloud databases, data warehouses. etc.. Additionally, bi-directional, real-time data synchronization will enable changes on either platform to be reflected on the other platform (e.g., a change to a PostgreSQL table is reflected back on the mainframe database). This allows governments to modernize applications on Cloud platforms without disrupting the existing critical work on the legacy system, and modern tools can now be used in the new environment, greatly enhancing agility.
Replicating mainframe data on the Cloud can happen within days during a tcVISION Proof of Concept (POC)…
An online tcVISION POC is approximately 10 business days, with the customer providing use case and goals for the POC. A Treehouse Software consultant will assist in downloading and installing tcVISION and conducting a limited-scope implementation of a tcVISION application. This application uses customer data and executes on customer facilities, usually in a non-production environment. A document is provided beforehand that outlines the requirements, use cases, and agenda for the POC.
By the end of the 10-day POC, customers can begin replicating mainframe data to their Cloud target database. It can happen that fast!
Further Reading…
Treehouse Software is an AWS Technology Partner, and the AWS Partner Network published a blog about tcVISION, our Mainframe-to-Cloud data replication product, which describes how tcVISION allows legacy mainframe environments to continue, while replicating data on highly available and secure Cloud platforms:
Contact Treehouse Software for a tcVISION Demo Today…
Simply fill out our Product Demonstration Request Form and a Treehouse representative will be contacting you to set up a time for your requested demonstration.
Treehouse Software's Blog 