AWS Services Provide Advanced Monitoring and Analytics of tcVISION’s Mainframe CDC Processing

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

____AI_Data_Monitoring_And_Analytics

Many Treehouse Software mainframe modernization customers have requirements for continuous near-real-time replication of mainframe data in order to keep a copy of the data synchronized on the Cloud. These customers are using tcVISION from Treehouse Software for changed data capture (CDC) for this synchronization, which allows changes occurring in any mainframe application data to be tracked and captured, and then published to a variety of AWS targets, including Amazon Simple Storage Service (S3). Some of these customers are also now asking us to recommend the best Cloud-based tools and methods to monitor and gain insights to these complex data processes. Coincidentally, while working with a current tcVISION customer, our technicians are testing out two particularly good, fully managed AWS services that can work hand-in-hand to address this need:

Amazon Athena

Since tcVISION supports Amazon S3 as a target, customers modernizing their mainframe systems on AWS can use Amazon Athena for monitoring and analysis of CDC processing from an S3 bucket.

Amazon Athena is a serverless, interactive analytics service built on open-source frameworks, supporting open-table and file formats. Athena provides a simplified, flexible way to analyze data from an S3 Bucket, as well as many other data sources, including on-premises data sources or other Cloud systems. Athena is built on open-source Trino and Presto engines and Apache Spark frameworks, with no provisioning or configuration effort required.

Figure 1: Example of an Athena query showing bulk-load statistics per table

____01_Amazon_Athena_Query

Amazon QuickSight

____01_Amazon_QuickSight

Once Athena is setup for monitoring an S3 Bucket, users can easily view their CDC processing and analytics with Amazon QuickSight. QuickSight utilizes advanced machine learning-powered insights and intuitive dashboards, so end users can make the best and quickest data-driven business decisions.

Figure 2: Example of Amazon QuickSight monitoring the throughput of our data to Snowflake

____01_Amazon_QuickSight02

Figure 3: Example of Amazon QuickSight pie chart showing the resulting rows loaded for each Snowflake table:

____01_Amazon_QuickSight03

Figure 4: Example of Amazon QuickSight chart showing statistics for our data bulk-load into Snowflake:

____01_Amazon_QuickSight04

Figure 5: Example of Amazon QuickSight chart showing our load time into Snowflake per table:

____01_Amazon_QuickSight05

View the Amazon QuickSite video here…


__001_TSI_LOGO

Interested in seeing a live, online demo of tcVISION?

Just fill out the Treehouse Software tcVISION Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.


Treehouse Software Customer Success: BMF uses tcVISION for Real-Time Data Replication Between Mainframe Adabas and PostgreSQL

BMF_Building

The Bundesministerium der Finanzen (BMF) is Germany’s Ministry of Finance and establishes sustainable fiscal policy that ensures financial empowerment of the federal budget. From tax policy via development of federal budget, to regulation of national and international financial markets – for these and other fiscal and economic questions of principle, the BMF creates strategies and concepts, and implements them. The Federal Tax Administration is part of BMF, and controls not only the cross-border goods traffic, but acts against illegal employment and other crimes. The tax administration also imposes consumer taxes (e.g., energy and tobacco tax, car tax, etc.). Financial relations between federation, countries, and communities are also coordinated by BMF.

Department II (federal budget) is part of the German government in charge of establishing the budget and financial planning of the federation. Throughout the year, it monitors execution of the budget for eventual intervention (e.g., with a budget freeze, or supplementary budget). After closing the fiscal year, the budget and balance sheet will be presented. The budget is a supplement of the budget act, legally binding.

The central service organization of BMF is the Informationstechnikzentrum Bund – ITZBund (Information technic center).

BUSINESS BACKGROUND

Drawing up the budget is a yearly, highly time consuming, and formalized business process. All departments are involved in nearly every sub-process, and budgeting and financial planning is supported by the application, “Haushaltsaufstellung / Budgetgeneration”. Using the generated reports, various addressees/receivers are supported (e.g., German Federal Government, German Federal Parliament, Federal Council of Germany, finance department in BMF, the employees in the departments, and the public).

Technically, the budget plan of the federation is based on technologies, including the IBM Mainframe with z/OS running Adabas and Natural.

The challenge was to provide an environment for employees in all departments that enables them to do their work quickly, easily, and efficiently. In the BMF, users must have an editorless, end-user driven, and real-time creation of ready-to-print products. An informative description of the workflow is shown on the website of the BMF.

The federal budget is available as download, or one can directly navigate through the data using the online application.

BUSINESS ISSUE

Some time ago, BMF decided to re-engineer the application for budget planning and port it to Open Source. To guarantee a seamless transition, the first step is propagation of data out of Adabas on z/OS to PostgreSQL, concluding with permanent synchronization.

The difficulties of this task are the complexities of setting up data definitions for the data structures in Natural and the propagation of data from Adabas on z/OS to PostgreSQL.

TECHNOLOGY SOLUTION: tcVISION

____Adabas_to_PostgreSQL_Diagram

After an analysis of the project, Treehouse Software proposed creating an extension to tcVISION’s change data capture (CDC) functionality for integration, so that tcVISION could enable BMF to continue using the implemented data definitions in a format suitable for the RDBMS.

The extension was developed within a few days, and a two-day on premise test demonstrated the solution fit the requirements of BMF.

BMF can now provide its data definitions from Natural LDA to the extension of tcVISION, and after the transformation, onto the PostgreSQL load process for processing. Another advantage of the tcVISION solution is that when needed, other targets can be integrated for propagation of data from the mainframe (e.g., Kafka, which BMF indicated is a future target environment).

Additionally, bi-directional propagation can be added in budget planning when BMF is ready.

Data structures are held in LDA, because this provides the advantages of higher flexibility in development and the adaption of new requirements to the data definitions. If definitions would have to be ported manually, in part, to PostgreSQL, it would have been a much bigger and error-prone effort.

Subsequent changes to Adabas structures can now use tcVISION’s newly developed extension to easily regenerate and load the correct definitions to the RDBMS, and tcVISION completely covers the customer’s requirements for special usage of *PEs and *MUs.

After thorough preparation and extensive testing, the solution was released to selected users first, then made available to all users.

* PEs and MUs are special Adabas formats for definition of tables. PE = Periodic Group, MU = Multiple Value Field.


__tsi_logo_400x200

Contact Treehouse Software for a Demo Today…

No matter where you want your mainframe data to go – the cloud, open systems, or any LUW target – tcVISION from Treehouse Software is your answer.

Just fill out the Treehouse Software Product Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.


Further reading: Treehouse Software Customer Success – ETS: tcVISION for Real-Time Synchronization Between Mainframe IDMS and AWS RDS for PostgreSQL

Providing a High Availability Framework for Mainframe-to-AWS Data Replication

by Dan Vimont, Cloud Solutions Architect at Treehouse Software, Inc.

tcV_HA_on_AWS

Treehouse Software customers are using tcVISION to enable mission-critical mainframe-to-AWS data replication pipelines.  Some of these production pipelines are providing vital near-real-time synchronization between source and target, and thus can’t afford any significant downtime in the event of failure.  So it’s only natural that a number of our customers have been asking for advice in setting up a high availability configuration for their tcVISION components that run on AWS EC2 instances.  The High Availability Framework discussed here provides for a Failover EC2 instance to automatically pick up tcVISION processing should the Primary instance (running in another Availability Zone) go down.

The Core Components:  Primary Instance & Failover Instance

The core components of a tcVISION high availability framework consist of two EC2 instances running in different Availability Zones:  a Primary EC2 instance and a Failover EC2 instance.  Both identically-configured EC2 instances are attached to a shared working-storage file system (either an EFS or FSx volume), which allows the Failover instance to seamlessly and quickly pick up tcVISION processing should the Primary instance suddenly become unavailable.

HA1

Use a Step Function to Automate the Failover Process

In the event of failure of the Primary instance, the recommended framework calls for automatic triggering of a Step Function for reliable failover processing, with steps that include the following:

  • verify that the Primary instance is unavailable (The tcVISION service cannot be active on both instances simultaneously, so this verification is vital.)
  • redirect all network traffic from the Primary instance to the Failover instance (via Route 53)
  • start tcVISION processing on the Failover instance

HA2

When Ready, Use a Step Function to Automate the Restoration Process

After operations personnel have completed recovery of the Primary EC2 instance, another Step Function may be manually triggered to reliably transfer tcVISION processing back to the Primary instance.

HA3.jp

Many More Details are Available Upon Request to Treehouse Customers

Full details regarding our recommended High Availability Framework for tcVISION are available upon request to Treehouse customers.  AWS services utilized in the complete recommended framework include Step Functions, Lambda Functions, EventBridge rules, CloudWatch alarms, SNS topics, a Route 53 Private Hosted Zone, and more.  The following diagram is a partial visual inventory of the recommended framework components.

HA5

Interested in seeing a live, online demo of tcVISION?

Just fill out the Treehouse Software tcVISION Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.


__001_TSI_LOGO

How to Synchronize Data in Real Time Between the Mainframe and AWS with Treehouse Software’s Enterprise CDC Tool

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

Bidirectional_Data_Replication

Many mainframe integration scenarios require continuous near-real-time replication of relational data to keep a copy of the data synched in the Cloud. Change Data Capture (CDC) is used for this near-real-time transactional replication by capturing change log activity to drive changes in the target dataset.

Just what is CDC anyway?

Simply put, and in relation to Mainframe-to-Cloud and open systems data replication, CDC is the use of processes to identify when data has been changed in a source system, so the replicated upstream or downstream (depending on how you look at it) target can be kept in sync with the changes.

In a recent AWS Architecture Blog, readers learn about integration using mainframe data to build Cloud native services with AWS, including transactional replication-based integration via CDC.

____AWS_Mainframe_CDC_Diagram

As mentioned in the blog, AWS Partner CDC Tools are available for connecting data center mainframes to the various data targets, and Treehouse Software’s tcVISION is one of those tools available in the AWS Marketplace.

tcVISION allows changes occurring in any mainframe application data to be tracked and captured, and then published to a variety of target AWS databases and applications. tcVISION provides an easy and fast approach for Hybrid Cloud projects, enabling real-time and bi-directional data replication between the hardware and AWS.

Example of Db2-to-AWS CDC using tcVISION Mainframe Manager:

tcVISION_Db2_To_AWS_CDC

tcVISION supports several CDC methods available, depending on each customer’s use case:

Bulk Transfer

  • Efficient transfer of entire databases
  • Analysis for data consistency (verification)
  • Initial load (ETL) and periodic mass data transfer
  • One-step data transfer

Log Processing

  • Transfer of changed data near-realtime or scheduled time frame
  • Reads both active logs and archived logs

Batch Compare

  • Comparison of data snapshots using checksums
  • Efficient transfer of changed data since last processing
  • Flexible processing options (SORT etc.)
  • Automatic creation of deltas by tcVISION

DBMS Extension

  • Real-time capture of changed data directly from the DBMS
  • Secure data storage even across DBMS restart
  • Flexible propagation methods

Interested in seeing a live, online demo of tcVISION CDC?

Just fill out the Treehouse Software tcVISION Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.


__001_TSI_LOGO

Treehouse Software Customer Case Study: A State Government Agency’s Real-time Data Synchronization Between IBM Mainframe Adabas and AWS

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

Mainframe_to_AWS_Graphic

Software AG’s Adabas is a mainframe database that is still heavily used by government sites throughout the U.S. and the world, and this blog focuses on a current Treehouse Software customer – a U.S. State Government Agency that uses Adabas on their mainframe system.

Business Issue

The Agency’s modernization team was looking for a Change Data Capture (CDC) technology solution that enables them to synchronize their mainframe Adabas data on AWS, particularly an Amazon RDS. As with most Treehouse customers, the State’s mainframe contains vital data that must always be highly available, so rather than attempting a complete migration from the mainframe, the modernization teams decided to implement a multi-year data replication plan. This allows the mainframe legacy teams to maintain existing critical applications, while the modernization team develops new applications on AWS.

After researching various technologies, the Agency discovered tcVISION on the AWS Parter Network Blog and contacted Treehouse Software to discuss their project and to see a demonstration of Mainframe-to-AWS data replication.

Addressing the Uniqueness of Adabas

Having specialized in tools and services complementary to Adabas/Natural applications since 1982, Treehouse Software has successfully encountered and addressed many unique scenarios within the Adabas environment. The Treehouse technical team documented three primary issues with Adabas/Natural that the Agency needed to consider when they began planning data replication on AWS:

  1. Adabas has no concept of “transaction isolation”, in that a program may read a record that another program has updated, in its updated state, even though the update has not been committed.  This means that programmatically reading a live Adabas database—one that is available to update users—will almost inevitably lead to erroneous extraction of data.  Record modifications (updates, inserts and deletes) that are extracted, and subsequently backed out, will be represented incorrectly—or not at all—in the target. Because of this, at Treehouse we say “the only safe data source is a static data source”—not the live database.
  2. Many legacy Adabas applications make use of “record typing”, i.e., multiple logical tables stored in a single Adabas file.  Often, each must be extracted to a separate table in the target RDBMS.  The classic example is that of the “code-lookup file”.  Most shops have a single file containing state codes, employee codes, product-type codes, etc.  Records belonging to a given “code table” may be distinguished by the presence of a value in a particular index (descriptor or superdescriptor in ADABAS parlance), or by a range of specific values.  Thus, the extraction process must be able to dynamically assign data content from a given record to different target tables depending on the data content itself.
  3. Adabas is most often used in conjunction with Software AG’s Natural 4GL, and “conveniently” provides for unique datatypes (“D” and “T”) that appear to be merely packed-decimal integers on the surface, but that represent date or date-time values when interpreted using Software AG’s proprietary Natural-oriented algorithm. The most appropriate way to migrate such datatypes is to recognize them and map them to the corresponding native RDBMS datatype (e.g., Oracle DATE) in conjunction with a transformation that decodes the Natural value and formats it to match the target datatype.

The tcVISION Technology Solution...

Adabas_To_AWS

After technical discussions and a successful proof of concept (POC) that proved out a set of use cases, all teams at the Agency determined that tcVISION real-time mainframe data replication capabilities were the perfect fit for meeting their goals.

tcVISION‘s modeling and mapping facilities are utilized to view and capture logical Adabas structures, as documented in Software AG’s PREDICT data dictionary, as well as physical structures as described in Adabas Field Definition Tables (FDTs).  Given that PREDICT is a “passive” data dictionary (there is no requirement that the logical and physical representations agree), it was necessary to scrutinize both to ensure that the source structures were accurately modeled.

Furthermore, tcVISION generates appropriate mappings and transformations for converting Adabas datatypes and structures to corresponding target datatypes and structures, including automatic handling of the proprietary “D” and “T” source datatypes.

The teams examined the three ways that tcVISION can access Adabas data:

  1. ETL – read the active database nucleus
  2. ETL – read datasets containing unloaded Adabas files created by the ADAULD utility
  3. CDC – read the active and archived PLOGs datasets

It was decided to access the data by reading the active and archived PLOGs datasets. The schema, mappings, and transformations from the metadata import were tailored to the customer’s specific requirements.  It is also now possible to import an existing RDBMS schema and retrofit it, via drag-and-drop in tcVISION, to the source Adabas elements.

Additionally, the Agency’s teams are very pleased with tcVISION‘s minimal usage of mainframe resources. The product’s “staged processing” methodology accomplishes this, whereby the only processing occurring on the mainframe is the capture of changes from Adabas PLOGs. The bulk of the processing occurs on the AWS side, minimizing tcVISION’s footprint on the mainframe as seen in this diagram:

tcVISION_Staged_Processing

The user defines on which platform stage their processing should be done. Do as little as possible on the mainframe: Stage 0 – capture data and send data (internal format) to target, and process data in Stages 1 – 3 in AWS.

Customer Outcome

All requirements were met by tcVISION, which led to a successful project implementation.


__001_TSI_LOGO
Contact Treehouse Software for a tcVISION Demo Today…

No matter where you want your mainframe data to go – the Cloud, open systems, or any LUW target – tcVISION from Treehouse Software is your answer.

Just fill out the Treehouse Software tcVISION Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.


Further reading:

Many more mainframe data migration and replication customer case studies can be read on the Treehouse Software Website.

Treehouse Software’s Differentiator: Enterprise Mainframe Expertise Since 1982

by Joseph Brady, Director of Business Development and Cloud Alliance Lead at Treehouse Software, Inc. 

Treehoue_Mainframe_Experience

This blog explores Treehouse Software‘s decades worth of experience in helping mainframe customers with innovative tools, services, and training. 

When Treehouse Software began in 1982, the business focused on software that was complementary to the Software AG mainframe product line (Adabas database management system and Natural programming language) in the areas of security, control, auditing, performance enhancement, etc.

In more recent years, Treehouse Software has become a global leader in providing solutions for real-time and bi-directional data replication between a variety of mainframe and non-mainframe sources, including (Mainframe): Adabas, Db2, VSAM, IMS, CA Datacom, and CA IDMS; and (Non-mainframe): Amazon Web Services (AWS), Google Cloud, Microsoft Azure, PostgreSQL, Kafka, Oracle, Microsoft SQL Server, IBM Db2 LUW and Db2 BLU, IBM Informix, MongoDB, Hadoop, and many more. Here is our list of supported data sources and targets.

Decades Worth of Mainframe Knowledge…

When asked by prospective customers, “What are your primary differentiators?”, we can be tempted to first talk about superior product features and capabilities, but in addition to our exceptional products, it is Treehouse Software’s depth of knowledge and experience in the mainframe world that is the real game changer.

Most mainframe users face critical data management challenges due to the complexity and proprietary nature of deeply entrenched databases on the platform. Our extensive experience, deep knowledge, and wide-ranging capabilities in mainframe technologies make the company a valued partner for third-party solution providers and a trusted advisor to customers.

Treehouse Software’s visionary leadership in this market has included pioneering Adabas-to-RDBMS ETL and CDC with tRelational/DPS in the mid-1990s.  Today, Treehouse Software stands alone in its product maturity, and capability, including expanded capabilities with the tcVISION product, which enables migration and synchronization of virtually any mainframe or non-mainframe database or data source.

Despite the rapid pace of change in the IT landscape, Treehouse Software’s customer base can be assured that there remains a strong commitment to providing continued support and upgrades for the product suite.

Treehouse Software provides tools and expertise for the riskiest and most-often overlooked parts of modernization and integration projects – data migration and integration.   Tapping into the vast experience of Treehouse’s technicians, and using proven products and services eliminates reliance on end-customer programming staff to write and maintain data extracts and middleware. Treehouse Software’s know-how reduces cost and mitigates risk in legacy modernization initiatives, where data migration and integration complexity is often underestimated, yet critical to success.

Our Mainframe Experts are Our Best Assets

We are fortunate to have a staff with a wealth of knowledge and skills that span not only Mainframe, but Cloud, LUW, and Open Systems technologies. Whether a customer wants to move data from their mainframe platform to other on-premises open systems or LUW databases, or to the Cloud (e.g., AWS, Google Cloud, Azure, etc.), Treehouse Software has the technical expertise and support needed to ensure successful project completion.

Treehouse Software‘s technicians have installed products and trained end-users in some of the largest mainframe sites around the world.  Mature, robust, and reliable, these products are also backed by our highly-rated 24X7 technical support.

The Treehouse Team Approach

TechniciansConnectivity

Treehouse Software has proven its ability to partner and work effectively as part of a larger team to solve client problems.  AWS, Google, Microsoft, Deloitte, Accenture, and other large vendors have selected our technology, services, and training for their mainframe data migration and application modernization practices.


__tsi_logo_400x200

Contact Treehouse Software for a Product Demo Today…

Just fill out the Treehouse Software Product Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online product demonstration.

The New 2020 IBM Z Solutions Directory Features a Wide Selection of Mainframe Tools and Services, Including Treehouse Software’s Data Integration Products

by Joseph Brady, Director of Business Development / Cloud Alliance Lead at Treehouse Software, Inc. 

2020_IBM_Systems_Solutions_Directory

Enterprises’ investments in IBM Z mainframe technology is significant, and the IBM Z Solutions Directory showcases some of the best hardware, accessories, software products, and services available to help customers maintain and expand the platform.

Treehouse Software encourages mainframe customers to explore the IBM Z Solutions Directory, where they will find neatly organized listings that help them quickly find the products and services they need. We are very pleased to have our products included, side-by-side, with most of the top mainframe solutions in the world in this valuable guide. 


__TSI_LOGO

More About Treehouse Software’s Mature and Proven Mainframe Enterprise Transformation Products…

Treehouse_Products_Diagram_General_Cloud.jpg

Treehouse Software has been developing, marketing, selling, and supporting mainframe software since 1982, and we are committed to helping customers easily access some of the most advanced Cloud and open systems technologies in the world, while maintaining their valuable legacy environments. 

Treehouse Software’s visionary leadership in the mainframe market has included pioneering Adabas-to-RDBMS ETL and CDC with tRelational/DPS in the mid-1990s.  Today, with the tcVISION product, Treehouse Software is a global leader in providing real-time and bi-directional data replication between a variety of mainframe and non-mainframe sources, including (Mainframe): VSAM, IMS, Db2, CA Datacom, Adabas, and CA IDMS; and (Non-mainframe): Amazon Web Services (AWS), Microsoft Azure, Google Cloud, Oracle Cloud, PostgreSQL, Microsoft SQL Server, IBM Db2 LUW and Db2 BLU, IBM Informix, Kafka, MongoDB, MariaDB, Hadoop, SAP Hana, and many more.

Despite the rapid pace of change in the IT landscape, Treehouse Software’s customer base can be assured that there remains a strong commitment to providing continued support and upgrades for the product suite.

Contact Treehouse Software for a Product Demo Today…

Just fill out the Treehouse Software Product Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online product demonstration.

 

Treehouse Software’s Differentiator: We’ve Been Helping Enterprise Mainframe Sites Since 1982

by Joseph Brady, Director of Business Development / AWS and Cloud Alliance Lead at Treehouse Software, Inc. 

Treehoue_Mainframe_Experience

This blog explores Treehouse Software‘s decades worth of experience in helping mainframe customers with innovative tools, services, and training. 

When Treehouse Software began in 1982, the business focused on software that was complementary to the Software AG mainframe product line (Adabas database management system and Natural programming language) in the areas of security, control, auditing, performance enhancement, etc.

In more recent years, Treehouse Software has become a global leader in providing solutions for real-time and bi-directional data replication between a variety of mainframe and non-mainframe sources, including (Mainframe): VSAM, IMS, Db2, CA Datacom, Adabas, and CA IDMS; and (Non-mainframe): Amazon Web Services (AWS), PostgreSQL, Oracle, Microsoft Azure, Microsoft SQL Server, IBM Db2 LUW and Db2 BLU, IBM Informix, Kafka, MongoDB, Hadoop, SAP Hana, and many more. Here is our list of supported data sources and targets.

Decades Worth of Mainframe Knowledge…

When asked by prospective customers, “What are your primary differentiators?”, we can be tempted to first talk about superior product features and capabilities, but in addition to our exceptional products, it is Treehouse Software’s depth of knowledge and experience in the mainframe world that is the real game changer.

Most mainframe users face critical data management challenges due to the complexity and proprietary nature of deeply entrenched databases on the platform. Our extensive experience, deep knowledge, and wide-ranging capabilities in mainframe technologies make the company a valued partner for third-party solution providers and a trusted advisor to customers.

Treehouse Software’s visionary leadership in this market has included pioneering Adabas-to-RDBMS ETL and CDC with tRelational/DPS in the mid-1990s.  Today, Treehouse Software stands alone in its product maturity, and capability, including expanded capabilities with the tcVISION product, which enables migration and synchronization of virtually any mainframe or non-mainframe database or data source.

Despite the rapid pace of change in the IT landscape, Treehouse Software’s customer base can be assured that there remains a strong commitment to providing continued support and upgrades for the product suite.

Treehouse Software provides tools and expertise for the riskiest and most-often overlooked parts of modernization and integration projects – data migration and integration.   Tapping into the vast experience of Treehouse’s technicians, and using proven products and services eliminates reliance on end-customer programming staff to write and maintain data extracts and middleware. Treehouse Software’s know-how reduces cost and mitigates risk in legacy modernization initiatives, where data migration and integration complexity is often underestimated, yet critical to success.

Our Mainframe Experts are Our Best Assets

We are fortunate to have a staff with a wealth of knowledge and skills that span not only Mainframe, but Cloud, LUW, and Open Systems technologies. Whether a customer wants to move data from their mainframe platform to other on-premises open systems or LUW databases, or to the Cloud (e.g., AWS, Azure, etc.), Treehouse Software has the technical expertise and support needed to ensure successful project completion.

Treehouse Software‘s technicians have installed products and trained end-users in some of the largest mainframe sites around the world.  Mature, robust, and reliable, these products are also backed by our highly-rated 24X7 technical support.

The Treehouse Team Approach

TechniciansConnectivity

Treehouse Software has proven its ability to partner and work effectively as part of a larger team to solve client problems.  AWS, Microsoft, Oracle, Accenture, and other large vendors have selected our technology, services, and training for their mainframe data migration and application modernization practices.


__tsi_logo_400x200

Contact Treehouse Software for a Product Demo Today…

Just fill out the Treehouse Software Product Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online product demonstration.

Treehouse Software TREETIP: Data Replication for a Mainframe Database that has no Primary Key, using tcVISION

by Joseph Brady, Director of Business Development / AWS and Cloud Alliance Lead at Treehouse Software, Inc. and Chris Rudolph, Senior Technical Representative at Treehouse Software, Inc.

This blog takes a look at tcVISION’s support for a mainframe database that has no primary key at the source (a mainframe primary key is a column, or set of columns that uniquely identifies one row of a table). 

In situations where the source database does not contain any unique values, Treehouse technicians discuss with the customer how the application currently works and their expectations for how the data should be treated when planning replication/migration to a new target environment. Depending on the application, Journal Replication or Data Warehouse Replication may be a better fit than a “normal” RDBMS table definition.

tcVISION‘s Key ID management can be used to create a column to use as the key on the target table. Another option would be to use a SQL Lookup to query another table. Either way, target column(s) that contain unique values will need to be identified. Once these columns are identified, they can be marked with the key identifier, using the tcVISION repository editor, or Key ID Management can be used to create a new unique column, then those columns are used within the wizard.

For example, this source table does not contain a primary key:

___tcV_NoKey01

Analysis of the target data shows the columns FIRST_NAME, LAST_NAME and MIDDLE_NAME together will provide a unique value. These fields can be marked in the tcVISION repository as being members of a key:

___tcV_NoKey02

As mentioned earlier, the tcVISION Key ID Management wizard can also be used to create a new unique target column:

___tcV_NoKey03

Specify the key name:

___tcV_NoKey04

Specify the key value creation:

___tcV_NoKey05

Various options are available: 

___tcV_NoKey06


__tsi_logo_400x200

Contact Treehouse Software for a Demo Today…

tcVISION_Overall_Diagram

No matter where you want your mainframe data to go – the cloud, open systems, or any LUW target – tcVISION from Treehouse Software is your answer.

Just fill out the Treehouse Software Product Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.


Further reading: tcVISION Mainframe data replication is featured on the AWS Partner Network Blog…

___tcVISION_AWS_HA_Architecture

AWS recently published a blog about tcVISION’s Mainframe data replication capabilities, including a technical overview, security, high availability, scalability, and a step-by-step example of the creation of tcVISION metadata and scripts for replicating mainframe Db2 z/OS data to Amazon Aurora. Read the blog here: AWS Partner Network (APN) Blog: Real-Time Mainframe Data Replication to AWS with tcVISION from Treehouse Software.

Treehouse Software Customer Success: Data Replication from Mainframe Adabas to PostgreSQL using tcVISION

BMF_Building

The Bundesministerium der Finanzen (BMF) is Germany’s Ministry of Finance and establishes sustainable fiscal policy that ensures financial empowerment of the federal budget. From tax policy via development of federal budget, to regulation of national and international financial markets – for these and other fiscal and economic questions of principle, the BMF creates strategies and concepts, and implements them. The Federal Tax Administration is part of BMF, and controls not only the cross-border goods traffic, but acts against illegal employment and other crimes. The tax administration also imposes consumer taxes (e.g., energy and tobacco tax, car tax, etc.). Financial relations between federation, countries, and communities are also coordinated by BMF.

Department II (federal budget) is part of the German government in charge of establishing the budget and financial planning of the federation. Throughout the year, it monitors execution of the budget for eventual intervention (e.g., with a budget freeze, or supplementary budget). After closing the fiscal year, the budget and balance sheet will be presented. The budget is a supplement of the budget act, legally binding.

The central service organization of BMF is the Informationstechnikzentrum Bund – ITZBund (Information technic center).

BUSINESS BACKGROUND

Drawing up the budget is a yearly, highly time consuming, and formalized business process. All departments are involved in nearly every sub-process, and budgeting and financial planning is supported by the application, “Haushaltsaufstellung / Budgetgeneration”. Using the generated reports, various addressees/receivers are supported (e.g., German Federal Government, German Federal Parliament, Federal Council of Germany, finance department in BMF, the employees in the departments, and the public).

Technically, the budget plan of the federation is based on technologies, including the IBM Mainframe with z/OS running Adabas and Natural.

The challenge was to provide an environment for employees in all departments that enables them to do their work quickly, easily, and efficiently. In the BMF, users must have an editorless, end-user driven, and real-time creation of ready-to-print products.
An informative description of the workflow is shown on the website of the BMF.

The federal budget is available as download, or one can directly navigate through the data using the online application.

BUSINESS ISSUE

Some time ago, BMF decided to re-engineer the application for budget planning and port it to Open Source. To guarantee a seamless transition, the first step is propagation of data out of Adabas on z/OS to PostgreSQL, concluding with permanent synchronization.
The difficulties of this task are the complexities of setting up data definitions for the data structures in Natural and the propagation of data from Adabas on z/OS to PostgreSQL.

TECHNOLOGY SOLUTION: tcVISION

tcVISION_Overall_Diagram

After an analysis of the project, Treehouse Software proposed creating an extension to tcVISION’s change data capture (CDC) functionality for integration, so that tcVISION could enable BMF to continue using the implemented data definitions in a format suitable for the RDBMS.

The extension was developed within a few days, and a two-day on premise test demonstrated the solution fit the requirements of BMF.

BMF can now provide its data definitions from Natural LDA to the extension of tcVISION, and after the transformation, onto the PostgreSQL load process for processing. Another advantage of the tcVISION solution is that when needed, other targets can be integrated for propagation of data from the mainframe (e.g., Kafka, which BMF indicated is a future target environment).

Additionally, bi-directional propagation can be added in budget planning when BMF is ready.

Data structures are held in LDA, because this provides the advantages of higher flexibility in development and the adaption of new requirements to the data definitions. If definitions would have to be ported manually, in part, to PostGreSQL, it would have been a much bigger and error-prone effort.

Subsequent changes to Adabas structures can now use tcVISION’s newly developed extension to easily regenerate and load the correct definitions to the RDBMS, and tcVISION completely covers the customer’s requirements for special usage of *PEs and *MUs.

After thorough preparation and extensive testing, the solution was released to selected users first, then made available to all users.

* PEs and MUs are special Adabas formats for definition of tables. PE = Periodic Group, MU = Multiple Value Field.


__tsi_logo_400x200

Contact Treehouse Software for a Demo Today…

No matter where you want your mainframe data to go – the cloud, open systems, or any LUW target – tcVISION from Treehouse Software is your answer.

Just fill out the Treehouse Software Product Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.


Further reading: tcVISION Mainframe data replication is featured on the AWS Partner Network Blog…

___tcVISION_AWS_HA_Architecture

AWS recently published a blog about tcVISION’s Mainframe data replication capabilities, including a technical overview, security, high availability, scalability, and a step-by-step example of the creation of tcVISION metadata and scripts for replicating mainframe Db2 z/OS data to Amazon Aurora. Read the blog here: AWS Partner Network (APN) Blog: Real-Time Mainframe Data Replication to AWS with tcVISION from Treehouse Software.