Providing a High Availability Framework for Mainframe-to-AWS Data Replication

by Dan Vimont, Cloud Solutions Architect at Treehouse Software, Inc.

tcV_HA_on_AWS

Treehouse Software customers are using tcVISION to enable mission-critical mainframe-to-AWS data replication pipelines.  Some of these production pipelines are providing vital near-real-time synchronization between source and target, and thus can’t afford any significant downtime in the event of failure.  So it’s only natural that a number of our customers have been asking for advice in setting up a high availability configuration for their tcVISION components that run on AWS EC2 instances.  The High Availability Framework discussed here provides for a Failover EC2 instance to automatically pick up tcVISION processing should the Primary instance (running in another Availability Zone) go down.

The Core Components:  Primary Instance & Failover Instance

The core components of a tcVISION high availability framework consist of two EC2 instances running in different Availability Zones:  a Primary EC2 instance and a Failover EC2 instance.  Both identically-configured EC2 instances are attached to a shared working-storage file system (either an EFS or FSx volume), which allows the Failover instance to seamlessly and quickly pick up tcVISION processing should the Primary instance suddenly become unavailable.

HA1

Use a Step Function to Automate the Failover Process

In the event of failure of the Primary instance, the recommended framework calls for automatic triggering of a Step Function for reliable failover processing, with steps that include the following:

  • verify that the Primary instance is unavailable (The tcVISION service cannot be active on both instances simultaneously, so this verification is vital.)
  • redirect all network traffic from the Primary instance to the Failover instance (via Route 53)
  • start tcVISION processing on the Failover instance

HA2

When Ready, Use a Step Function to Automate the Restoration Process

After operations personnel have completed recovery of the Primary EC2 instance, another Step Function may be manually triggered to reliably transfer tcVISION processing back to the Primary instance.

HA3.jp

Many More Details are Available Upon Request to Treehouse Customers

Full details regarding our recommended High Availability Framework for tcVISION are available upon request to Treehouse customers.  AWS services utilized in the complete recommended framework include Step Functions, Lambda Functions, EventBridge rules, CloudWatch alarms, SNS topics, a Route 53 Private Hosted Zone, and more.  The following diagram is a partial visual inventory of the recommended framework components.

HA5

Interested in seeing a live, online demo of tcVISION?

Just fill out the Treehouse Software tcVISION Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.


__001_TSI_LOGO

How to Synchronize Data in Real Time Between the Mainframe and AWS with Treehouse Software’s Enterprise CDC Tool

by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.

Bidirectional_Data_Replication

Many mainframe integration scenarios require continuous near-real-time replication of relational data to keep a copy of the data synched in the Cloud. Change Data Capture (CDC) is used for this near-real-time transactional replication by capturing change log activity to drive changes in the target dataset.

Just what is CDC anyway?

Simply put, and in relation to Mainframe-to-Cloud and open systems data replication, CDC is the use of processes to identify when data has been changed in a source system, so the replicated upstream or downstream (depending on how you look at it) target can be kept in sync with the changes.

In a recent AWS Architecture Blog, readers learn about integration using mainframe data to build Cloud native services with AWS, including transactional replication-based integration via CDC.

____AWS_Mainframe_CDC_Diagram

As mentioned in the blog, AWS Partner CDC Tools are available for connecting data center mainframes to the various data targets, and Treehouse Software’s tcVISION is one of those tools available in the AWS Marketplace.

tcVISION allows changes occurring in any mainframe application data to be tracked and captured, and then published to a variety of target AWS databases and applications. tcVISION provides an easy and fast approach for Hybrid Cloud projects, enabling real-time and bi-directional data replication between the hardware and AWS.

Example of Db2-to-AWS CDC using tcVISION Mainframe Manager:

tcVISION_Db2_To_AWS_CDC

tcVISION supports several CDC methods available, depending on each customer’s use case:

Bulk Transfer

  • Efficient transfer of entire databases
  • Analysis for data consistency (verification)
  • Initial load (ETL) and periodic mass data transfer
  • One-step data transfer

Log Processing

  • Transfer of changed data near-realtime or scheduled time frame
  • Reads both active logs and archived logs

Batch Compare

  • Comparison of data snapshots using checksums
  • Efficient transfer of changed data since last processing
  • Flexible processing options (SORT etc.)
  • Automatic creation of deltas by tcVISION

DBMS Extension

  • Real-time capture of changed data directly from the DBMS
  • Secure data storage even across DBMS restart
  • Flexible propagation methods

Interested in seeing a live, online demo of tcVISION CDC?

Just fill out the Treehouse Software tcVISION Demonstration Request Form and a Treehouse representative will contact you to set up a time for your online tcVISION demonstration.


__001_TSI_LOGO

Shopping for an AWS Mainframe Data Replication Solution? Look no further than Treehouse Software’s tcVISION on the AWS Marketplace…

___AWS_Marketplace_tcVISION_AMIs

Using Treehouse Software’s tcVISION to replicate data between a mainframe database and AWS, an enterprise with mission critical data can rapidly deploy databases globally within minutes, and with minimal administration requirements. Supported mainframe databases include Db2, VSAM, IMS/DB, Adabas, IDMS, Datacom, or even sequential files. tcVISION is available on the AWS Marketplace for many global regions, including the AWS GovCloud (US) East and (US) West. 

There are three versions of the tcVISION Amazon Machine Image (AMI) on the AWS Marketplace:

1. tcVISION Enterprise Change Data Capture (CDC) Integration for AWS – Bring Your Own License (Customers must contact Treehouse Software for license key to run the software): This AMI allows transfer of mainframe data to AWS targets continuously and in real-time. Using this tcVISION AMI, a customer can deploy databases globally within minutes.

3. tcVISION Distributed Database Integration (DDI) for AWS – Pay as You Go on AWS: This AMI license installation includes security keys for Amazon S3 access and a license to use one source (PostgreSQL, Oracle, MySQL, MariaDB; MSSQL on Windows OS only) and one target (AWS S3, AWS Aurora, RDS – PostgreSQL, Oracle, MySQL, MariaDB; MSSQL on Windows OS only).

3. tcVISION Mainframe Batch Integration (MBI) for AWS – Bring Your Own License (Customers must contact Treehouse Software for license key to run the software): This AMI does not require an active connection to the mainframe, because the existing backup and recovery files from the mainframe, such as Db2 imagecopies, z/OS archive log files or IMS Unloads, Adabas unload, and PLOG files can be used.

Like what you see? Replicating mainframe data on AWS can happen within days during a tcVISION Proof of Concept (POC)…

An online tcVISION POC is approximately 10 business days, with the customer providing use case and goals for the POC. A Treehouse Software consultant will assist in downloading and installing tcVISION and conducting a limited-scope implementation of a tcVISION application. This application uses customer data and executes on customer facilities, usually in a non-production environment. A document is provided beforehand that outlines the requirements, use cases, and agenda for the POC.

By the end of the 10-day POC, customers can begin replicating mainframe data to their Cloud target database.  It can happen that fast!


Further Reading…

_0_AWS_Logo

Treehouse Software is an AWS Select Technology Partner, and the AWS Partner Network published a blog about tcVISION, our Mainframe-to-Cloud data replication product, which describes how tcVISION allows legacy mainframe environments to continue, while replicating data on highly available and secure Cloud platforms:

https://aws.amazon.com/blogs/apn/real-time-mainframe-data-replication-to-aws-with-tcvision-from-treehouse-software/


__TSI_LOGO

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.

The Mainframe-to-Hybrid Cloud Wave has Arrived, and Treehouse Software is Helping Customers Begin the Ride!

by Joseph Brady, Director of Business Development / AWS and Cloud Alliance Lead at Treehouse Software and Andy Jones, Certified AWS Solutions Architect at Treehouse Software

There are many pioneering organizations with mainframe systems that have already begun their movement to the Cloud, and are now taking advantage of the new and powerful technologies delivering significant business benefits and competitive advantage, including:

Trade Capital Expense for Variable Expense – Instead of having to invest heavily in data centers and servers before you know how you’re going to use them, you can pay only when you consume computing resources, and pay only for how much you consume. https://aws.amazon.com/pricing/

Global Deployments – The AWS Cloud spans 22 geographic regions globally. Enterprises can easily deploy applications in multiple regions around the world with just a few clicks. This means you can provide lower latency and a better experience for your customers at minimal cost. https://aws.amazon.com/about-aws/global-infrastructure/regions_az/

Economies to Scale – By using Cloud computing, you can achieve a lower variable cost than you can get on your own, because usage from hundreds of thousands of customers is aggregated in the Cloud, providers such as AWS can achieve higher economies of scale, which translates into lower pay as-you-go prices. https://aws.amazon.com/economics/

Scale of Services – Amazon Web Services offers a broad set of global Cloud-based products including compute, storage, databases, analytics, networking, mobile, developer tools, management tools, IoT, security, and enterprise applications. These services help organizations move faster, lower IT costs, and scale. https://aws.amazon.com/products/

World Class Security – AWS security compliance is second to none and complies with the most stringent government and private sector requirements. https://aws.amazon.com/compliance/programs/

Extreme High Availability (HA) – The AWS Cloud spans 69 Availability Zones within 22 geographic Regions around the world https://aws.amazon.com/about-aws/global-infrastructure/regions_az/.  By designing your services and applications to be redundant across AWS availability zones or regions, HA is enhanced far beyond a single on premises data center. https://aws.amazon.com/marketplace/solutions/infrastructure-software/high-availability 

Testing at Scale – AWS servers and services can be created and charged on demand for a specific amount of time.  This allows customer to create temporary large-scale test environments prior to deployment that are not practical for on premises environments.  Large scale testing reduces deployment risks and helps to provide a better customer experience. https://aws.amazon.com/  

Auto Scaling and Serverless Deployments – AWS has many serverless and autoscaling options available, allowing for scalable computing capacity as required.  For example, AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume – there is no charge when your code is not running. Another example is Amazon Aurora Serverless, which is an on-demand, auto-scaling configuration for Amazon Aurora (MySQL-compatible edition), where the database will automatically start up, shut down, and scale capacity up or down based on your application’s needs. https://aws.amazon.com/serverless/

Customer Agility and Innovation – In a Cloud computing environment, new IT resources are only a click away, which means that you reduce the time to make those resources available to your developers from weeks to just minutes. This results in a dramatic increase in agility for the organization, since the cost and time it takes to experiment and develop is significantly lower. https://aws.amazon.com/architecture/

Infrastructure as Code – AWS CloudFormation provides a common language for you to describe and provision all the infrastructure resources in your Cloud environment. CloudFormation allows you to use programming languages or a simple text file to model and provision, in an automated and secure manner, all the resources needed for your applications across all regions and accounts. This gives you a single source of truth for your AWS resources. https://aws.amazon.com/cloudformation/ 

However, compared to the number of mainframe shops in enterprises that are just beginning to plan their moves to the Cloud, existing adopters’ numbers are still relatively small.

As an example of the current boom in overall Cloud growth, the worldwide public sector Cloud market will be growing to a staggering $331B by 2022 according to Gartner. By the end of 2019, more than 30% of technology providers’ new software investments will shift from Cloud-first to Cloud-only, further reducing license-based software spending and increasing subscription-based Cloud revenue. (Source: Forbes) 

Treehouse Software is a Trusted Partner on Your Mainframe-to-Cloud Journey

Treehouse Software is a well-established company serving mainframe customers since 1982. We are currently developing a strong presence in the emerging Cloud market space related to mainframe data migration, primarily through our partnership with Amazon Web Services (AWS).  AWS is aware that most large enterprises use mainframe systems that are housing vast amounts of data encompassing historical, customer, logistics, etc., and they have helped us bring our tcVISION solution to the AWS Marketplace. tcVISION provides real-time replication between a variety of mainframe and non-mainframe sources, including (Mainframe): VSAM, IMS, Db2, CA Datacom, Adabas, CA IDMS, and Flat Files; and (Non-mainframe): AWS RDS databases, AWS Aurora, AWS S3, AWS Kinesis,  PostgreSQL, MySQL, Kafka, MongoDB, Hadoop, Oracle, Microsoft SQL Server, IBM Db2 LUW and Db2 BLU, IBM Informix, SAP Hana, and many more. 

AWS sales and technical leaders within various verticals (GovCloud, Nonprofit, K12/Higher Ed, Automotive, etc.) are also beginning to engage with Treehouse Software to learn more about our unique skills and solution that can help them tap into this potential goldmine of massive amounts of legacy data that needs to be moved to AWS. 

Treehouse Software Helps Customers Begin Moving Their Mainframe Data to the Cloud Immediately

Treehouse Software specializes in providing data replication for enterprise customers who want a fully developed and automated way to move data from their mainframe systems to the Cloud. Treehouse Software’s tcVISION is a low risk option that allows customers to immediately begin moving data to the Cloud while they work on the sometimes massive complexity of application migration. Our experience has shown that projects can become stalled while the application side is being figured out. For example, Treehouse Software recently became involved in a project with a government agency that was facing the complexity of a “big bang” migration, which is slowing the project. We are now presenting them with our tcVISION data replication solution option, where they can replicate data to AWS while maintaining their current environment for modernization and migration of their applications. 

Additionally, Treehouse Software’s decades of experience developing software and working in the IBM mainframe environment, in addition to selling and supporting a comprehensive automated data replication product, is making us a desirable partner for AWS and many Cloud migration companies.  

AWS recently published a blog about tcVISION, our Mainframe-to-Cloud data replication product: https://aws.amazon.com/blogs/apn/real-time-mainframe-data-replication-to-aws-with-tcvision-from-treehouse-software/ 

Additionally, here is a blog about Treehouse Software’s extensive mainframe experience: https://treehousesoftware.wordpress.com/2019/09/12/treehouse-softwares-differentiator-weve-been-helping-enterprise-mainframe-sites-since-1982/

If your enterprise is planning on riding the wave with a Mainframe-to-Cloud migration project, we would welcome the opportunity to help get you moving immediately with an online presentation and demonstration of our tcVISION data replication solution. Contact Treehouse Software today!

Treehouse Software’s tcVISION on the AWS Marketplace… AWS Architectural Considerations and Challenges

Andy Jones, Senior Technical Representative for Treehouse Software recently gave a presentation at the Pittsburgh AWS User Group Meetup that explores our experiences in bringing the tcVISION Mainframe-to-AWS Data Replication Solution to the AWS Marketplace.  Andy discussed issues that had to be addressed, such as High Availability and AWS Availability Zones, Autoscaling, Security, creating the tcVISON AMIs, etc. Here is his presentation:

Additionally, Treehouse Software’s tcVISION is Featured on the AWS Partner Network Blog. Read the Blog Here.


__tsi_logo_400x200

Treehouse Software is an AWS Technology Partner and a global leader in providing real-time and bi-directional data replication between a variety of mainframe and non-mainframe sources, including (Mainframe – z/OS and VSE): VSAM, IMS, Db2 z/OS, Adabas, CA Datacom, and CA IDMS; and (Non-mainframe): AWS, Microsoft Azure, PostgreSQL, Oracle, Microsoft SQL Server, IBM Db2 LUW and Db2 BLU, IBM Informix, Kafka, MongoDB, Hadoop, SAP Hana, and more.

Contact us today to discuss your project!

U.S. Government Customers Can Now Use Treehouse Software’s tcVISION Mainframe Data Replication on the AWS GovCloud

Treehouse Software’s tcVISION Mainframe-to-AWS data replication product is now available in the AWS GovCloud (US) Region of the AWS Marketplace.

The AWS GovCloud (US) Region is designed to host sensitive data, regulated workloads, and address the most stringent U.S. government security and compliance requirements.

The AWS GovCloud (US) is available to government customers, organizations in government-regulated industries, and other private entities that meet AWS GovCloud (US) requirements.

AWS GovCloud (US) gives government customers and their partners the flexibility to architect secure cloud solutions that comply with: the FedRAMP High baseline, the DOJ’s Criminal Justice Information Systems (CJIS) Security Policy, U.S. International Traffic in Arms Regulations (ITAR), Export Administration Regulations (EAR), Department of Defense (DoD) Cloud Computing Security Requirements Guide (SRG) for Impact Levels 2, 4 and 5, FIPS 140-2, IRS-1075, and other compliance regimes.

From Personally Identifiable Information (PII), sensitive patient medical records, and financial data to law enforcement data, export controlled data and other forms of CUI, AWS GovCloud (US) Regions can help customers address compliance at every stage of their cloud journey.


_0_AWS_Marketplace_Logo01

The tcVISION Amazon Machine Images (AMIs) are available on the AWS Marketplace. A tcVISION AMI automatically sets up the product on the AWS Cloud and allows customers to use it with either backup/recovery mainframe files or Windows files, or to set up connectivity for real-time, on-premises mainframe data replication to AWS. An enterprise with mission critical data in a mainframe environment can rapidly deploy databases globally within minutes, and with minimal administration requirements. U.S Government sites can simply choose the region “AWS GovCloud (US)” under “Pricing Information” when choosing and AMI.

_0_tcV_To_AWS_Cloud

There are three versions of the tcVISION AMI on the AWS Marketplace, and #1 and #3 require two-week trial license keys provide by Treehouse Software:

1. tcVISION Mainframe Batch Integration (MBI) for AWS – Bring Your Own License (Customers must contact Treehouse Software for license key to run the software): This AMI does not require an active connection to the mainframe, because the existing backup and recovery files from the mainframe, such as Db2 imagecopies, z/OS archive log files or IMS Unloads, Adabas unload, and PLOG files can be used.

2. tcVISION Distributed Database Integration (DDI) for AWS – Pay as You Go on AWS: This AMI license installation includes security keys for Amazon S3 access and a license to use one source (PostgreSQL, Oracle, MySQL, MariaDB; MSSQL on Windows OS only) and one target (AWS S3, AWS Aurora, RDS – PostgreSQL, Oracle, MySQL, MariaDB; MSSQL on Windows OS only).

3. tcVISION Enterprise Change Data Capture (CDC) Integration for AWS – Bring Your Own License (Customers must contact Treehouse Software for license key to run the software): This AMI allows transfer of mainframe data to AWS targets continuously and in real-time. Using this tcVISION AMI, a customer can deploy databases globally within minutes.


About the AWS Marketplace…

_0_AWS_Marketplace_Logo01

The AWS Marketplace is an online software store that helps customers find, buy, and immediately start using the software and services that run on AWS.


__TSI_LOGO

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.

Join Us: Treehouse Software is Sponsoring the Pittsburgh Amazon Web Services (AWS) User Group April Meetup at Bakery Square

AWS_Meetup

Treehouse Software’s, “AWS Architectural Considerations and Challenges When Bringing a Product to the AWS Marketplace” will be presented by Andy Jones, Senior Technical Representative and AWS Certified Solutions Architect for Treehouse Software.

Treehouse Software will also be providing food and drinks for the Meetup.

Date/Time: Tuesday, April 9, 2019 • 7:00 PM to 9:00 PM

Place: Bakery Square – UPMC Enterprises • 6425 Penn Ave. #200 · Pittsburgh, PA

Agenda:

  • 07:00 – 07:15 => Grab food, beverage, and a seat
  • 07:15 – 07:20 => Introductions from Nathan Menge (AWS Pittsburgh User Group Organizer)
  • 07:20 – 08:30 => Topic: “AWS Architectural Considerations and Challenges When Bringing a Product to the AWS Marketplace” – Presentation by Andy Jones, Senior Technical Representative and AWS Certified Solutions Architect for Treehouse Software
  • 08:30 – 09:00 => Q&A with Speaker

Register to attend here:

April meetup

Tuesday, Apr 9, 2019, 7:00 PM

UPMC Enterprises
6425 Penn Ave #200 Pittsburgh, PA

9 Members Attending

Basic agenda 07:00 – 07:15 => Grab food, beverage, and a seat 07:15 – 07:20 => Introductions from Nate 07:20 – 08:30 => Topic: “AWS Architectural Considerations and Challenges When Bringing a Product to the AWS Marketplace” – Presentation by Andy Jones, Senior Technical Representative and AWS Certified Solutions Architect for Treehouse Software 08:…

Check out this Meetup →

Try out Mainframe Data Replication on the AWS Cloud with tcVISION Amazon Machine Images (AMIs)

Treehouse Software is pleased  to announce that the tcVISION Amazon Machine Images (AMIs) are now officially released on the AWS Marketplace. The tcVISION AMI automatically sets up the product on the AWS Cloud and allows customers to use it with either backup/recovery mainframe files or Windows files, or to set up connectivity for real-time, on-premises mainframe data replication to AWS. An enterprise with mission critical data in a mainframe environment can rapidly deploy databases globally within minutes, and with minimal administration requirements.

_0_tcV_To_AWS_Cloud

There are three versions of the tcVISION AMI on the AWS Marketplace, and #1 and #3 require two-week trial license keys provide by Treehouse Software:

1. tcVISION Mainframe Batch Integration (MBI) for AWS – Bring Your Own License (Customers must contact Treehouse Software for license key to run the software): This AMI does not require an active connection to the mainframe, because the existing backup and recovery files from the mainframe, such as Db2 imagecopies, z/OS archive log files or IMS Unloads, Adabas unload, and PLOG files can be used.

2. tcVISION Distributed Database Integration (DDI) for AWS – Pay as You Go on AWS: This AMI license installation includes security keys for Amazon S3 access and a license to use one source (PostgreSQL, Oracle, MySQL, MariaDB; MSSQL on Windows OS only) and one target (AWS S3, AWS Aurora, RDS – PostgreSQL, Oracle, MySQL, MariaDB; MSSQL on Windows OS only).

3. tcVISION Enterprise Change Data Capture (CDC) Integration for AWS – Bring Your Own License (Customers must contact Treehouse Software for license key to run the software): This AMI allows transfer of mainframe data to AWS targets continuously and in real-time. Using this tcVISION AMI, a customer can deploy databases globally within minutes.

tcVISION Offers a Wide Variety of Connectivity

_0_tcVISION_Connection_Overview01

tcVISION supports AWS Relational Database Service (RDS) databases, such as Aurora, PostgreSQL, MySQL, SQL Server, and Maria DB as targets, in addition to traditional Linux/Unix/Windows (LUW) targets such as Oracle, IBM Db2 LUW, Software AG Adabas LUW, IBM Informix, Sybase, Microsoft SQL Server, PostgreSQL, and ODBC.

tcVISION on-premises acquires large amounts of data in bulk or via change data capture methods, including in real time, from virtually any IBM mainframe data source (Software AG Adabas, IBM Db2, IBM VSAM, IBM IMS/DB, CA IDMS, CA Datacom, even sequential files), and transforms and delivers it to the AWS Cloud.

Users can easily model and map their data, and create links between data sources and target systems with tcVISION‘s GUI interface. tcVISION also has features to prevent looping and provide conflict detection for bidirectional data processing, allowing mainframe updates to be reflected on the AWS databases and changes on the AWS databases to be reflected back to the mainframe again.

About the AWS Marketplace…

_0_AWS_Marketplace_Logo01

The AWS Marketplace is an online software store that helps customers find, buy, and immediately start using the software and services that run on AWS.


__TSI_LOGO

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.