by Joseph Brady, Director of Business Development at Treehouse Software, Inc.
Treehouse Software and AWS are collaborating on several AWS-centric initiatives in the coming months. The focus of these efforts is to market our new Treehouse Dataflow Toolkit (TDT), a set of microservices that provides the turn-key solution for transferring data from Kafka into advanced Analytics/AI/ML-friendly targets, such as Amazon Redshift, Snowflake, Amazon Athena/S3, Amazon S3 Express One Zone Buckets, as well as Amazon Aurora PostgreSQL. We have worked with an AWS Marketing Manager to create the following TDT AWS Partner Solution Brief downloadable PDF that provides a one-minute overview of TDT, its benefits, and resource links for your team…
by Joseph Brady, Director of Business Development at Treehouse Software, Inc. and Dan Vimont, Director of Innovation at Treehouse Software, Inc.
We are beginning to see a pleasant and welcomed trend with Treehouse customers who are looking to modernize their valuable mainframe legacy data on the Cloud—they are including their data science teams in the important planning phase of architecting new Cloud environments and targets. This is especially vital for customers who want to incorporate advanced analytics and ML/AI in their strategic data usage plans on the Cloud. Who can contribute better understandings of ultimate data usage than your resident data scientists?
We have heard from many of these data scientists that a primary item on their “wish lists” is for a fully managed, AI powered, massively parallel processing (MPP) architecture to extract maximum value and insights. They specifically mention Amazon Redshift as the Cloud data warehouse (which is much more than a data warehouse) of choice for driving digitization across the enterprise, as well as help personalizing customer experiences. Redshift uses SQL to analyze structured and semi-structured data across data warehouses, operational databases, and data lakes, using AWS-designed hardware and ML to deliver the highest performance at any scale. To this desire/question, we can answer with a resounding, “Yes, Treehouse Software has got you covered with Redshift connectivity!”.
The Treehouse Software solution…
Enterprise customers have come to Treehouse Software, because we bring not only proven mainframe data replication tools, but deep subject matter expertise in mainframe technologies, as well as the know-how to target relevant AWS offerings, such as Redshift, S3 (including S3 Express One Zone – see our recent blog on S3 Express One Zone), etc.
The Rocket Data Replicate and Sync (RDRS) solution allows customers’ legacy mainframe environment to operate normally while replicating data on AWS. The technology focuses on changed data capture (CDC) when transferring information between mainframe data sources and Cloud-based databases and applications. Through an innovative set of technologies, changes occurring in any mainframe datastore are tracked and captured, and ultimately published to Redshift.
How does it work?
We start at the source – the mainframe – where an agent (with a very small footprint) extracts data (in the context of either bulk-load or CDC processing).
The raw data is securely passed from the mainframe to RDRS, which speedily transforms mainframe-formatted data into Unicode/JSON and publishes the results to a Kafka topic.
Our efficient, autoscaling microservices take it from there. Treehouse Dataflow Toolkit functions consume the data from Kafka and land it in S3 buckets, where Treehouse’s proprietary crawler technology is used to automatically prepare landing tables, views, and additional infrastructure in Redshift. Thenthe mainframe data is loaded into Redshift (all the while adhering to AWS’ recommended “best practices” for massive data loading, thus assuring shortest and surest loads). The inherent reliability and scalability of the entire pipeline infrastructure assure near-real-time synchronization between mainframe sources and Redshift target tables.
Redshift tables and views: something for everybody
Within this framework, the Redshift staging tables (often referred to as “delta tables”) are constantly accruing historical data, ideally suited for data scientists looking to do trend analysis, predictive analytics, ML, and AI work. For business analysts and others who prefer structured data representations of potentially complex hierarchical data, the Treehouse framework also automatically provides structured user-views, providing the look and feel of a SQL database.
…as innovations move faster along the timeline, keep your options open!
Publishing both bulk-load and CDC data to a reliable and scalable framework like Kafka allows you to maintain a broad array of options to ultimately feed your legacy data to any number of JSON-friendly ETL tools, target datastores, and data analytics packages (some of which may not even have been invented yet!). In addition to Redshift, the Treehouse Dataflow Toolkit also currently targets Snowflake, Amazon DynamoDB, and Amazon Athena/S3.
Video – Introduction to Data Warehousing on AWS with Amazon Redshift…
Contact Treehouse Software today to discuss your project, or to schedule a demo of our Mainframe-to-AWS real-time and bi-directional data replication solution.
by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.
Treehouse Software specializes in helping enterprise customers with Mainframe-to-Cloud, Multi-Cloud, and Hybrid Cloud data modernization projects. Many times, our customers not only discuss strategies for replicating their mainframe data, but also their plans for what they want to do with that data on the Cloud side. This makes it important to our team to stay current on the latest Cloud offerings that can benefit our customers’ enterprise modernization planning. Consequently, a very exciting announcement caught our attention during the 2023 AWS re:Invent conference—the general availability of a new type of S3 storage service referred to as Amazon S3 Express One Zone Storage Class.
For those unfamiliar, Amazon S3 (“simple storage service”) is the basic file storage service of AWS, and as such it forms a foundational pillar of the entire AWS world. Amazon S3 Express One Zone is a new type of S3 bucket called a “directory bucket”, which is purpose-built to deliver consistent, single-digit millisecond data access for an enterprise’s most frequently used data and latency-sensitive applications. The new S3 directory buckets allow customers to store data in a single Availability Zone (AZ) that they specifically select, as opposed to the default of three AZs for standard S3. This eliminates the latency associated with spreading data across multiple AZs, providing applications with lower-latency storage. S3 directory buckets also follow a different request scaling model compared to traditional buckets, and their authentication is based on sessions rather than on a per-request basis. Bottom line… reduction in compute time = greater cost reduction.
S3 Express One Zone is ideally suited for services such as Amazon SageMaker Model Training, Amazon Athena, Amazon EMR, and AWS Glue Data Catalog to accelerate Machine Learning (ML) and interactive analytics workloads. With S3 Express One Zone, storage automatically scales up or down based on consumption and need, and customers no longer need to manage multiple storage systems for low-latency workloads.
So, why is S3 Express One Zone important to Treehouse mainframe modernization customers?
Amazon S3 Express One Zone just made the Amazon S3 targeting in the Treehouse Dataflow Toolkit (TDT) potentially much more potent and valuable to our enterprise mainframe customers. When an enterprise uses TDT to land their mission critical data in Express One Zone flavored Athena/S3 buckets, it becomes more directly accessible and manipulable by the various AWS ML and AI tools. In short, if customers choose, Express One Zone Athena/S3 becomes an intermediate data store for big data processing workloads and advanced analytics.
So, when we are asked, “What should Treehouse Software be doing to respond to the burgeoning interest in ML, Generative AI, etc.?”, the answer is — We are doing exactly what we need to be doing. AI and ML frameworks are the newest incentive for people to use RDRS (Rocket Data Replicate and Sync — formerly called tcVISION) and TDT from Treehouse Software to replicate their mainframe data on advanced data analytics frameworks, or possibly into super-charged S3 Express One Zone buckets.
Video – Deep Dive Introduction to Amazon S3 Express One Zone Storage Class:
Contact Treehouse Software today to discuss your project, or to schedule a demo of our Mainframe-to-AWS real-time and bi-directional data replication solution.
by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.
Treehouse Software’s customer base of enterprises using mainframes for storing mission-critical data are asking for ways to take advantage of the most advanced Cloud-based data warehouses. Recently, one of our customers sent us an inquiry asking about how tcVISION can be used to move some of their data to Amazon Redshift, the fully-managed data warehouse from AWS, for reporting and analytics using their existing Business Intelligence (BI) tools. They also expressed concerned about scalability of the data warehouse, since they may increase their data movement quite quickly in the near future.
Fortunately, one of our Cloud experts has been testing out the fastest and cleanest ways to use tcVISION to pump z/OS mainframe data into Amazon Redshift. His testing proved that tcVISION fully supports an AWS best practices framework of: Amazon MSK(Kafka)-to-Amazon S3-to-Amazon Redshift. To achieve this within the framework, tcVISION is used to convert mainframe-based data (Adabas, VSAM, Db2, etc.) to JSON format and publish it to Kafka topics (Amazon MSK), then ultimately to Amazon Redshift as seen here…
About Amazon Redshift…
Your journey to Amazon Redshift begins today with Treehouse Software and tcVISION…
tcVISION focuses on changed data capture (CDC) when transferring information between mainframe data sources AWS. Through an innovative technology, changes occurring in any mainframe application data are tracked and captured, and then published to AWS-based data stores. From there, the door is open for the enterprise to take advantage of the latest and most popular AWS AI and ML resources.
by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.
Artificial Intelligence (AI) and Machine Learning (ML) are not only revolutionizing the computing and business worlds in general, but the value proposition for Treehouse Software’s customer base who are replicating large amounts of mainframe data on AWS can be profound. AI/ML is proving to be a powerful force in improving customer experience, optimizing business operations, and accelerating innovation. For example, this powerful and predictive technology can help improve customer engagement and conversion by creating personalized web experiences that are based on data that shows individual preferences and behaviors. Additionally, business forecasting AI and ML tools can accurately predict demand and manage, optimize, and augment supply chain decisions by combining historical time series data with additional variables, such as new product features, pricing, holiday demand, etc.
AWS makes AI and ML technologies available at your fingertips…
AWS is making incredible leaps in creating the most sophisticated AI tools available today — all literally at one’s fingertips. Never before has such powerful and useful technology been so easily available to so many, bringing the deepest set of ML services and supporting Cloud infrastructures instantly into the hands of developers and data scientists.
We are hearing first-hand how customers want to improve customer experience, optimize business processes, and accelerate innovation. For these purposes, businesses can use ready-made, purpose-built AI services, or customized models with AWS AI and ML offerings.
Examples of leading AI product available on AWS:
Amazon SageMaker – A fully managed service that provides every developer and data scientist with the ability to build, train, and deploy ML models at scale.
AWS pre-trained ML Models – Provide ready-made intelligence for your applications and workflows that can be quickly deployed on Amazon SageMaker.
tcVISION focuses on changed data capture (CDC) when transferring information between mainframe data sources AWS. Through an innovative technology, changes occurring in any mainframe application data are tracked and captured, and then published to AWS-based data stores. From there, the door is open for the enterprise to take advantage of the latest and most popular AWS AI and ML resources.
Treehouse Software is the worldwide distributor of tcVISION, the leading tool for using changed data capture (CDC) when transferring information between most mainframe data sources (IBM Db2, IBM VSAM, IBM IMS/DB, Software AG Adabas, CA IDMS, CA Datacom, or even sequential files) and Cloud and open systems-based databases and applications.
The following video takes a quick look at how tcVISION’s repository is used to import mainframe IDMS schema and builds out a target system on AWS:
Interested in seeing a live, online demo of tcVISION?
Treehouse Software is the worldwide distributor of tcVISION, the leading tool for using changed data capture (CDC) when transferring information between most mainframe data sources (IBM Db2, IBM VSAM, IBM IMS/DB, Software AG Adabas, CA IDMS, CA Datacom, or even sequential files) and Cloud and open systems-based databases and applications.
The following video briefly takes a look at how the tcVISION Control Board (a Windows-based GUI interface) functions as a central point of administration, data mapping and modeling, script generation, and overall monitoring. In this example, we show how to manage connectivity and data replication between a Mainframe database and AWS PostgreSQL using the tcVISION Control Board:
Further reading: tcVISION is featured on the AWS Partner Network Blog showing a walk-through of data replication between Mainframe and Amazon Aurora…
Treehouse Software is the worldwide distributor of tcVISION, the leading tool for using changed data capture (CDC) when transferring information between most mainframe data sources (IBM Db2, IBM VSAM, IBM IMS/DB, Software AG Adabas, CA IDMS, CA Datacom, or even sequential files) and Cloud and open systems-based databases and applications. Changes occurring in the mainframe application data are then tracked and captured, and published to a variety of targets.
This video demonstrates real-time data replication between Db2 and PostgreSQL on AWS:
This video demonstrates real-time data replication between Db2 and Google Cloud SQL:
This video demonstrates real-time data replication between Db2 and Azure SQL:
Further reading: tcVISION is featured on the AWS Partner Network Blog showing a walk-through of data replication between Mainframe DB2 z/OS and Amazon Aurora…
by Joseph Brady, Director of Business Development and Cloud Alliance Leader at Treehouse Software, Inc.
Treehouse Software has been helping enterprise mainframe customers since 1982, and in recent years, we have been developing a strong presence in the Mainframe-to-Cloud data replication market space. This blog takes a quick look at three of the most popular Treehouse-supported Cloud-based SQL database services…
Amazon RDS, a collection of managed services that makes it simple to set up, operate, and scale databases in the Cloud. Users can control the type of database, as well as where data is stored. Specific database formats that are supported include Amazon Aurora, PostgreSQL, MySQL, MariaDB, Oracle Database, and SQL Server:
Google Cloud SQL, a fully managed relational database service for MySQL, PostgreSQL, and SQL server. You can connect with nearly any application, anywhere in the world. Cloud SQL automates backups, replication, and failover to ensure your database is reliable, highly available, and flexible to your performance needs:
Microsoft Azure SQL, a part of the Azure SQL family, Azure SQL Database is an always-up-to-date, fully managed relational database service built for the Cloud:
Wherever you want to target your mainframe data on the Cloud, Treehouse Software helps to make the process easy…
Treehouse Software is the worldwide distributor of tcVISION, the leading tool for using changed data capture (CDC) when transferring information between most mainframe data sources (IBM Db2, IBM VSAM, IBM IMS/DB, Software AG Adabas, CA IDMS, CA Datacom, or even sequential files) and Cloud and open systems-based databases and applications. Changes occurring in the mainframe application data are then tracked and captured, and published to a variety of targets.
Additionally, tcVISION supports bi-directional data replication, where changes on either platform are reflected on the other platform (e.g., a change to a PostgreSQL table in the Cloud is reflected back on mainframe), allowing the customer to modernize their application on the Cloud or open systems without disrupting the existing critical work on the legacy system. tcVISION’s bi-directional replication writes directly to the mainframe database, thereby bypassing all mainframe business logic, so this architecture requires careful planning, as well as thorough and repeated testing.
Sales and technical leaders at the major Cloud platform companies, as well as systems integrators are engaging with Treehouse Software to take advantage of our tcVISION data replication solution to help them tap into the mainframe data that customers want to be made available on new technologies.
Further reading: tcVISION is featured on the AWS Partner Network Blog showing a walk-through of data replication between Mainframe DB2 z/OS and Amazon Aurora…
tcVISION is a data replication solution that provides an IBM mainframe integration solution for mainframe data replication, data synchronization, data migration, and change data capture (CDC) to multiple Azure data platform services.
Contact Treehouse Software Today…
Treehouse Software is the worldwide distributor of tcVISION, a software product that allows immediate data replication between many Mainframe sources and Cloud and Open Systems targets, enabling government, healthcare, supply chain, financial, and a variety of public service organizations meet spikes in demand for vital information. No matter where you want your mainframe data to go – the Cloud, Open Systems, or any LUW target, tcVISION from Treehouse Software is your answer.
How does it work?
Treehouse Software's Blog 