Treehouse Proof of Concept: Bi-directional Replication Between Adabas and SQL Server

Chris Rudolph and Kevin Heimbaugh, Senior Technical Representatives for Treehouse Software, visited a customer site (a large retail and distribution company) to perform a five-day proof of concept (POC) of tcVISION with bi-directional replication between Software AG’s Adabas and Microsoft SQL Server.

Chris and Kevin initially met with the customer team, consisting of the DBA, Applications Manager, and a technical applications person. The agenda for the week was set to:

  • Import metadata from several Adabas files
  • Bulk load the Adabas data into SQL Server
  • Set up replication from Adabas to SQL Server
  • Add the bi-directional replication back to Adabas

Additionally, there were a few other items the customer wanted the Treehouse team to address, including support for date formats; timestamps for bi-directional replication to avoid update conflicts; using Predict views to define multiple SQL Server tables; and support for MUs and PEs. Chris noted that everything on the customer’s list is easily supported, and there are several options for the update scenarios that can be used.

_0_tcVISION_Adabas_To_SQLServer

After the tcVISION components were installed, the POC began by using tcVISION’s Control Board to define a metadata repository database in SQL Server. Once that was set, the teams moved on to import the first Adabas file’s metadata using tcVISON’s Metadata Import Wizard. As part of this process tcVISION generated Adabas to SQL Server schemas and field-to-column links as well as created target tables in SQL Server. Bulk Transfer scripts were created using a wizard to read the Adabas file on the mainframe, and load the data into SQL Server using the SQL Server bulk loader. Chris created a control script to show how tcVISION can concurrently bulk transfer multiple Adabas files into SQL Server This required increasing the tcVISION Manager’s VSE partition size to successfully test multiple load scripts executing in parallel.

The teams moved on to define the real-time change data capture (CDC) scripts necessary to process the Adabas PLOG. The tcVISION scripts use a two-phase approach to queue captured Adabas transaction on the open platform, then transform and apply the transactions to SQL Server. The scripts were set up to automatically generate detailed logs to track the PLOG transactions captured, SQL statements successfully applied to SQL Server, failed SQL statements, and informational items such as auto-corrected data and transactions rejected due to processing rules.

Now that several tables were defined and loaded, the bi-directional process was set up. SQL Server CDC was enabled for each table to be replicated. The team made a change within SQL Server and verified that the change show up in the SQL Server CDC tables. The SQL Server-to-Adabas mappings were defined in the tcVISION metadata repository, including the “back update check” to ensure only non-tcVISION transactions are captured, and the scripts on both Windows and mainframe were defined to create the LUWs from the SQL Server CDC and apply the changes to Adabas.

CDC from SQL Server to Adabas was successfully tested. Chris then showed the ability to create Journal replication where each change can be captured by replication type. The team spent time creating a few more mappings so multiple file / table updates could be tested, in addition to doing updates while the scripts were stopped to simulate a lost connection. This included setting up a new script to process copied PLOG datasets created by the ADARES utility.

The team defined the remainder of their Adabas files to the metadata repository. Some were set them up for bi-directional replication, and others were setup for unidirectional replication and Journal replication. Everything work as expected at the wrap-up meeting where the team provided a live demonstration to management of tcVISION and the items accomplished. The final tcVISION presentation and demo went very well, and everyone was pleased with the progress made during the week.


Find out more about tcVISION — Enterprise ETL and Real-Time Data Replication Through Change Data Capture

tcVISION provides easy and fast data migration for mainframe application modernization projects and enables bi-directional data replication between mainframe, Linux, Unix and Windows platforms.

_0_tcvision_connection_overview

tcVISION acquires data in bulk or via change data capture methods, including in real time, from virtually any IBM mainframe data source (Software AG Adabas, IBM DB2, IBM VSAM, IBM IMS/DB, CA IDMS, CA Datacom, even sequential files), and transform and deliver to virtually any target. In addition, the same product can extract and replicate data from a variety of non-mainframe sources, including Adabas LUW, Oracle Database, Microsoft SQL Server, IBM DB2 LUW and DB2 BLU, IBM Informix and PostgreSQL.


__tsi_logo_400x200

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.

TREETIP: tcVISION Allows for Surprisingly Innovative Uses

tcVISONv6

Treehouse Professional Services consultants help Adabas / Natural customers in a variety of ways, including DBA services, performance tuning, change management implementation and training, and data replication planning and training. Our experience and long history of service to the Adabas / Natural community helps us create innovative solutions for our customers’ challenges.

Recently Treehouse Senior Technical Representative Chris Rudolph assisted a customer with a tricky data replication problem. The customer uses tcVISION to perform bi-directional replication between Adabas and an RDBMS during the phase-in of a new application. Unfortunately, the new application incorrectly updated certain columns in the RDBMS, which were then replicated to Adabas. The customer attempted to address the issue by running a series of ADASEL reports against the Adabas PLOG and manually checking for “bad” transactions, which was a very time consuming process that pulled the Adabas DBA away from her normal duties.

Chris explained that tcVISION could expedite the process by replicating all transactions for the Adabas file to a journal table capturing the “before” and “after” values of the problematic columns. The developers working on the new application could then identify invalid values, correct the application, and patch the data themselves. This also allowed the Adabas DBA to return to their normal duties.

The journal table now includes columns to display the “before” and “after” values of the corrupted column, Adabas transaction time, end transaction time, operation and Adabas userid. The customer’s developers immediately recognized immense value from being able to query the journal table to find bad data, patch the data, prove that corruption is no longer taking place, and verify that all corrupted instances of the data have been patched. Journal tables have been added for all replicated Adabas files, and the developers now rely on the journal tables for all of their data patches.


Find out more about tcVISION — Enterprise ETL and Real-Time Data Replication Through Change Data Capture

tcVISION provides easy and fast data migration for mainframe application modernization projects and enables bi-directional data replication between mainframe, Linux, Unix and Windows platforms.

_0_tcVISION_Simple_Diagram

tcVISION acquires data in bulk or via change data capture methods, including in real time, from virtually any IBM mainframe data source (Software AG Adabas, IBM DB2, IBM VSAM, IBM IMS/DB, CA IDMS, CA Datacom, even sequential files), and transform and deliver to virtually any target. In addition, the same product can extract and replicate data from a variety of non-mainframe sources, including Adabas LUW, Oracle Database, Microsoft SQL Server, IBM DB2 LUW and DB2 BLU, IBM Informix and PostgreSQL.


__TSI_LOGO

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.

Smart Solutions for “Data Source-to-Customer Experience” from Treehouse Software

_0_treehouse_to_cognistxsm

Treehouse Software provides world-class capabilities to acquire data from relevant structured sources–especially mainframe-based sources–across the enterprise, enabling solutions leveraging Cognistx technology to offer the most advanced interaction with your customers.

Retailers Adopting Cognitive Technology to Drive Business

Meet a new cognitive-powered app that’s changing the customer experience. Using the SmartCognitives Engine™, it harnesses the full power of customer data to create intelligent, individualized offers specifically designed for each customer. It’s how Cognistx is helping Mr. Tire reach their millions of customers as true individuals, drive incremental revenue, convert data into insights and lead the cognitive revolution in the automotive industry. Check out the video below to see it in action.

Turning data into insights, insights into customer actions, and those actions into revenue…

approach_2

Cognistx applies the latest in machine learning technology (AI) — including Big Data, advanced analytics, artificial intelligence and natural language processing — to solve your business challenge. From app-based retail solutions to help desk chat interfaces, clickstream interceptors to call center systems and more, this technology helps solve your most pressing issues.

Smarter with each interaction

The SmartCognitives Recommendation Engine is always learning and automatically delivering smarter, more relevant offers so your marketing is faster, more accurate and more efficient. It can be integrated into your mobile apps, web sites, databases, POS systems and more — for smarter solutions right away.

Transforming consumers into Individuals

Imagine being able to target consumers as unique individuals and serving up exactly what they want exactly when they need it most. No one wants to be treated like a consumer — but people do want to be treated like individuals. That’s why Cognistx is here– to celebrate the power of the individual and transform the customer experience.

Engaging with the SmartCognitives Engine

technology_highres_3

The Cognistx engine is technology-agnostic, so it can be easily integrated into apps, chatbots, clickstream interceptors, call center systems, decision support systems, or other custom applications.

The Treehouse Software / Cognistx Connection

_0_treehouse_to_cognistx002

The Cognistx platform complements Treehouse Software’s tcVISION, the comprehensive product that can acquire data in bulk or via change data capture methods, including in real time, from virtually any IBM mainframe data source (Software AG Adabas, IBM DB2, IBM VSAM, IBM IMS/DB, CA IDMS, CA Datacom, even sequential files), and transform and deliver to virtually any target. In addition, the same product can extract and replicate data from a variety of non-mainframe sources, including Adabas LUW, Oracle Database, Microsoft SQL Server, IBM DB2 LUW and DB2 BLU, IBM Informix and PostgreSQL.


Let’s talk…

__TSI_LOGO

If this exciting new technology is of interest to you, we would be happy to have a conversation about your company’s needs, so do not hesitate to contact us if you have questions.  Meanwhile, please visit the Treehouse Software Cognistx Web Page for more information.

 

Two Local Technology Companies Partner to Advance Cognitive Computing; Complementary Areas of Expertise Mean Better Data Integration and Individualization

Treehouse Software, Inc., of Sewickley, PA and Cognistx of Pittsburgh, PA announced a partnership to help customers with improved data integration and individualization to fully leverage the power of cognitive computing.

Technology industry leaders from Accenture to Gartner to McKinsey recognize the future of computing will be cognitive, calling it a disruptive force and estimating the industry to reach $200 billion by 2020. Cognitive computing is based on leading edge technology including artificial intelligence, natural language processing, Big Data, advanced analytics and machine learning algorithms.

cog_push_med2

The Treehouse – Cognistx partnership will allow customers to ingest massive amounts of data, whether that data is numbers, images, or audio files, and mine it to find insights that lead to action, and ultimately to increased revenue from improved customer engagement.

Since the mid-1990s, Treehouse Software has been a global leader in mainframe data migration, replication and integration, offering robust and flexible solutions for ETL, CDC and real-time, multidirectional replication between databases on various platforms.

Cognistx is an applied technology company harnessing state-of-the-art cognitive computing tools to help retailers reach individuals with intuitive, intelligent and individualized offers based on their past transactions, preferences, context and profile.

cog_mobile_tech_image

Cognistx complements Treehouse’s capability to deliver data with its machine learning algorithms that become more accurate with every transaction, delivering customized, personalized, prescriptive actions in the right context. Together, the two companies will co-market their capabilities, bringing new competitive advantages to customers who want to expand the use of their most valuable asset — data.

“We’re excited to partner with Cognistx to bring our world-class enterprise data acquisition capabilities to companies that recognize the massive opportunity cognitive computing represents,” said Wayne Lashley, Treehouse Chief Business Development Officer. “We provide the data foundation and Cognistx translates that data into insights, those insights into customer actions, and those actions into incremental revenue.”

“Few retailers do a good job of marrying technology with a customized customer experience that is tailored to their behaviors and timed according to how they might use a retailer’s offer,” said Sanjay Chopra, CEO of Cognistx. “With our proprietary algorithms and Treehouse’s enterprise data solutions, both our customers win. Only with large amounts of data can our system learn about the consumer and their preferences and how those change in order to deliver only the smartest, most individualized offers.”

About Treehouse Software, Inc.

Privately-held Treehouse Software was founded in 1982, and is a global leader in providing data migration, replication, and integration solutions for the most complex and demanding heterogeneous environments. Treehouse offers a comprehensive and flexible portfolio of software and tools for mainframe platforms, and also includes feature-rich, accelerated-ROI offerings for information delivery, and application modernization. http://www.treehouse.com

About Cognistx

Privately-held Cognistx was founded in 2015 and has a technology hub in Pittsburgh and operations offices in the Innovation Quarter in Winston-Salem, NC, and Raleigh. The company’s co-founders include Sanjay Chopra, a serial technology entrepreneur; Eric Nyberg, professor at Carnegie Mellon University’s School of Computer Science, who consulted with IBM on the Watson project and Jeffrey Battin, former owner of Communefx, a successful data analytics company. Other partners include Florian Metze, professor at Carnegie Mellon University’s School of Computer Science; Jill Zoria, SVP Enterprise Development; Pete Minnelli, SVP Creative; and Karen Barnes, SVP Operations. http://www.cognistx.com

PRODUCT SPOTLIGHT: tcVISION v6 Overview and Updates

tcVISONv6

Several exciting new features and updates are now in tcVISION v6, including new output targets Adabas LUW, DB2 BLU, EXASOL, Hadoop,and MongoDB. Additionally, new input sources include z/OS VSAM Logstream (CICS and Coupling Facility / Shared VSAM),z/OS VSAM Batch Extension, z/OS DBMS to Logstream, CA IDMS v17, CA Datacom CDC, IMS Active Log and SMF data.

Another feature recently announced is the tcVISION “Direct Loader” for BULK_LOAD processing. The function does not require output to a sequential file, and the loader utility for the target DBMS is called via API with data passed directly. Direct loader supports PostGreSQL, Microsoft SQL Server and DB2 LUW / DB2 BLU. The advantage of using the Direct Loader is the elimination of disk access in writing and reading the sequential loader data file. File output is still supported (e.g., where loader data is to be distributed to other machines).

Finally, as mentioned in a previous Treehouse Blog, with tcVISION v6 comes the newly enhanced web statistics functionality and web server. Any standard web browser can access this server (Internet Explorer, Firefox, Opera, Chrome, Safari, etc.)

This valuable feature enables users to view data from the tcVISION Manager Monitor, and statistical and operational information from the tcVISION Manager network.


Find out more about tcVISION — Enterprise ETL and Real-Time Data Replication Through Change Data Capture

tcVISION provides easy and fast data migration for mainframe application modernization projects and enables bi-directional data replication between mainframe, Linux, Unix and Windows platforms.

_0_tcVISION_Simple_Diagram

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.

Mainframe CDC from Treehouse Software

The globalization of markets, increase of data volumes, and high demand for up-to-date information require new data transfer and exchange solutions for heterogeneous IT architectures, and as many customers have discovered, Treehouse Software has the right product (or combination of products) to meet any conceivable mainframe data migration, replication, or integration requirement. To meet many of these needs, Treehouse Software’s proven and mature tcVISION product moves data – as little as possible – as much as necessary. tcVISION is an innovative software solution that processes changed data in real time, in intervals, or event based.

ChangeDataCapture

The tcVISION solution focuses on changed data capture (CDC) when transferring information between mainframe data sources and LUW databases and applications. Changes occurring in any mainframe application data are tracked and captured, and then published to a variety of RDBMS and other targets.

_0_tcVISION_Simple_Diagram

tcVISION enables bidirectional replication for DB2, Oracle, and SQL Server running on Linux/Unix/Windows, and synchronizes each data source, first by doing a bulk load from source(s) to target and then by replicating only changes— only committed changes—from source(s) to target. So there can never be ambiguity as to whether a query against the target database involves uncommitted data.

Read some tcVISON customer success stories here.


Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.

 

Subscribe to Treehouse Software’s Monthly eBulletin…

___ConstantContact_eBulletin

Stay in the loop with all of the happenings at Treehouse Software by signing up for Treehouse Software’s monthly eBulletin! You’ll receive tips and tricks for Treehouse products, news, and interesting mainframe-related articles from around the web. SUBSCRIBE HERE.