TREETIP: tcVISION Allows for Surprisingly Innovative Uses

tcVISONv6

Treehouse Professional Services consultants help Adabas / Natural customers in a variety of ways, including DBA services, performance tuning, change management implementation and training, and data replication planning and training. Our experience and long history of service to the Adabas / Natural community helps us create innovative solutions for our customers’ challenges.

Recently Treehouse Senior Technical Representative Chris Rudolph assisted a customer with a tricky data replication problem. The customer uses tcVISION to perform bi-directional replication between Adabas and an RDBMS during the phase-in of a new application. Unfortunately, the new application incorrectly updated certain columns in the RDBMS, which were then replicated to Adabas. The customer attempted to address the issue by running a series of ADASEL reports against the Adabas PLOG and manually checking for “bad” transactions, which was a very time consuming process that pulled the Adabas DBA away from her normal duties.

Chris explained that tcVISION could expedite the process by replicating all transactions for the Adabas file to a journal table capturing the “before” and “after” values of the problematic columns. The developers working on the new application could then identify invalid values, correct the application, and patch the data themselves. This also allowed the Adabas DBA to return to their normal duties.

The journal table now includes columns to display the “before” and “after” values of the corrupted column, Adabas transaction time, end transaction time, operation and Adabas userid. The customer’s developers immediately recognized immense value from being able to query the journal table to find bad data, patch the data, prove that corruption is no longer taking place, and verify that all corrupted instances of the data have been patched. Journal tables have been added for all replicated Adabas files, and the developers now rely on the journal tables for all of their data patches.


Find out more about tcVISION — Enterprise ETL and Real-Time Data Replication Through Change Data Capture

tcVISION provides easy and fast data migration for mainframe application modernization projects and enables bi-directional data replication between mainframe, Linux, Unix and Windows platforms.

_0_tcVISION_Simple_Diagram

tcVISION acquires data in bulk or via change data capture methods, including in real time, from virtually any IBM mainframe data source (Software AG Adabas, IBM DB2, IBM VSAM, IBM IMS/DB, CA IDMS, CA Datacom, even sequential files), and transform and deliver to virtually any target. In addition, the same product can extract and replicate data from a variety of non-mainframe sources, including Adabas LUW, Oracle Database, Microsoft SQL Server, IBM DB2 LUW and DB2 BLU, IBM Informix and PostgreSQL.


__TSI_LOGO

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.

PRODUCT SPOTLIGHT: tcVISION v6 Overview and Updates

tcVISONv6

Several exciting new features and updates are now in tcVISION v6, including new output targets Adabas LUW, DB2 BLU, EXASOL, Hadoop,and MongoDB. Additionally, new input sources include z/OS VSAM Logstream (CICS and Coupling Facility / Shared VSAM),z/OS VSAM Batch Extension, z/OS DBMS to Logstream, CA IDMS v17, CA Datacom CDC, IMS Active Log and SMF data.

Another feature recently announced is the tcVISION “Direct Loader” for BULK_LOAD processing. The function does not require output to a sequential file, and the loader utility for the target DBMS is called via API with data passed directly. Direct loader supports PostGreSQL, Microsoft SQL Server and DB2 LUW / DB2 BLU. The advantage of using the Direct Loader is the elimination of disk access in writing and reading the sequential loader data file. File output is still supported (e.g., where loader data is to be distributed to other machines).

Finally, as mentioned in a previous Treehouse Blog, with tcVISION v6 comes the newly enhanced web statistics functionality and web server. Any standard web browser can access this server (Internet Explorer, Firefox, Opera, Chrome, Safari, etc.)

This valuable feature enables users to view data from the tcVISION Manager Monitor, and statistical and operational information from the tcVISION Manager network.


Find out more about tcVISION — Enterprise ETL and Real-Time Data Replication Through Change Data Capture

tcVISION provides easy and fast data migration for mainframe application modernization projects and enables bi-directional data replication between mainframe, Linux, Unix and Windows platforms.

_0_tcVISION_Simple_Diagram

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.

TREETIP: tcVISION Supports Hadoop

by Joseph Brady, Manager of Marketing and Technical Documentation at Treehouse Software, Inc.

tcVISONv6

Hadoop and Big Data are revolutionizing data processing, and because of the increasing digitalization, the Internet, the rising importance of Social Media, and the presence of “Internet of Things”, the data diversity is growing in dimensions that did not exist before.

To process and maintain large and diverse data sets in a meaningful way, new technologies (such as Hadoop) have been developed. What is Hadoop? Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a distributed computing environment. It is part of the Apache project sponsored by the Apache Software Foundation.

Enterprises with heterogenous IT infrastructures, especially larger corporation of all industry sectors and public institutions, very often include mainframe technology. These enterprises are now facing the challenge to integrate existing mainframe data into a Hadoop platform – in real-time.

Data integration technology also has experienced great evolution over the past decades. Today, a standard ETL solution is not sufficient, and the understanding of data integration must now include the entire data exchange process in terms of replication and synchronization. Data exchange is now a time critical process. Near real-time is more and more the only accepted method to meet the high, up-to-date requirements in an increasing co-existence of mainframe and Hadoop technologies.

The tcVISION Solution

An important part of the added value of modern IT systems is the latency-free data- and process-integration of transactional and analytical areas. The cross-system integration platform from Treehouse Software, tcVISION, is unique, efficient, and reliable. With tcVISION, mainframe data can quickly and easily be integrated in near real-time into Hadoop-based operative applications or Business Intelligence and Analytics.

The tcVISION solution is proven and mature, and is constantly under development to meet the requirements of new technologies, including support for Hadoop in Version 6.

The main focus of the tcVISION integration platform is to allow real-time synchronization to integrate mainframe data into Hadoop based solutions.

hadoop_solution

The tcVISION Technology Components

The tcVISION integration platform consists of a variety of state-of-the-art technology components, which cover much more than simply an ETL process.

  1. Data exchange in the sense of a real-time synchronization becomes a single step operation with tcVISION.
  2. No additional middleware is required.
  3. Modern Change Data Capture Technologies allow an efficient selection of the required data from the source system with focus on the changed data. The data exchange process is reduced to the necessary minimum which results in lower costs for the cross-system data integration.
  4. tcVISION also supports the fast and efficient load of large volumes of mainframe data into Hadoop. In this context the processor costs of the mainframe are low and negligible.
  5. An integrated Data Repository guarantees an overall cross-platform and transparent data management. Mainframe knowledge is not required.
  6. tcVISION include a rule-engine to transform data into a target compliant format or allows user-specific processing via supplied APIs.
  7. The integrated staging concept supports the offload of changed data in “Raw Format” to less expensive processor systems. This reduces mainframe processor resources to a minimum. The preparation of the data for the target system can be performed on a less expensive platform (Linux, UNIX or MS-Windows).
  8. The transfer to and feeding of data into Hadoop is part of the tcVISION data exchange process. No intermediate files are required.
  9. The exchange of large volumes of data between a production mainframe environment and Hadoop can run in parallel processes to reduce latencies to a minimum.
  10. The tcVISION integration platform contains comprehensive control mechanisms and monitoring functions for an automated data exchange.
  11. tcVISION has been designed in a way that Hadoop-based projects can be deployed with total project autonomy and maximum reduction of mainframe resources.

With tcVISION, data synchronization between mainframe and Hadoop pays off

  • Near real-time replication of mainframe data to Hadoop allows actual real-time analytics, or the relocation of mainframe applications (i.e., Internet applications like Online-Banking, e-Government, etc.) to Hadoop with synchronous data on both platforms.
  • Because of the concentration on changed data, the costs of the data exchange are greatly reduced.
  • The utilization of mainframe resources is reduced to a level that minimizes costs for mainframe know how and mainframe MIPS.
  • Data exchange processes can be deployed and maintained with tcVISION without mainframe knowledge, hence costs can be saved and Hadoop projects can be faster developed and put into production.
  • The near real-time replication of tcVISION from mainframe to Hadoop allows the relocation of BI reporting and analytic applications to the more cost efficient and – for these applications – more powerful Hadoop platform.

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs..

PRODUCT SPOTLIGHT: tcACCESS

Transparent integration of mainframe data sources and mainframe programs into LUW applications

__tcACCESS_Cubes

tcACCESS is a comprehensive software solution that enables two-way integration between IBM mainframe systems and client/server, Web and SOA technologies — without the need for mainframe knowledge or programming effort. tcACCESS is a proven platform that facilitates SQL-based integration of mainframe data sources and programs into LUW applications using industry standards such as SQL, ODBC, JDBC, and .NET. SQL queries to access mainframe data can be easily created using drag and drop techniques — no programming required. Read More


__TSI_LOGO

Subscribe to Treehouse Software’s Monthly eBulletin

TREETIP: tcVISION’s Enhanced Web Statistics Functionality

by Joseph Brady, Manager of Marketing and Technical Documentation at Treehouse Software, Inc.

tcVISIONv6_Logo

With version 6 of tcVISION (Treehouse Software’s product for transferring information between mainframe data sources and LUW databases and applications), a newly enhanced web statistics functionality and web server have been introduced. Any standard web browser can access this server (Internet Explorer, Firefox, Opera, Chrome, Safari, etc.).

This valuable feature enables users to oversee data from the tcVISION Manager Monitor, and statistical and operational information from the tcVISION Manager network. The Manager Monitor displays diagrams in a large, scalable window that shows “running” transfers and “not running” processes, and a diagram that shows statistical data.

The following screen shot shows how the tcVISION web statistics can be displayed to the user…

tcVISION_Web_Stats001

An easy-to-use menu allows complete control of, and access to, all processes within the tcVISION Manager Monitor…

_0_tcVISION_Webstats002

  • “Scripts” shows running processes, completed processes, and not running processes.
  • “Server messages” shows the server messages.
  • “Manager network” shows all Managers of connected Manager Networks.
  • “Statistics” shows a selection of Managers and corresponding scripts from the monitoring database. A selection list with different diagram types is available and a button to save chosen diagrams and their selected settings.
  • “Profiles” shows the profiles of the monitoring database.
  • “Manager” shows the name, the version, the revision and the operating system of the observing Managers.

Find out more about tcVISION — Enterprise ETL and Real-Time Data Replication Through Change Data Capture

tcVISION provides easy and fast data migration for mainframe application modernization projects and enables bi-directional data replication between mainframe, Linux, Unix and Windows platforms.

_0_tcVISION_Simple_Diagram

Visit the Treehouse Software website for more information on tcVISION, or contact us to discuss your needs.

TREETIP: Did You Know About tRelational’s Schema Auto-Generation Feature?

by Joseph Brady, Manager of Marketing and Technical Documentation at Treehouse Software, Inc.

tRelational, the data analysis, modeling, and mapping component of Treehouse Software’s Adabas-to-RDBMS product set provides three options for developing RDBMS data models and mapping ADABAS fields to RDBMS columns:

Option 1: Auto-generation

Option 2: Importation of existing RDBMS schema elements

Option 3: Detailed definition and manipulation of schema and mapping elements using tRelational

The Auto-generation function can be an extremely useful productivity aid. By simply invoking this function and specifying an Adabas file structure, a fully-functional corresponding RDBMS schema (Tables, Columns, Primary Keys, Foreign Key relationships and constraints) and appropriate mappings are created virtually instantaneously. The table and column names, datatypes, lengths, and mappings/transformations are all automatically tailored specifically for the RDBMS product, platform, and version–the user need not be an expert in the RDBMS.

tRelational’s schema auto-generation simply requires specification of an Adabas file structure and associated options…

tRe_AutoGen

The auto-generated model can be immediately used to generate both RDBMS DDL and parameters for the Data Propagation System (DPS) component. Within minutes of identifying the desired Adabas file or files to tRelational, the physical RDBMS schema can be implemented on the target platform and DPS can begin materializing and propagating data to load into the tables.

It is important to note that these modeling options complement each other and can be used in combination to meet any requirements. Auto-generated schema elements can be completely customized “after the fact”, as can imported elements. Auto-generation can be used at the file level to generate complete tables and table contents at the field level, making it easy to then manually define and map one or more columns within a table, or even to denormalize MU/PE structures into a set of discrete columns.


About Treehouse Software’s tRelational / DPS Product Set

tReDPSMainDiagram01

tRelational / DPS is a robust product set that provides modeling and data transfer of legacy Adabas data into modern RDBMS-based platforms for Internet/Intranet/Business Intelligence applications. Treehouse Software designed these products to meet the demands of large, complex environments requiring product maturity, productivity, feature-richness, efficiency and high performance.

The tRelational component provides complete analysis, modeling and mapping of Adabas files and data elements to the target RDBMS tables and columns. DPS (Data Propagation System) performs Extract, Transformation, and Load (ETL) functions for the initial bulk RDBMS load and incremental Change Data Capture (CDC) batch processing to synchronize Adabas updates with the target RDBMS.

Visit the Treehouse Software website for more information on tRelational / DPS, or contact us to discuss your needs.

Treehouse Software Products and Professional Services for Adabas / Natural Help with ALLETE’s Mainframe Retirement

by Mike Kuechenberg, Senior Technical Representative at Treehouse Software, Inc. __Allete

Since 2010, Treehouse Software consultants have provided product training and Professional Services for the planned retirement of the mainframe environment at ALLETE.

Here’s a look at three parts of the project in which Treehouse was involved:

tRelational / DPS

In order to replicate Adabas data to Oracle, ALLETE licensed Treehouse’s tRelational / DPS product set. After some initial training by Treehouse Professional Services, ALLETE staff were able to model and map Adabas files to Oracle schemata using tRelational, and to deploy DPS jobs to materialize (ETL) and propagate (CDC) the Adabas data into Oracle. This continued until the final cutover to the new systems. Just before the mainframe was decommissioned, some additional Adabas files were materialized by Treehouse Professional Services for archival purposes.

DBA Consulting

Treehouse Professional Services took over the DBA responsibilities for ALLETE starting in September 2010. The responsibilities included Software AG product upgrades, daily checks on available database space and file extents, and requests from ALLETE staff like file restores, field maintenance, and any other request requiring DBA involvement.

Systems Support

Additionally, Treehouse Professional Services assumed responsibility for ALLETE’s mainframe systems programming activities. Tasks included monthly IPLs to the system, applying licenses to products, handling system error situations, defining new DASD, upgrading IBM and third-party products, providing periodic SOX reports, and handling other requests from ALLETE staff.

Conclusion of a Successful Engagement

As this five-year Treehouse Software / ALLETE partnership ends, we wish our friends all the best as they move forward with their new systems and strategies.

“During the time our companies have been working together, Treehouse personnel were always responsive and are experts in their fields. Thanks for the flexibility and great support over the years.”

Eric Peterson, Manager, IT Infrastructure, ITS Systems, ALLETE Inc.


About Treehouse Software Professional Services _0_Consulting_Services002

Treehouse Professional Services offer a proven, cost-effective alternative to full-time in-house systems programmers, system administrators and DBAs. Using our team of experts and monitoring techniques, Treehouse offers world-class, on-site and remote system support and DBA service at affordable prices. Treehouse Professional Services offer various levels of system monitoring and administration. Our team will monitor critical factors within your Software AG environment, apply preventive maintenance for your systems and databases, and keep your software up-to-date. We even provide z/OS systems programming services! We are strongly committed to our customers’ success and we deliver the highest quality DBA and support solutions.

Visit the Treehouse Software website for more information on Treehouse Professional Services, or contact us to discuss your needs.


About Treehouse Software’s tRelational / DPS Product Set tReDPSMainDiagram01

tRelational / DPS is a robust product set that provides modeling and data transfer of legacy ADABAS data into modern RDBMS-based platforms for Internet/Intranet/Business Intelligence applications. Treehouse Software designed these products to meet the demands of large, complex environments requiring product maturity, productivity, feature-richness, efficiency and high performance.

The tRelational component provides complete analysis, modeling and mapping of ADABAS files and data elements to the target RDBMS tables and columns. DPS (Data Propagation System) performs Extract, Transformation, and Load (ETL) functions for the initial bulk RDBMS load and incremental Change Data Capture (CDC) batch processing to synchronize ADABAS updates with the target RDBMS.

Visit the Treehouse Software website for more information on tRelational / DPS, or contact us to discuss your needs.