IBM Mainframe (IBM Z) GCP Migration

IBM Mainframe (IBM Z) GCP Migration

N Br*wn (NBG), 2024
This project modernized

NBG’s legacy IBM Z Mainframe data warehouse

, replacing its

Teradata-based infrastructure

with a

scalable, automated ETL pipeline on GCP

, fully funded by Google (£110K).

The new

prod-ready ETL pipeline

, completed in just

one month

, extracts K90 data from IBM DB2, transfers it via SFTP, and ingests it into Cloud Storage using the GCP Mainframe Connector. Data is processed in Cloud Run, validated in BigQuery, and automated with Cloud Composer to ensure repeatability and scalability.

This first-of-its-kind migration at NBG improved efficiency, enhanced data accessibility, and is projected to save

£2M in Teradata licensing over two years

, while establishing

standardized naming conventions and best practices

to support future cloud transformations.

  • IBM Mainframe (IBM Z) GCP Migration
  • N Br*wn (NBG) - 160-year-old UK top 10 retailer
  • Head of Data Engineering
  • 2024
  • Teamwork with Google (2 from NBG, 4 from Google)
  • £110K project fully refunded by Google
  • Manchester, England, United Kingdom
  • 1st Prod-ready Mainframe-to-GCP migration done in one month
  • Potential £2M Teradata savings over two years at NBG
  • Established naming conventions & best practices at NBG
  • Google Cloud Platform (GCP)
  • IBM Mainframe (IBM Z)
  • GCP Cloud Run
  • GCP Big Query
  • GCP Cloud Composer (Airflow)
  • Terraform CI/CD (IaC)
  • GCP Dataplex
  • GCP Logging and Monitoring

Current Legacy Data Warehouse

NBG Current Legacy Teradata Data Warehouse on IBM Z Mainframe
[1] NBG Current Legacy Teradata Data Warehouse on IBM Z Mainframe.

NBG Current Legacy Teradata Data Warehouse on IBM Z Mainframe

NBG integrates

IBM Mainframe (IBM Z)

data with modern cloud infrastructure and

Teradata CIM

for advanced data processing and customer engagement. K90 Extracts from IBM DB2 are securely processed within the IBM Z Cloud environment before being ingested into Teradata CIM, running on virtual machines at Griffin House (NBG HQ Office). Users interact with CIM via a desktop interface for analytics and customer management.

Processed data is then distributed to

AWS

,

GCP

, and

Azure

for further analytics or routed to

Mailing House

and

Oracle Responsys

for marketing campaigns. This hybrid cloud architecture enhances scalability, efficiency, and data-driven decision-making while optimizing legacy mainframe data for modern applications.

Prod-Ready GCP Migration via Mainframe Connector

Prod-Ready GCP Migration via Mainframe Connectorr
[2] Prod-Ready GCP Migration via Mainframe Connector

From Legacy to Prod-Ready GCP Migration: ETL Pipeline via Mainframe Connector

The

Mainframe-to-GCP migration

leverages

Infrastructure as Code (IaC) with Terraform

to ensure scalability, security, and cost efficiency. By automating infrastructure provisioning, the solution modularizes components such as

BigQuery, Cloud Run, and Dataplex

, enabling seamless integration with cloud services.

The migration process starts with

extracting mainframe data (K90 Extracts) from IBM DB2

, transferred via

SFTP

. Using the

GCP Mainframe Connector

, data is loaded into

Cloud Storage

, processed by

Cloud Run

, validated in

BigQuery

, and structured using generated schemas.

For automation,

Cloud Composer

orchestrates pipeline execution. New files are onboarded by configuring

environment variables

, updating

DAG templates

, and uploading necessary scripts. This

scalable, repeatable architecture

modernizes legacy mainframe workloads, enhancing data accessibility and operational efficiency in GCP.