Migration
IBM DataStage to Databricks migration: converting jobs, stages and sequences
How to move IBM InfoSphere DataStage parallel jobs and sequences to Databricks: export and inventory, map each stage to Spark SQL and Delta, rebuild sequences as Lakeflow Jobs and reconcile.
Hemant Walchale
· 2 min read
IBM InfoSphere DataStage runs ETL at many banks, insurers and retailers, often with thousands of parallel jobs built over many years. When the warehouse moves to Databricks, those jobs need rebuilding. Most stages have a direct Spark SQL equivalent; the effort concentrates in Transformer logic and routines.
This is the approach we recommend.
1. Export and inventory
Export projects with the DataStage Designer or istool export. The export records each stage’s type, so a count across the estate shows how much is standard and how much is custom. Also collect sequence jobs, parameter sets, BASIC routines, shared containers, the scheduler (DataStage Director, Control-M or another tool) and job run history. Retire jobs that no longer run.
2. Map stages to Spark SQL and Delta
| DataStage stage | On Databricks |
|---|---|
| DB2, Oracle, ODBC connectors (source) | Ingest into bronze Delta tables (Lakeflow Connect or JDBC) |
| Sequential File, Complex Flat File | Auto Loader; fixed-width files parsed in SQL or Python |
| Transformer | SELECT expressions, CASE, window functions; stage variables become window or running calculations |
| Filter, Switch | WHERE, or one query per output |
| Lookup, Join, Merge | Joins (broadcast small lookups) |
| Aggregator | GROUP BY |
| Sort, Remove Duplicates | Window functions such as ROW_NUMBER(), or DISTINCT |
| Funnel | UNION ALL |
| Pivot | PIVOT, UNPIVOT or stack |
| Change Capture, Change Apply | MERGE INTO, or AUTO CDC |
| Slowly Changing Dimension | AUTO CDC with SCD type 2, or MERGE |
| Surrogate Key Generator | Identity columns |
| Shared containers | Shared SQL views, functions or Python modules |
| BASIC routines, custom stages | Rewrite in SQL or Python; test each |
| Target connectors | Writes to Delta tables |
3. Rebuild sequences as Lakeflow Jobs
Job activities become tasks; triggers (OK, failed, conditional) become dependencies with run-if conditions; loop activities become for-each tasks; Execute Command activities become Python tasks; notifications become job notifications; parameter sets become job parameters kept in version control.
4. Reconcile and cut over by domain
Run old and new against the same sources and compare every target table. Typical differences come from Transformer type conversions, null handling, string padding and trimming, decimal precision and date formats. Cut over a domain when comparisons are clean and its owner signs off, and disable its DataStage jobs the same day.
The short version
Export and count stage types, map standard stages to SQL and Delta, rewrite Transformer logic and routines carefully, rebuild sequences as Lakeflow Jobs, drop partitioning settings, reconcile every target and retire jobs domain by domain.
See the similar Informatica to Databricks guide, browse all migration guides, or estimate effort with the migration estimator.
Not sure where your own platform stands? Take the free 3-minute data platform scorecard, or see how we work on Databricks. Unfamiliar terms are explained in the data engineering glossary.