Migration
Teradata to Databricks migration: a wave-by-wave plan that avoids the big bang
How to move a Teradata estate to the Databricks lakehouse in waves (inventory, code conversion, parallel runs and decommissioning) without losing the business's trust in the numbers.
Hemant Walchale
· 3 min read
Most Teradata migrations don’t fail on technology. They fail when the business loses trust in the numbers halfway through: a report that used to match finance’s figures no longer does, nobody can explain why, and the program stalls with two warehouses running and two bills to pay.
The fix is not a better conversion tool. It’s a plan that proves every number before anyone is asked to switch. Here’s the structure we use.
1. Inventory before you estimate
The single biggest source of overruns is an estate that’s larger than anyone thought. Before quoting a timeline, build a complete inventory:
- Objects: tables, views, macros, stored procedures, BTEQ and TPT scripts, and their sizes.
- Jobs: every scheduled load and transformation, its schedule and its runtime.
- Consumers: BI reports, extracts, downstream applications and models, and who owns each one.
- Dependencies: which objects feed which, so you can see what has to move together.
The inventory usually surfaces a large share of objects nobody uses. Retiring them before migration is the cheapest saving in the whole program.
2. Group the estate into waves
A wave is a set of objects that can move, be validated and be cut over together without breaking anything that stays behind. Good waves follow business domains (finance, risk, customer) rather than technical layers, because domain owners can then sign off their own numbers.
Order waves by dependency first, value second. Start with a domain that matters but isn’t the most fragile: you want the first wave to build confidence, not to be a rescue.
3. Convert code with tools, then test like it’s hand-written
Conversion tooling (including Databricks’ own migration tools) handles most Teradata SQL, BTEQ and stored procedures. The remainder needs engineers: procedural logic, Teradata-specific functions, QUALIFY-heavy reporting queries with odd edge cases, and volatile tables used to chain steps together.
Treat converted code as untested code. Every object gets the same CI pipeline and data tests as anything written from scratch.
4. Run both platforms side by side and reconcile everything
This is the step most programs shortcut, and the one that makes the rest work. For each wave:
- Load the same source data into both Teradata and Databricks.
- Run both pipelines on the same schedule.
- Compare outputs automatically: row counts, column-level checksums, and a handful of business metrics the domain owner cares about (e.g. month-end balances, policy counts).
- Publish a reconciliation report per table, per day.
A wave is ready to cut over when its reconciliation has been clean for an agreed number of runs and the domain owner has seen the report. That report is what keeps the business’s trust.
5. Cut over consumers, then decommission on a date
Switch consumers wave by wave: reports first, then extracts, then downstream applications. Keep Teradata read-only for a short fallback window, then switch it off on the planned date.
Put decommission dates in the plan from day one. A Teradata renewal date is the most effective deadline a migration program can have, and the licence saving is usually the clearest number in the business case.
What a realistic timeline looks like
| Phase | Typical duration |
|---|---|
| Inventory, wave plan and target architecture | 2–3 weeks |
| First (pilot) wave to production | 4–6 weeks |
| Remaining waves, in parallel streams | Depends on estate size |
| Final decommission | 2–4 weeks |
For context on outcomes, see our financial services benchmark of published modernization programs and the cross-industry outcomes report. Whatever the published figures, the saving only arrives once the legacy warehouse is switched off rather than kept running “for now”.
The short version
Inventory everything, migrate in domain waves, reconcile every table in a parallel run, and decommission on a date. If you want a second opinion on your own estate, the Data Platform Assessment produces exactly this plan, with a price on each wave.
Moving ETL as well? See our guides to Informatica PowerCenter to Databricks and converting Pentaho (PDI) jobs to Databricks.
Not sure where your own platform stands? Take the free 3-minute data platform scorecard, or see how we work on Databricks. Unfamiliar terms are explained in the data engineering glossary.