Migration
Hadoop to Databricks migration: moving HDFS, Hive, Spark and Oozie
How to migrate a Hadoop (Cloudera or Hortonworks) estate to Databricks: HDFS to cloud storage, Hive tables to Delta, Hive metastore and Ranger to Unity Catalog, Spark and Oozie jobs to Lakeflow Jobs.
Hemant Walchale
· 3 min read
Many Hadoop clusters, Cloudera and Hortonworks alike, now run on hardware past its prime, with a shrinking pool of people who know how to keep them alive. Databricks is a natural destination because it runs Apache Spark, which many Hadoop jobs already use. The migration is less about rewriting and more about storage, metadata, security and scheduling.
This is the approach we recommend.
1. Inventory the cluster
List HDFS directories and sizes, Hive databases and tables (with format and partitioning), Spark, MapReduce, Pig and Hive jobs, Oozie workflows and coordinators, Sqoop imports, Ranger or Sentry policies, and the consumers of each dataset. Cluster audit logs show what is actually read. Old HDFS data that nobody reads can go to cheap archive storage instead of the lakehouse.
2. Map each Hadoop component
| Hadoop | On Databricks |
|---|---|
| HDFS | Cloud object storage (S3, ADLS or GCS) |
| Hive tables (Parquet, ORC, text) | Delta tables; CONVERT TO DELTA for Parquet |
| Hive metastore | Unity Catalog (the Databricks Labs UCX tool helps with the upgrade) |
| Ranger or Sentry policies | Unity Catalog grants, row filters and column masks |
| Hive SQL, Impala queries | Spark SQL and Databricks SQL warehouses |
| Spark jobs | Spark jobs on Databricks, with updated paths and libraries |
| MapReduce, Pig | Rewritten in Spark SQL or PySpark |
| Oozie workflows and coordinators | Lakeflow Jobs with schedules and file-arrival triggers |
| Sqoop imports | Lakeflow Connect, JDBC reads or change data capture |
| HBase | Delta tables for analytics, or an operational database |
| Kafka | Keeps running; read with Structured Streaming or Lakeflow Declarative Pipelines |
3. Move data in waves
Copy data with distcp or a cloud transfer service, starting with the datasets behind one business domain. Convert to Delta, register in Unity Catalog, and re-create permissions there. Partitioning schemes designed for HDFS (many small partitions) are often worth simplifying: Delta handles file sizes and statistics better than Hive did.
4. Port jobs and schedules
Update Spark jobs for cloud paths and the target runtime, rewrite MapReduce and Pig, and rebuild Oozie workflows as Lakeflow Jobs. Use job compute (not shared all-purpose clusters) for scheduled work, which is usually the biggest cost saving after the cluster itself.
5. Reconcile and switch off nodes
Run old and new side by side per domain, compare outputs table by table, and cut over when they match and owners sign off. Decommission cluster capacity as domains move, so savings start before the last wave.
The short version
Inventory the cluster, archive what nobody reads, move data to cloud storage as Delta, put Unity Catalog in place of the Hive metastore and Ranger, port Spark and rewrite MapReduce and Pig, rebuild Oozie as Lakeflow Jobs and shrink the cluster domain by domain.
Browse all migration guides. For a fixed-price plan for your own cluster, see the Data Platform Assessment.
Not sure where your own platform stands? Take the free 3-minute data platform scorecard, or see how we work on Databricks. Unfamiliar terms are explained in the data engineering glossary.