Migration
Hadoop to Microsoft Fabric migration: HDFS, Hive and Spark to OneLake
How to migrate a Hadoop (Cloudera or Hortonworks) cluster to Microsoft Fabric: HDFS to OneLake, Hive tables to Lakehouse Delta tables, Spark jobs to Fabric Spark, Oozie to Data Factory, Kafka to Real-Time Intelligence.
Hemant Walchale
· 2 min read
Ageing on-premises Hadoop clusters are expensive to keep alive, and Microsoft-centric organizations increasingly move them to Fabric rather than to another cluster. Because Fabric runs Spark and stores Delta tables in OneLake, much Hadoop logic carries over. The work is in storage, metadata, security and scheduling.
This is the approach we recommend.
1. Inventory the cluster
List HDFS directories and sizes, Hive databases and tables (format and partitioning), Spark, MapReduce, Pig and Hive jobs, Oozie workflows, Sqoop imports, Ranger or Sentry policies, Kafka topics and the consumers of each dataset. Archive data nobody reads to cheap storage instead of the Lakehouse.
2. Map Hadoop to Fabric
| Hadoop | Microsoft Fabric |
|---|---|
| HDFS | OneLake (Lakehouse Files and Tables) |
| Hive tables (Parquet, ORC, text) | Lakehouse Delta tables |
| Hive metastore | Lakehouse table catalog |
| Hive and Impala queries | Lakehouse SQL analytics endpoint (read-only T-SQL), or a Fabric Warehouse |
| Spark jobs | Fabric notebooks and Spark job definitions |
| MapReduce, Pig | Rewritten in Spark SQL or PySpark |
| Oozie workflows and coordinators | Data Factory pipelines with schedules and triggers |
| Sqoop | Copy activity, Dataflow Gen2 or mirroring |
| Ranger or Sentry policies | Workspace roles, item permissions and OneLake security |
| Kafka, Spark Streaming | Eventstreams, Eventhouse or Spark Structured Streaming |
| HBase | Delta tables for analytics, or an operational database |
3. Move data in waves
Copy data from HDFS to ADLS Gen2 or directly into OneLake with distcp, Azure Data Box for very large volumes, or Data Factory copy activities. Write tables as Delta and simplify over-partitioned Hive layouts; Delta manages file sizes and statistics far better than Hive did.
4. Port jobs and reconcile
Update Spark jobs for OneLake paths and the Fabric runtime, rewrite MapReduce and Pig, and rebuild Oozie workflows as pipelines. Run old and new side by side per domain and compare outputs table by table before cutting over.
5. Shrink the cluster
Decommission nodes as domains move so savings start before the last wave, and stop paying for Hadoop support at the end.
The short version
Archive what nobody reads, move data to OneLake as Delta, query it through the SQL analytics endpoint, port Spark jobs and rewrite MapReduce and Pig, replace Oozie with pipelines and Ranger with Fabric permissions, and shrink the cluster domain by domain.
Compare with Hadoop to Databricks or Databricks vs Microsoft Fabric. Browse all migration guides.
Not sure where your own platform stands? Take the free 3-minute data platform scorecard. Unfamiliar terms are explained in the data engineering glossary.