Migration
Hadoop to Snowflake migration: moving Hive, Spark and HDFS data
How to migrate a Hadoop (Cloudera or Hortonworks) cluster to Snowflake: HDFS data to Snowflake or Iceberg tables, Hive SQL conversion, Spark jobs to SQL or Snowpark, Oozie to Tasks, Ranger to Snowflake access policies.
Hemant Walchale
· 2 min read
Hadoop clusters built for big data a decade ago often now serve mostly SQL analytics, which a managed warehouse handles with far less effort. Snowflake is a common destination when the main users are analysts and BI tools rather than data scientists. The move is mostly about data, Hive SQL and replacing cluster-era tooling.
This is the approach we recommend.
1. Inventory and archive
List HDFS directories, Hive databases and tables (format and partitioning), Hive, Impala, Spark, MapReduce and Pig jobs, Oozie workflows, Sqoop imports, Ranger or Sentry policies, Kafka topics and consumers. Archive data nobody reads to cheap object storage instead of loading it.
2. Map Hadoop to Snowflake
| Hadoop | On Snowflake |
|---|---|
| HDFS files (Parquet, ORC, Avro) | Cloud storage, loaded with COPY INTO, or Iceberg tables |
| Hive tables and partitions | Snowflake tables; clustering keys only where measured |
| Hive metastore | Snowflake databases and schemas (or an Iceberg catalog) |
| Hive and Impala SQL | Snowflake SQL; most converts with small changes |
| Spark jobs | SQL, Snowpark, or Spark elsewhere via the Spark connector |
| MapReduce, Pig | Rewritten in SQL or Snowpark |
| Oozie workflows | Snowflake Tasks, dbt or an orchestrator such as Airflow |
| Sqoop | Replication or CDC tools, connectors |
| Ranger or Sentry | Role-based access control, masking and row access policies |
| Kafka ingestion | Snowflake Kafka connector, Snowpipe Streaming |
| HBase | An operational database; analytics copies in Snowflake |
3. Move the data
Copy data from HDFS to cloud storage (distcp, a transfer service or a physical appliance for very large volumes), then load it with COPY INTO or register it as Iceberg tables. Simplify Hive’s deep partition hierarchies: Snowflake prunes micro-partitions automatically.
4. Convert SQL and jobs
Hive SQL mostly converts directly; test date functions, LATERAL VIEW explode (which becomes FLATTEN), complex types (which become VARIANT, ARRAY and OBJECT) and implicit casts. Rewrite MapReduce and Pig, and convert Spark jobs to SQL or Snowpark where practical.
5. Reconcile and shrink the cluster
Run old and new side by side per domain, compare every table, cut over when owners sign off, and decommission nodes as domains move.
The short version
Archive what’s unused, move data to cloud storage and load it into Snowflake or Iceberg tables, convert Hive SQL, move Spark to SQL or Snowpark, replace Oozie with Tasks and Ranger with Snowflake policies, reconcile and shrink the cluster by domain.
Compare with Hadoop to Databricks and Hadoop to Microsoft Fabric, or try the platform chooser.
Not sure where your own platform stands? Take the free 3-minute data platform scorecard, or see how we work on Snowflake. Unfamiliar terms are explained in the data engineering glossary.