Skip to content
METRIC OAK

New: the 3-week Data Platform Assessment: fixed fee, priced roadmap. See what's included →

Databricks

Auto Loader schema evolution: what each mode really does

How Databricks Auto Loader handles new columns and type changes: why addNewColumns stops the stream, what goes into _rescued_data, schema hints, inferColumnTypes, type widening and how to run it in production.

Hemant Walchale

Hemant Walchale
· 4 min read

Auto Loader is the standard way to ingest files on Databricks, and schema evolution is the part teams most often get wrong. The usual complaint is “the stream keeps failing when the source adds a column”. That’s not a bug: it’s how the default mode is meant to work.

The modes at a glance

Set with cloudFiles.schemaEvolutionMode:

Mode New column appears Stream behaviour
addNewColumns (default when inferring) Added to the schema Stops once, then continues after restart
addNewColumnsWithTypeWidening Added, and compatible types are widened Stops on change, continues after restart. Public Preview, Runtime 16.4+
rescue Not added; values go into _rescued_data Never stops for schema changes
failOnNewColumns Not added Fails until you update the schema
none Ignored Never stops; new columns are dropped unless rescued

Why addNewColumns stops the stream

When Auto Loader sees a column that isn’t in its schema, it can’t change the running stream’s schema mid-flight. So it:

  1. records the new schema in cloudFiles.schemaLocation,
  2. stops the stream with an UnknownFieldException,
  3. uses the new schema when the stream starts again.

So “it stops” is expected. The fix is to make the restart automatic: run the stream as a Lakeflow job with retries, or in a Lakeflow declarative pipeline, which restarts for you. Don’t wrap it in a loop that swallows every exception, or real failures will look like schema changes.

The schema location isn’t optional

When Auto Loader infers the schema, it needs cloudFiles.schemaLocation. That’s where the inferred schema and its history live. Without it, Auto Loader would re-infer on every restart and could reach a different answer. Keep one schema location per stream, next to its checkpoint, and never share it between streams.

Type changes are the silent problem

addNewColumns adds columns. It does not change the type of an existing column. If amount was inferred as bigint and the source starts sending "12.50", those values don’t fit, and with a rescued data column they go into _rescued_data instead of amount. The stream doesn’t fail and nothing alerts. The column just fills with NULLs.

What to do:

  • Monitor _rescued_data. Count non-null values per batch and alert above zero for important tables.
  • Use schema hints for the columns you care about: cloudFiles.schemaHints = "amount DECIMAL(18,2), order_id BIGINT, created_at TIMESTAMP". Hints win over inference, so a bad early sample can’t pick the wrong type.
  • Consider type widening on Runtime 16.4 and above. addNewColumnsWithTypeWidening widens compatible types, such as int to long or float to double, without a rewrite. The target Delta table needs type widening enabled. It’s in Public Preview, so test it first.

inferColumnTypes: strings or types?

For JSON and CSV, Auto Loader infers every column as string unless cloudFiles.inferColumnTypes is true. That’s a deliberate trade-off:

  • All strings never fail on type changes, but every downstream query has to cast, and bad values surface later, in silver or gold.
  • Inferred types catch problems at ingestion, but the inference depends on the files in the first sample.

A common middle path: keep bronze as strings (or VARIANT for nested JSON), and apply types and checks when building silver. If you infer types in bronze, add schema hints for keys, amounts and timestamps.

A production setup that holds up

(spark.readStream.format("cloudFiles")
   .option("cloudFiles.format", "json")
   .option("cloudFiles.schemaLocation", "/Volumes/raw/orders/_schema")
   .option("cloudFiles.inferColumnTypes", "true")
   .option("cloudFiles.schemaHints", "order_id BIGINT, amount DECIMAL(18,2), created_at TIMESTAMP")
   .option("cloudFiles.schemaEvolutionMode", "addNewColumns")
   .load("/Volumes/raw/orders/landing")
   .writeStream
   .option("checkpointLocation", "/Volumes/raw/orders/_checkpoint")
   .option("mergeSchema", "true")
   .trigger(availableNow=True)
   .toTable("bronze.orders"))

Run it as a job task with retries, alert on rows in _rescued_data, and review the schema location’s history when columns are added. If a stream must never stop, use rescue mode and promote fields out of _rescued_data deliberately.

The short version

addNewColumns stops the stream once per new column by design, so automate the restart. The schema location is required when inferring. Type changes don’t fail: they fill _rescued_data, so monitor it and use schema hints for important columns. Related: idempotent Databricks jobs, what is medallion architecture and what is data observability.

Sources

Not sure where your own platform stands? Take the free 3-minute data platform scorecard, or see how we work on Databricks. Need people to do this alongside your team? See senior data engineers for hire. Unfamiliar terms are explained in the data engineering glossary.

Frequently asked questions

Why does my Auto Loader stream fail when a new column appears?

That's the designed behaviour of the default addNewColumns mode. The stream stops with an UnknownFieldException, records the new schema in the schema location, and picks it up when restarted. Run the stream as a job with retries, or in a Lakeflow declarative pipeline, so the restart happens automatically.

Do I need cloudFiles.schemaLocation?

Yes, whenever Auto Loader infers the schema. It stores the inferred schema and every change to it there, so a restart doesn't infer again from scratch. Keep one schema location per stream.

What happens when a column's type changes?

addNewColumns doesn't change existing column types. Values that don't fit the current type go into the _rescued_data column, so monitor it. On Databricks Runtime 16.4 and above, the addNewColumnsWithTypeWidening mode (Public Preview) can widen types such as int to long.

Should I set cloudFiles.inferColumnTypes?

For JSON and CSV, Auto Loader infers every column as a string unless you set inferColumnTypes to true. Strings never fail, but push type problems downstream. Inferring types catches them earlier but depends on the sample, so pair it with schema hints for important columns.

Have 30 minutes? Get a senior architect’s view of your data platform.

No sales deck. You talk to the engineer who would lead the work.

Book an architecture review
Hemant WalchaleYou talk to Hemant Walchale, founder145 published guidesProfiles: LinkedIn · Clutch · GoodFirms

Free, takes 20 seconds

Get your starter data platform roadmap

Pick your platform and your main goal. You get a 90-day plan written by senior data engineers: what to do first, next and after.

You’ll also get our monthly email (confirm it in your inbox). Unsubscribe any time. Privacy policy.