Databricks
Auto Loader schema evolution: what each mode really does
How Databricks Auto Loader handles new columns and type changes: why addNewColumns stops the stream, what goes into _rescued_data, schema hints, inferColumnTypes, type widening and how to run it in production.

Hemant Walchale
· 4 min read
Auto Loader is the standard way to ingest files on Databricks, and schema evolution is the part teams most often get wrong. The usual complaint is “the stream keeps failing when the source adds a column”. That’s not a bug: it’s how the default mode is meant to work.
The modes at a glance
Set with cloudFiles.schemaEvolutionMode:
| Mode | New column appears | Stream behaviour |
|---|---|---|
addNewColumns (default when inferring) |
Added to the schema | Stops once, then continues after restart |
addNewColumnsWithTypeWidening |
Added, and compatible types are widened | Stops on change, continues after restart. Public Preview, Runtime 16.4+ |
rescue |
Not added; values go into _rescued_data |
Never stops for schema changes |
failOnNewColumns |
Not added | Fails until you update the schema |
none |
Ignored | Never stops; new columns are dropped unless rescued |
Why addNewColumns stops the stream
When Auto Loader sees a column that isn’t in its schema, it can’t change the running stream’s schema mid-flight. So it:
- records the new schema in
cloudFiles.schemaLocation, - stops the stream with an
UnknownFieldException, - uses the new schema when the stream starts again.
So “it stops” is expected. The fix is to make the restart automatic: run the stream as a Lakeflow job with retries, or in a Lakeflow declarative pipeline, which restarts for you. Don’t wrap it in a loop that swallows every exception, or real failures will look like schema changes.
The schema location isn’t optional
When Auto Loader infers the schema, it needs cloudFiles.schemaLocation. That’s where the inferred schema and its history live. Without it, Auto Loader would re-infer on every restart and could reach a different answer. Keep one schema location per stream, next to its checkpoint, and never share it between streams.
Type changes are the silent problem
addNewColumns adds columns. It does not change the type of an existing column. If amount was inferred as bigint and the source starts sending "12.50", those values don’t fit, and with a rescued data column they go into _rescued_data instead of amount. The stream doesn’t fail and nothing alerts. The column just fills with NULLs.
What to do:
- Monitor
_rescued_data. Count non-null values per batch and alert above zero for important tables. - Use schema hints for the columns you care about:
cloudFiles.schemaHints = "amount DECIMAL(18,2), order_id BIGINT, created_at TIMESTAMP". Hints win over inference, so a bad early sample can’t pick the wrong type. - Consider type widening on Runtime 16.4 and above.
addNewColumnsWithTypeWideningwidens compatible types, such asinttolongorfloattodouble, without a rewrite. The target Delta table needs type widening enabled. It’s in Public Preview, so test it first.
inferColumnTypes: strings or types?
For JSON and CSV, Auto Loader infers every column as string unless cloudFiles.inferColumnTypes is true. That’s a deliberate trade-off:
- All strings never fail on type changes, but every downstream query has to cast, and bad values surface later, in silver or gold.
- Inferred types catch problems at ingestion, but the inference depends on the files in the first sample.
A common middle path: keep bronze as strings (or VARIANT for nested JSON), and apply types and checks when building silver. If you infer types in bronze, add schema hints for keys, amounts and timestamps.
A production setup that holds up
(spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", "/Volumes/raw/orders/_schema")
.option("cloudFiles.inferColumnTypes", "true")
.option("cloudFiles.schemaHints", "order_id BIGINT, amount DECIMAL(18,2), created_at TIMESTAMP")
.option("cloudFiles.schemaEvolutionMode", "addNewColumns")
.load("/Volumes/raw/orders/landing")
.writeStream
.option("checkpointLocation", "/Volumes/raw/orders/_checkpoint")
.option("mergeSchema", "true")
.trigger(availableNow=True)
.toTable("bronze.orders"))
Run it as a job task with retries, alert on rows in _rescued_data, and review the schema location’s history when columns are added. If a stream must never stop, use rescue mode and promote fields out of _rescued_data deliberately.
The short version
addNewColumns stops the stream once per new column by design, so automate the restart. The schema location is required when inferring. Type changes don’t fail: they fill _rescued_data, so monitor it and use schema hints for important columns. Related: idempotent Databricks jobs, what is medallion architecture and what is data observability.
Sources
- Configure schema inference and evolution in Auto Loader, Databricks documentation
- Type widening in Auto Loader, Databricks documentation
- Auto Loader options, Databricks documentation
- What is Auto Loader?, Databricks documentation
Not sure where your own platform stands? Take the free 3-minute data platform scorecard, or see how we work on Databricks. Need people to do this alongside your team? See senior data engineers for hire. Unfamiliar terms are explained in the data engineering glossary.