Databricks
Are you paying for real time nobody uses? Databricks freshness vs cost
How to decide between batch, triggered, table-triggered and continuous pipelines on Databricks, measure when tables are read versus refreshed with system tables, and set freshness tiers the business will pay for.

Hemant Walchale
· 5 min read
Every Databricks team gets asked for “real time”. In a recent r/databricks thread, an engineer asked whether teams are paying for 2-minute CDC and continuous pipelines when the business opens the dashboard once a morning. Twenty-five replies mostly agreed. Nobody explained how to measure the gap. This page covers both: the options between batch and streaming, and the SQL that shows when tables are read compared with when they refresh.
The short answer
Start every table at hourly or daily. Move it faster only when someone can name the decision that gets worse if the data is an hour old, and the cost of the faster option is written next to it. Most dashboards don’t need streaming: they need a pipeline that reliably finishes before people look.
The options between batch and streaming
| Option | Typical freshness | Compute between runs | Fits |
|---|---|---|---|
| Daily or hourly job | Hours | None | Finance, executive and planning dashboards |
| Triggered pipeline on a schedule | 10 minutes to hourly | None | Operational reporting during the working day |
| Job with a table update trigger | Minutes after the source lands | None (pays for storage listing only) | “Fresh as soon as the file or table arrives” |
| Continuous job wrapping a triggered pipeline | Seconds to a few minutes | Always on | Customer-facing flows, fraud, control systems |
| Pipeline in continuous mode | Seconds to a few minutes | Always on | Same, but Databricks now recommends the continuous job instead |
The middle rows are the ones most threads skip. A triggered pipeline processes what’s new and shuts down. A table update trigger starts a job when the upstream table actually changes, not on a clock, so a late source doesn’t waste a run and an early one isn’t left waiting. See data-aware job triggers on Databricks for how they compare with condition and polling patterns.
Databricks’ pipeline mode guidance puts triggered mode at 10 minutes, hourly or daily, and continuous at 10 seconds to a few minutes.
What 2-minute CDC really costs
The compute bill is the obvious part. The rest shows up later:
- Always-on clusters. A continuous pipeline keeps compute running overnight and at weekends, when nobody reads the output.
- Small files. Frequent micro-batches write many small files, so you pay again for
OPTIMIZEand compaction. - Source-side costs. Longer log retention on the database, handling deletes and ordering, and backfills after an outage.
- On-call. A pipeline that must be fresh within minutes pages someone when it isn’t.
Measure reads against refreshes
This is the question the thread left open. system.access.table_lineage records each read and write against a table, with the type of entity (job, pipeline, notebook, dashboard, SQL query) and the time. Compare the two per table:
WITH writes AS (
SELECT target_table_full_name AS t, count(*) AS writes_30d, max(event_time) AS last_write
FROM system.access.table_lineage
WHERE event_time >= current_date() - INTERVAL 30 DAYS AND target_table_full_name IS NOT NULL
GROUP BY 1
), reads AS (
SELECT source_table_full_name AS t, count(*) AS reads_30d, max(event_time) AS last_read,
count(DISTINCT date(event_time)) AS days_read
FROM system.access.table_lineage
WHERE event_time >= current_date() - INTERVAL 30 DAYS AND source_table_full_name IS NOT NULL
GROUP BY 1
)
SELECT w.t, w.writes_30d, coalesce(r.reads_30d, 0) AS reads_30d, coalesce(r.days_read, 0) AS days_read, r.last_read
FROM writes w LEFT JOIN reads r USING (t)
ORDER BY w.writes_30d / greatest(coalesce(r.reads_30d, 0), 1) DESC;
A table written 8,000 times a month and read on 20 days is a candidate for an hourly or morning schedule. A table written daily and read on zero days is a candidate for switching off.
Then put a price on it. Group system.billing.usage by job or pipeline ID (the system tables cost audit has the query) and divide by reads. “This pipeline costs $1,900 a month and its table is read 22 times” is a sentence a business owner can decide on.
Caveat: lineage only sees access through Unity Catalog. Path-based reads and external engines may not appear, so check with the table’s owner before switching anything off.
Set freshness tiers, not per-table arguments
Agree three or four tiers with the business and put every table in one:
| Tier | Freshness | How it runs | Who can ask for it |
|---|---|---|---|
| Daily | Ready by 07:00 | Scheduled job | Default for everything |
| Working hours | Every 30 to 60 minutes, 07:00 to 19:00 | Triggered pipeline on a schedule | Team lead |
| Near real time | Minutes | Table update trigger or continuous job | Written business case and named budget |
| Real time | Seconds | Continuous job, streaming | Customer-facing or safety use only |
Two rules keep it honest: anything faster than “working hours” needs a named owner who signs off the cost, and every tier is reviewed against the read-versus-refresh query each quarter.
Check the whole path
Freshness is set by the slowest step. If the Power BI model imports eight times a day, a 2-minute pipeline upstream buys nothing. The same applies to cached dashboards and extracts. Measure from source commit to the number on screen before buying speed in the middle.
The opposite view in the thread is fair too: support, onboarding and buying-signal flows lose value by the minute. Tiers aren’t about refusing real time. They make sure it’s spent where someone acts on it.
Not sure where your own platform stands? Take the free 3-minute data platform scorecard, or see how we work on Databricks. Need people to do this alongside your team? See how our consulting engagements work. Unfamiliar terms are explained in the data engineering glossary.