Skip to content
METRIC OAK

New: the 3-week Data Platform Assessment: fixed fee, priced roadmap. See what's included →

Databricks

Are you paying for real time nobody uses? Databricks freshness vs cost

How to decide between batch, triggered, table-triggered and continuous pipelines on Databricks, measure when tables are read versus refreshed with system tables, and set freshness tiers the business will pay for.

Hemant Walchale

Hemant Walchale
· 5 min read

Every Databricks team gets asked for “real time”. In a recent r/databricks thread, an engineer asked whether teams are paying for 2-minute CDC and continuous pipelines when the business opens the dashboard once a morning. Twenty-five replies mostly agreed. Nobody explained how to measure the gap. This page covers both: the options between batch and streaming, and the SQL that shows when tables are read compared with when they refresh.

The short answer

Start every table at hourly or daily. Move it faster only when someone can name the decision that gets worse if the data is an hour old, and the cost of the faster option is written next to it. Most dashboards don’t need streaming: they need a pipeline that reliably finishes before people look.

The options between batch and streaming

Option Typical freshness Compute between runs Fits
Daily or hourly job Hours None Finance, executive and planning dashboards
Triggered pipeline on a schedule 10 minutes to hourly None Operational reporting during the working day
Job with a table update trigger Minutes after the source lands None (pays for storage listing only) “Fresh as soon as the file or table arrives”
Continuous job wrapping a triggered pipeline Seconds to a few minutes Always on Customer-facing flows, fraud, control systems
Pipeline in continuous mode Seconds to a few minutes Always on Same, but Databricks now recommends the continuous job instead

The middle rows are the ones most threads skip. A triggered pipeline processes what’s new and shuts down. A table update trigger starts a job when the upstream table actually changes, not on a clock, so a late source doesn’t waste a run and an early one isn’t left waiting. See data-aware job triggers on Databricks for how they compare with condition and polling patterns.

Databricks’ pipeline mode guidance puts triggered mode at 10 minutes, hourly or daily, and continuous at 10 seconds to a few minutes.

What 2-minute CDC really costs

The compute bill is the obvious part. The rest shows up later:

  • Always-on clusters. A continuous pipeline keeps compute running overnight and at weekends, when nobody reads the output.
  • Small files. Frequent micro-batches write many small files, so you pay again for OPTIMIZE and compaction.
  • Source-side costs. Longer log retention on the database, handling deletes and ordering, and backfills after an outage.
  • On-call. A pipeline that must be fresh within minutes pages someone when it isn’t.

Measure reads against refreshes

This is the question the thread left open. system.access.table_lineage records each read and write against a table, with the type of entity (job, pipeline, notebook, dashboard, SQL query) and the time. Compare the two per table:

WITH writes AS (
  SELECT target_table_full_name AS t, count(*) AS writes_30d, max(event_time) AS last_write
  FROM system.access.table_lineage
  WHERE event_time >= current_date() - INTERVAL 30 DAYS AND target_table_full_name IS NOT NULL
  GROUP BY 1
), reads AS (
  SELECT source_table_full_name AS t, count(*) AS reads_30d, max(event_time) AS last_read,
         count(DISTINCT date(event_time)) AS days_read
  FROM system.access.table_lineage
  WHERE event_time >= current_date() - INTERVAL 30 DAYS AND source_table_full_name IS NOT NULL
  GROUP BY 1
)
SELECT w.t, w.writes_30d, coalesce(r.reads_30d, 0) AS reads_30d, coalesce(r.days_read, 0) AS days_read, r.last_read
FROM writes w LEFT JOIN reads r USING (t)
ORDER BY w.writes_30d / greatest(coalesce(r.reads_30d, 0), 1) DESC;

A table written 8,000 times a month and read on 20 days is a candidate for an hourly or morning schedule. A table written daily and read on zero days is a candidate for switching off.

Then put a price on it. Group system.billing.usage by job or pipeline ID (the system tables cost audit has the query) and divide by reads. “This pipeline costs $1,900 a month and its table is read 22 times” is a sentence a business owner can decide on.

Caveat: lineage only sees access through Unity Catalog. Path-based reads and external engines may not appear, so check with the table’s owner before switching anything off.

Set freshness tiers, not per-table arguments

Agree three or four tiers with the business and put every table in one:

Tier Freshness How it runs Who can ask for it
Daily Ready by 07:00 Scheduled job Default for everything
Working hours Every 30 to 60 minutes, 07:00 to 19:00 Triggered pipeline on a schedule Team lead
Near real time Minutes Table update trigger or continuous job Written business case and named budget
Real time Seconds Continuous job, streaming Customer-facing or safety use only

Two rules keep it honest: anything faster than “working hours” needs a named owner who signs off the cost, and every tier is reviewed against the read-versus-refresh query each quarter.

Check the whole path

Freshness is set by the slowest step. If the Power BI model imports eight times a day, a 2-minute pipeline upstream buys nothing. The same applies to cached dashboards and extracts. Measure from source commit to the number on screen before buying speed in the middle.

The opposite view in the thread is fair too: support, onboarding and buying-signal flows lose value by the minute. Tiers aren’t about refusing real time. They make sure it’s spent where someone acts on it.

Not sure where your own platform stands? Take the free 3-minute data platform scorecard, or see how we work on Databricks. Need people to do this alongside your team? See how our consulting engagements work. Unfamiliar terms are explained in the data engineering glossary.

Frequently asked questions

When is streaming or CDC worth it on Databricks?

When a decision or a customer-facing action gets worse if the data is minutes old: support queues, fraud checks, onboarding flows, stock or control systems. For dashboards opened once a morning, a triggered pipeline that finishes before the first report gives the same answer for a fraction of the compute.

What is between a daily batch job and a continuous pipeline?

A triggered pipeline on a 10 to 15 minute or hourly schedule, and a job with a table update trigger that runs only when the upstream table changes. Neither keeps a cluster up between runs. Databricks' own guidance puts triggered mode at 10 minutes to daily and continuous at seconds to a few minutes.

How do I find tables that refresh more often than anyone reads them?

Compare write events with read events per table in system.access.table_lineage, and add warehouse reads from system.query.history. Tables written every few minutes but read once a day, or not read in 90 days, are the first to slow down or switch off.

Does a faster pipeline make the dashboard fresher?

Only if nothing downstream caps it. A Power BI import model refreshed eight times a day, or a cached dashboard, sets the real latency. Measure end to end before paying for streaming upstream.

Have 30 minutes? Get a senior architect’s view of your data platform.

No sales deck. You talk to the engineer who would lead the work.

Book an architecture review
Hemant WalchaleYou talk to Hemant Walchale, founder145 published guidesProfiles: LinkedIn · Clutch · GoodFirms

Free, takes 20 seconds

Get your starter data platform roadmap

Pick your platform and your main goal. You get a 90-day plan written by senior data engineers: what to do first, next and after.

You’ll also get our monthly email (confirm it in your inbox). Unsubscribe any time. Privacy policy.