Skip to content
METRIC OAK

New: the 3-week Data Platform Assessment: fixed fee, priced roadmap. See what's included →

Databricks

Jobs that check the data before they run: Databricks table update triggers vs SQL conditions vs polling

When to start a Databricks job on a table update trigger, when to add a SQL condition, and when a hand-built sentinel table is still the answer, with what each costs and how to define 'ready' for a messy upstream.

Hemant Walchale

Hemant Walchale
· 4 min read

Almost every Databricks team has hand-built the same thing: a job that wakes up on a cron, checks whether the upstream data is there, and exits if it isn’t. Databricks now has built-in versions of that pattern. A product manager asked on r/databricks what “ready” means for your worst upstream. This page answers that question and compares the three ways to wait for data.

The short answer

  • Table update trigger when “the upstream table changed” is enough. No polling and no extra charge beyond storage listing.
  • Table trigger plus a SQL condition when you need “changed and good”, for example a complete day or a row count in range.
  • A sentinel or audit table when the source is outside Unity Catalog, loads in several steps, or another team needs to tell you explicitly that it’s done.

How the three options compare

Table update trigger SQL condition Cron plus sentinel table
Starts the run when Listed tables change (any or all) A query returns the expected result Your check task finds the flag
Cost of waiting Cloud cost of listing and reading table updates only Billed as SQL warehouse usage per check A job run (often a cluster start) per poll
Knows if the data is complete No, only that it changed Yes, if you write the check Yes, if the source writes the flag
Works for non-UC sources No Only what the warehouse can query Yes
Limits Up to 25 tables per trigger (file events needed above 10) One condition per trigger; 5-minute timeout including warehouse start Whatever you build
Status No preview label in the docs at the time of writing Beta (admin enables it on the Previews page) Your own code

Databricks’ table update trigger documentation describes the any/all option, the “wait after last change” setting (more updates during the wait reset the timer), and the cost note. Supported sources include Unity Catalog managed and external Delta tables, materialized views, streaming tables and views. SQL conditions run read-only on a SQL warehouse you choose, and continuous triggers don’t support them (SQL conditions docs).

What “ready” means for a messy upstream

“The table changed” isn’t the same as “the data is ready”. The definitions that hold up:

  1. A load-control row from the source team. The upstream job writes batch_id, status = 'COMPLETE' and a row count when it finishes. This is the strongest signal because the source decides, not you.
  2. A watermark past your last run. max(load_ts) is later than the watermark your job last processed. Good for append-only feeds.
  3. A volume check against history. Today’s row count is at least, say, 80% of the trailing seven-day average for the same weekday. This catches partial loads that a fixed threshold like count(*) > 100 never will.
  4. Freshness on every input. For jobs with several upstreams, all of them must pass, not just the largest.

A volume check as a SQL condition:

WITH today AS (
  SELECT count(*) AS n FROM sales.raw_orders WHERE load_date = current_date()
), baseline AS (
  SELECT avg(n) AS avg_n FROM (
    SELECT load_date, count(*) AS n FROM sales.raw_orders
    WHERE load_date BETWEEN current_date() - 28 AND current_date() - 1
      AND dayofweek(load_date) = dayofweek(current_date())
    GROUP BY load_date)
)
SELECT today.n >= 0.8 * baseline.avg_n AS ready FROM today, baseline;

Cost traps

  • Frequent checks on a classic warehouse. A cron every five minutes with a SQL condition on a non-serverless warehouse means repeated warehouse starts. The check times out after 5 minutes including warehouse start-up, and a timed-out check starts no run. Use a serverless warehouse, or check less often.
  • Polling jobs on job clusters. The hand-built pattern often starts a cluster just to read one row. Moving that to a table trigger is usually the quickest saving.
  • Bursty writers. Without “wait after last change”, a source that writes ten small commits can start ten runs. Set the delay longer than the gap between the source’s commits.

Make skipped runs visible

The failure people notice at 7am is the run that never started. With triggers, a run that didn’t start because the condition wasn’t met looks the same as a broken pipeline unless you check for it. Two habits help:

  • an alert on “no successful run since X”, built from system.lakeflow.job_run_timeline, for every job with a freshness promise;
  • a short line in the job description saying what “ready” means, so whoever is on call knows what to check upstream.

Where this fits

Triggers decide when a job runs. How fresh a table needs to be in the first place is a separate decision. See are you paying for real time nobody uses?. For making the triggered job safe to run twice, see idempotent Databricks jobs.

Not sure where your own platform stands? Take the free 3-minute data platform scorecard, or see how we work on Databricks. Need people to do this alongside your team? See data engineering consulting. Unfamiliar terms are explained in the data engineering glossary.

Frequently asked questions

What is a table update trigger in Databricks?

A job trigger that starts a run when one or more Unity Catalog tables change, instead of on a schedule. You can wait for any or all of the listed tables, and set a 'wait after last change' delay so a burst of writes starts one run, not ten. Databricks says there is no extra cost beyond storage listing.

Do SQL condition triggers cost more than table update triggers?

Yes. Each condition check runs a query on a SQL warehouse, so you pay for warehouse time on every check. Use a table update trigger when 'the table changed' is enough, and add a SQL condition only when you need 'changed and good'.

How do you know an upstream load is complete, not partial?

Use a signal the source controls if one exists, such as a load-control row marked complete. Otherwise compare the new data with history: today's row count at least a set share of the trailing seven-day average catches partial loads that a fixed 'more than 100 rows' check misses.

Should I still build a sentinel table?

Only when the upstream is outside Unity Catalog or you need logic the built-in triggers can't express. A sentinel or audit table is still the cleanest contract with a source team that loads in several steps.

Have 30 minutes? Get a senior architect’s view of your data platform.

No sales deck. You talk to the engineer who would lead the work.

Book an architecture review
Hemant WalchaleYou talk to Hemant Walchale, founder145 published guidesProfiles: LinkedIn · Clutch · GoodFirms

Free, takes 20 seconds

Get your starter data platform roadmap

Pick your platform and your main goal. You get a 90-day plan written by senior data engineers: what to do first, next and after.

You’ll also get our monthly email (confirm it in your inbox). Unsubscribe any time. Privacy policy.