Databricks
Jobs that check the data before they run: Databricks table update triggers vs SQL conditions vs polling
When to start a Databricks job on a table update trigger, when to add a SQL condition, and when a hand-built sentinel table is still the answer, with what each costs and how to define 'ready' for a messy upstream.

Hemant Walchale
· 4 min read
Almost every Databricks team has hand-built the same thing: a job that wakes up on a cron, checks whether the upstream data is there, and exits if it isn’t. Databricks now has built-in versions of that pattern. A product manager asked on r/databricks what “ready” means for your worst upstream. This page answers that question and compares the three ways to wait for data.
The short answer
- Table update trigger when “the upstream table changed” is enough. No polling and no extra charge beyond storage listing.
- Table trigger plus a SQL condition when you need “changed and good”, for example a complete day or a row count in range.
- A sentinel or audit table when the source is outside Unity Catalog, loads in several steps, or another team needs to tell you explicitly that it’s done.
How the three options compare
| Table update trigger | SQL condition | Cron plus sentinel table | |
|---|---|---|---|
| Starts the run when | Listed tables change (any or all) | A query returns the expected result | Your check task finds the flag |
| Cost of waiting | Cloud cost of listing and reading table updates only | Billed as SQL warehouse usage per check | A job run (often a cluster start) per poll |
| Knows if the data is complete | No, only that it changed | Yes, if you write the check | Yes, if the source writes the flag |
| Works for non-UC sources | No | Only what the warehouse can query | Yes |
| Limits | Up to 25 tables per trigger (file events needed above 10) | One condition per trigger; 5-minute timeout including warehouse start | Whatever you build |
| Status | No preview label in the docs at the time of writing | Beta (admin enables it on the Previews page) | Your own code |
Databricks’ table update trigger documentation describes the any/all option, the “wait after last change” setting (more updates during the wait reset the timer), and the cost note. Supported sources include Unity Catalog managed and external Delta tables, materialized views, streaming tables and views. SQL conditions run read-only on a SQL warehouse you choose, and continuous triggers don’t support them (SQL conditions docs).
What “ready” means for a messy upstream
“The table changed” isn’t the same as “the data is ready”. The definitions that hold up:
- A load-control row from the source team. The upstream job writes
batch_id,status = 'COMPLETE'and a row count when it finishes. This is the strongest signal because the source decides, not you. - A watermark past your last run.
max(load_ts)is later than the watermark your job last processed. Good for append-only feeds. - A volume check against history. Today’s row count is at least, say, 80% of the trailing seven-day average for the same weekday. This catches partial loads that a fixed threshold like
count(*) > 100never will. - Freshness on every input. For jobs with several upstreams, all of them must pass, not just the largest.
A volume check as a SQL condition:
WITH today AS (
SELECT count(*) AS n FROM sales.raw_orders WHERE load_date = current_date()
), baseline AS (
SELECT avg(n) AS avg_n FROM (
SELECT load_date, count(*) AS n FROM sales.raw_orders
WHERE load_date BETWEEN current_date() - 28 AND current_date() - 1
AND dayofweek(load_date) = dayofweek(current_date())
GROUP BY load_date)
)
SELECT today.n >= 0.8 * baseline.avg_n AS ready FROM today, baseline;
Cost traps
- Frequent checks on a classic warehouse. A cron every five minutes with a SQL condition on a non-serverless warehouse means repeated warehouse starts. The check times out after 5 minutes including warehouse start-up, and a timed-out check starts no run. Use a serverless warehouse, or check less often.
- Polling jobs on job clusters. The hand-built pattern often starts a cluster just to read one row. Moving that to a table trigger is usually the quickest saving.
- Bursty writers. Without “wait after last change”, a source that writes ten small commits can start ten runs. Set the delay longer than the gap between the source’s commits.
Make skipped runs visible
The failure people notice at 7am is the run that never started. With triggers, a run that didn’t start because the condition wasn’t met looks the same as a broken pipeline unless you check for it. Two habits help:
- an alert on “no successful run since X”, built from
system.lakeflow.job_run_timeline, for every job with a freshness promise; - a short line in the job description saying what “ready” means, so whoever is on call knows what to check upstream.
Where this fits
Triggers decide when a job runs. How fresh a table needs to be in the first place is a separate decision. See are you paying for real time nobody uses?. For making the triggered job safe to run twice, see idempotent Databricks jobs.
Not sure where your own platform stands? Take the free 3-minute data platform scorecard, or see how we work on Databricks. Need people to do this alongside your team? See data engineering consulting. Unfamiliar terms are explained in the data engineering glossary.