Databricks
A Databricks cost and reliability audit with system tables
A first-30-days audit for anyone taking over a Databricks platform: cost by job and owner from system.billing, failures and reruns from system.lakeflow, the usual waste patterns, and how to turn findings into a 90-day plan.

Hemant Walchale
· 3 min read
If you’ve just taken over a Databricks platform, the first month is for measuring, not changing. System tables answer most of the questions you need without anyone’s permission: what costs money, who owns it, what fails, and what nobody uses. This is the audit we run.
1. Cost by job and owner
Start with system.billing.usage, priced with system.billing.list_prices. Databricks’ jobs cost documentation has the reference queries. The key points:
- join prices on SKU and cloud, and only where the usage time falls in the price’s validity window,
- group by
usage_metadata.job_idfor jobs, and by warehouse and cluster IDs for everything else, - join
system.lakeflow.jobs(latest row per job) for names and creators, - group by
custom_tagsto see cost by team or project.
The result is cost at list price, which is right for ranking and finding waste, but won’t match your invoice exactly after discounts.
Output: the top 20 jobs, warehouses and clusters by cost over the last 30 days, each with an owner. Anything without an owner goes on a list.
2. Failures and reruns
system.lakeflow.job_run_timeline records each run with its result and duration; job_task_run_timeline does the same per task. Look for:
- jobs with the most failed runs, and their cost (a failing job pays for every attempt),
- runs whose duration is creeping up week by week,
- jobs that are retried manually every morning.
Output: the worst 10 jobs by failures times cost.
3. The usual waste patterns
| Pattern | How to spot it | Typical fix |
|---|---|---|
| Scheduled jobs on all-purpose clusters | Jobs usage on all-purpose SKUs | Move to job compute or serverless jobs |
| Long auto-termination | Clusters idle for hours | Shorter auto-termination, enforced by cluster policies |
| Oversized clusters | Low utilisation, high cost | Right-size, or switch to serverless |
| Always-on SQL warehouses | Warehouses running overnight with no queries | Auto-stop, serverless warehouses |
| Unused outputs | Tables written daily, not read in 90 days | Stop the pipeline, archive the table |
| No tags | Usage with empty custom_tags |
Cluster policies that require tags |
To find unused outputs, compare what pipelines write with what’s read, using lineage tables (system.access.table_lineage) where they’re available. Tables written daily and read by nobody in 90 days are often the easiest win to show.
4. Governance gaps (quick checks)
- Who has admin rights, and are any personal accounts running production jobs?
- Are production jobs owned by service principals?
- Do cluster policies exist, and are they enforced?
- Are costs tagged consistently by team or project?
5. Turn it into a 90-day plan
Pick one measurable target with a number attached, for example “cut job compute cost by 25%”, “halve failed runs” or “a freshness SLA for finance tables”. Then:
- agree it with a business owner who cares about that number,
- fix the top items from the lists above, starting with the cheapest changes (auto-termination, job compute, stopping unused pipelines),
- build a small dashboard on the same system tables queries, so progress is visible every week.
The short version
In the first month, measure: cost by job and owner from system.billing, failures from system.lakeflow, the usual waste patterns, unused outputs, and governance gaps. Then commit to one number for the next 90 days, and track it on a dashboard built from the same queries. Related: Databricks cluster policies, Databricks pricing explained and where data platform costs hide.
Sources
- Monitor job costs and performance with system tables, Databricks documentation
- Jobs system table reference, Databricks documentation
- Billable usage system table reference, Databricks documentation
- Lineage system tables reference, Databricks documentation
Not sure where your own platform stands? Take the free 3-minute data platform scorecard, or see how we work on Databricks. Need people to do this alongside your team? See data engineering consulting. Unfamiliar terms are explained in the data engineering glossary.