Skip to content
METRIC OAK

New: the 3-week Data Platform Assessment: fixed fee, priced roadmap. See what's included →

Databricks

A Databricks cost and reliability audit with system tables

A first-30-days audit for anyone taking over a Databricks platform: cost by job and owner from system.billing, failures and reruns from system.lakeflow, the usual waste patterns, and how to turn findings into a 90-day plan.

Hemant Walchale

Hemant Walchale
· 3 min read

If you’ve just taken over a Databricks platform, the first month is for measuring, not changing. System tables answer most of the questions you need without anyone’s permission: what costs money, who owns it, what fails, and what nobody uses. This is the audit we run.

1. Cost by job and owner

Start with system.billing.usage, priced with system.billing.list_prices. Databricks’ jobs cost documentation has the reference queries. The key points:

  • join prices on SKU and cloud, and only where the usage time falls in the price’s validity window,
  • group by usage_metadata.job_id for jobs, and by warehouse and cluster IDs for everything else,
  • join system.lakeflow.jobs (latest row per job) for names and creators,
  • group by custom_tags to see cost by team or project.

The result is cost at list price, which is right for ranking and finding waste, but won’t match your invoice exactly after discounts.

Output: the top 20 jobs, warehouses and clusters by cost over the last 30 days, each with an owner. Anything without an owner goes on a list.

2. Failures and reruns

system.lakeflow.job_run_timeline records each run with its result and duration; job_task_run_timeline does the same per task. Look for:

  • jobs with the most failed runs, and their cost (a failing job pays for every attempt),
  • runs whose duration is creeping up week by week,
  • jobs that are retried manually every morning.

Output: the worst 10 jobs by failures times cost.

3. The usual waste patterns

Pattern How to spot it Typical fix
Scheduled jobs on all-purpose clusters Jobs usage on all-purpose SKUs Move to job compute or serverless jobs
Long auto-termination Clusters idle for hours Shorter auto-termination, enforced by cluster policies
Oversized clusters Low utilisation, high cost Right-size, or switch to serverless
Always-on SQL warehouses Warehouses running overnight with no queries Auto-stop, serverless warehouses
Unused outputs Tables written daily, not read in 90 days Stop the pipeline, archive the table
No tags Usage with empty custom_tags Cluster policies that require tags

To find unused outputs, compare what pipelines write with what’s read, using lineage tables (system.access.table_lineage) where they’re available. Tables written daily and read by nobody in 90 days are often the easiest win to show.

4. Governance gaps (quick checks)

  • Who has admin rights, and are any personal accounts running production jobs?
  • Are production jobs owned by service principals?
  • Do cluster policies exist, and are they enforced?
  • Are costs tagged consistently by team or project?

5. Turn it into a 90-day plan

Pick one measurable target with a number attached, for example “cut job compute cost by 25%”, “halve failed runs” or “a freshness SLA for finance tables”. Then:

  1. agree it with a business owner who cares about that number,
  2. fix the top items from the lists above, starting with the cheapest changes (auto-termination, job compute, stopping unused pipelines),
  3. build a small dashboard on the same system tables queries, so progress is visible every week.

The short version

In the first month, measure: cost by job and owner from system.billing, failures from system.lakeflow, the usual waste patterns, unused outputs, and governance gaps. Then commit to one number for the next 90 days, and track it on a dashboard built from the same queries. Related: Databricks cluster policies, Databricks pricing explained and where data platform costs hide.

Sources

Not sure where your own platform stands? Take the free 3-minute data platform scorecard, or see how we work on Databricks. Need people to do this alongside your team? See data engineering consulting. Unfamiliar terms are explained in the data engineering glossary.

Frequently asked questions

How do I see Databricks cost by job?

Join system.billing.usage (filtered to jobs usage) to system.billing.list_prices on SKU, cloud and the price's validity window, then group by job ID. Join system.lakeflow.jobs to get job names. Databricks publishes example queries for this in its jobs cost documentation. The result is a list-price estimate, not your invoice.

Which system tables show job failures and reruns?

system.lakeflow.job_run_timeline has one row per job run with its result state and timings; system.lakeflow.job_task_run_timeline has the same at task level. Join them to cost to find jobs that fail often and cost the most.

What are the most common sources of Databricks waste?

Scheduled jobs running on all-purpose clusters instead of job compute, clusters with long auto-termination times, oversized or always-on clusters, jobs that fail and rerun repeatedly, and pipelines refreshing tables nobody reads. Untagged resources make all of these hard to attribute.

Can Databricks cost be attributed to individual tables?

Not natively. Billing is recorded per compute usage, and one job can write many tables. You can attribute cost by job, task, cluster, warehouse or tag, and approximate per-table cost with heuristics, but tags and one-table-per-task design make attribution much easier.

Have 30 minutes? Get a senior architect’s view of your data platform.

No sales deck. You talk to the engineer who would lead the work.

Book an architecture review
Hemant WalchaleYou talk to Hemant Walchale, founder145 published guidesProfiles: LinkedIn · Clutch · GoodFirms

Free, takes 20 seconds

Get your starter data platform roadmap

Pick your platform and your main goal. You get a 90-day plan written by senior data engineers: what to do first, next and after.

You’ll also get our monthly email (confirm it in your inbox). Unsubscribe any time. Privacy policy.