Databricks
Databricks well-architected assessment: the seven pillars and what to check
How to run a well-architected review of a Databricks lakehouse against the seven pillars of Databricks' framework: governance, interoperability, operational excellence, security, reliability, performance and cost, with the evidence to collect from system tables.
Hemant Walchale
· 3 min read
A Databricks estate that grew quickly usually works, but it carries hidden risk: workspaces outside Unity Catalog, all-purpose clusters running jobs, no disaster recovery plan, and costs nobody can attribute. A well-architected assessment reviews the platform against Databricks’ own framework and turns the findings into a prioritised plan.
The seven pillars
Databricks’ well-architected lakehouse framework groups its best practices into seven pillars. A useful assessment scores each one from evidence.
| Pillar | What to check | Evidence |
|---|---|---|
| Data and AI governance | Unity Catalog adoption, ownership, lineage, access on groups, data quality | Catalog and grant inventory, legacy Hive metastore tables, lineage coverage |
| Interoperability and usability | Open formats, standard interfaces, self-service for analysts | Delta or Iceberg usage, BI connections, SQL warehouse use |
| Operational excellence | Code in Git, CI/CD, environments, monitoring, infrastructure as code | Repos and bundles, job definitions, alerting |
| Security, privacy and compliance | Identity, network isolation, secrets, encryption, audit | Workspace settings, audit logs, Security Analysis Tool results |
| Reliability | Retries, recovery, backups, disaster recovery | Job failure history, table history retention, DR runbooks |
| Performance efficiency | Right compute for each workload, Photon, clustering, small files | Query history, cluster sizes, table layout |
| Cost optimization | Job compute, auto-termination, policies, tagging, budgets | system.billing.usage, cluster policies, idle time |
How to run it
- Collect evidence. Query system tables for billing, compute, jobs and audit events. Export workspace, Unity Catalog and policy configuration. Run Databricks’ Security Analysis Tool for a security baseline.
- Interview owners. Short sessions with the platform team, two or three domain teams and security. Ask what breaks, what is slow and what nobody owns.
- Score each pillar. Rate current practice against the framework’s best practices, with the evidence next to each score.
- Prioritise. Rank findings by risk and value. Cost and security fixes are often quick; governance and reliability need a plan.
- Write the roadmap. Quick wins for the first month, then larger changes such as moving to Unity Catalog, adding CI/CD or setting up disaster recovery.
Findings we see most often
- Tables still in the legacy Hive metastore. See upgrading Hive metastore to Unity Catalog.
- Production jobs on all-purpose clusters, and no cluster policies. See Databricks cluster policies.
- Untagged compute, so costs cannot be allocated. See the data platform FinOps guide.
- Permissions granted to individuals rather than groups.
- No tested recovery plan for critical tables and jobs.
The short version
Score the platform against the seven pillars using evidence from system tables, configuration and the Security Analysis Tool, then turn findings into a ranked plan. Our data platform assessment checklist covers the wider questions, and our managed data platform service runs these reviews and fixes what they find.
Sources
- Well-architected lakehouse framework, Databricks documentation
- Data lakehouse architecture, Databricks documentation
- System tables reference, Databricks documentation
- Security Analysis Tool, Databricks Industry Solutions on GitHub
Not sure where your own platform stands? Take the free 3-minute data platform scorecard, or see how we work on Databricks. Unfamiliar terms are explained in the data engineering glossary.