What is data lineage? Why it matters and how to capture it

Data lineage records where data comes from and every step that transforms it on its way to reports, dashboards and models. It answers two questions every data team is asked: where did this number come from? and what breaks if we change this?

Upstream and downstream

• Upstream lineage traces a figure back to its sources: which tables, columns and transformations produced this revenue number?
• Downstream lineage shows the impact of a change: if the billing system renames a column, which tables, dashboards and models are affected?

Table-level and column-level

• Table-level lineage shows which tables feed which. Useful for orientation and impact at a glance.
• Column-level lineage shows which source columns feed each column, through which transformations. Needed to trace a single metric, or to find every place a sensitive field such as a national ID is used.

What lineage is used for

1. Impact analysis: check what a schema change, migration or deprecation will break before making it.
2. Root-cause analysis: when a dashboard is wrong, follow the lineage upstream to the failing step.
3. Regulation and audit: show how reported figures are produced, for example under BCBS 239 for banks, and where personal data flows under privacy law.
4. Migrations: find every consumer of a legacy table before switching it off.
5. Trust and discovery: let analysts see whether a table comes from trusted, tested sources.

How it is captured

Hand-drawn lineage goes stale within weeks. Modern lineage is captured automatically:

• From the platform: Unity Catalog on Databricks records table and column lineage for queries and jobs it runs; Snowflake and Microsoft Purview provide lineage for their ecosystems.
• From transformation code: dbt builds a lineage graph from model dependencies.
• From orchestration: the OpenLineage open standard lets orchestrators and engines (such as Airflow and Spark) send lineage events to a shared store.

Gaps usually appear at the edges, in spreadsheets, scripts on servers and SaaS tools, so combine automatic capture with ownership and documentation for those.

The short version

Data lineage maps where data comes from and what it feeds, at table and column level. Capture it automatically from your platform, transformation tools and orchestrator, and use it for impact analysis, root-cause analysis, regulation and migrations.

Related: what is Unity Catalog, data contracts for data leaders, medallion architecture and our data quality and governance service.

Sources

• Capture and view data lineage using Unity Catalog, Databricks documentation
• OpenLineage, an open standard for lineage metadata
• Principles for effective risk data aggregation and risk reporting (BCBS 239), Bank for International Settlements