Glossary
Data engineering, in plain English
Short definitions of the terms that come up when you modernize a data platform, with links to where we go deeper.
A
- ACID transactions
- Guarantees that a set of changes to data is applied completely or not at all (atomicity), leaves the data valid (consistency), does not interfere with other changes running at the same time (isolation), and survives failures once committed (durability). Table formats such as Delta Lake and Apache Iceberg bring ACID transactions to files in cloud storage. Concurrency control guide
- Apache Iceberg
- An open table format for large analytic datasets, supported by Snowflake, Databricks, AWS, Google Cloud and many query engines. It lets several engines read and write the same tables safely.
B
- Batch processing
- Processing data in scheduled groups, for example every hour or every night, rather than as each record arrives. It is simpler and cheaper than streaming and is still the right choice for most reporting.
C
- Change data capture (CDC)
- Reading inserts, updates and deletes from a source database's transaction log and copying only those changes downstream. It keeps a warehouse or lakehouse in sync without reloading whole tables. How change data capture works
D
- Data catalog
- A searchable inventory of an organization's datasets with their owners, descriptions, classifications and lineage. Unity Catalog on Databricks and Horizon on Snowflake are built-in examples. Data quality & governance
- Data contract
- An agreement between the team that produces data and the teams that use it, covering schema, meaning, quality checks and change rules. Breaking changes are caught before they reach dashboards or models. Data contracts for data leaders
- Data lake
- Low-cost cloud storage holding raw files in open formats such as Parquet or JSON. Flexible and cheap, but without a table format and governance on top it tends to become hard to trust.
- Data lakehouse
- An architecture that adds warehouse features (ACID tables, schemas, access control, fast SQL) to data stored in open formats in a data lake, so BI, data science and AI share one copy of the data. What is a data lakehouse?
- Data lineage
- A record of where each dataset and column came from and what transformed it on the way. It answers 'where did this number come from?' and shows what breaks if a source changes. What is data lineage?
- Data mesh
- An operating model in which business domains own and publish their data as products, on a shared self-service platform with common governance standards. It is an organizational change as much as a technical one.
- Data observability
- Monitoring of data freshness, volume, schema and distribution so that pipeline problems are detected, and alerted to an owner, before business users notice them. Managed data platform
- Data product
- A dataset built and maintained for a defined set of users, with an owner, documentation, quality guarantees and a stable interface, rather than a by-product of one pipeline.
- Data quality testing
- Automated checks that data meets expectations: not null, unique, within valid ranges, matching a reference, arriving on time. Tests run in the pipeline and stop bad data from being published. Data quality & governance
- Data warehouse
- A database designed for analytics: structured tables, SQL, and fast queries over large volumes. Snowflake, Amazon Redshift, Google BigQuery and older on-premises systems such as Teradata are data warehouses.
- dbt
- An open-source tool for writing data transformations as SQL select statements, with version control, testing, documentation and dependency management. Widely used on Snowflake, Databricks and BigQuery.
- Delta Lake
- An open table format that adds ACID transactions, schema enforcement, time travel and efficient updates to Parquet files. It is the default table format on Databricks.
E
- ELT and ETL
- Extract, transform, load (ETL) reshapes data before it reaches the target system. Extract, load, transform (ELT) loads raw data first and transforms it inside the warehouse or lakehouse, which is the common pattern on modern cloud platforms.
F
- FinOps for data
- The practice of making data platform spend visible, allocated and actively managed: tagging workloads, setting budgets, right-sizing compute and reviewing costs with the teams that create them. Where data platform costs hide
I
- Index (database)
- A data structure that lets a database find rows without scanning a whole table, at the cost of extra storage and slower writes. Database indexes guide
M
- Medallion architecture
- A common way to layer a lakehouse: bronze holds raw ingested data, silver holds cleaned and conformed data, and gold holds business-ready aggregates and data products. Medallion architecture explained
- Microsoft Fabric
- Microsoft's software-as-a-service analytics platform that combines data engineering, warehousing, real-time analytics and Power BI on a shared storage layer called OneLake.
O
- Orchestration
- Scheduling and coordinating pipeline steps so that they run in the right order, retry on failure and alert when something breaks. Examples include Airflow, Databricks Workflows and Azure Data Factory.
P
- Parallel run
- Running a legacy system and its replacement side by side on the same inputs and comparing outputs before switching over. It is the safest way to prove a migration is correct. Teradata to Databricks migration plan
R
- Retrieval-augmented generation (RAG)
- A pattern in which a large language model answers questions using documents or records retrieved at query time, so answers are grounded in your own data. It only works well when that data is clean, current and permissioned. RAG and enterprise data permissions
- Reverse ETL
- Copying modelled data from the warehouse back into operational tools such as a CRM, marketing platform or support desk, so teams act on the same data used for reporting.
- Role-based access control (RBAC)
- Granting permissions to roles rather than individuals, so access follows a person's job. Combined with row- and column-level security it keeps sensitive data visible only to those who need it.
S
- Schema evolution
- Changing a table's structure, such as adding or renaming columns, without breaking the pipelines and reports that read it.
- Semantic layer
- A shared definition of business metrics and dimensions (revenue, active customer, churn) that every BI tool and AI assistant uses, so the same question gets the same answer everywhere.
- Slowly changing dimension (SCD)
- A technique for keeping the history of attributes that change over time, such as a customer's address. Type 2 keeps every version with valid-from and valid-to dates.
- Snowpark
- Snowflake's framework for running Python, Java and Scala code inside Snowflake, so data engineering and machine learning run next to the data instead of on separate infrastructure. Snowflake
- Streaming
- Processing records continuously as they arrive, with latency of seconds or less. It suits fraud detection, real-time personalization and operational monitoring, but costs more to build and run than batch.
U
- Unity Catalog
- Databricks' governance layer for data and AI assets: one place to manage permissions, lineage, auditing and discovery across workspaces. What is Unity Catalog?
V
- Vector database
- A store optimized for similarity search over embeddings, the numeric representations of text, images or records that AI models use. It powers semantic search and RAG. AI-ready data
Have 30 minutes? Get a senior architect’s view of your data platform.
No sales deck. You talk to the engineer who would lead the work.
Book an architecture review