Insights
Practical notes on building data platforms that AI can trust
Migration plans, platform choices, governance and cost, written for data leaders and the engineers who report to them.
Comparison
Partitioning vs bucketing vs clustering: which data layout should you use?
Partitioning, bucketing and clustering compared: what each does, when it speeds up queries, the traps, and what Databricks, Snowflake, BigQuery and Spark support today, including Databricks' guidance to skip partitioning under 1 TB.
AI
AI agents on governed data: keeping permissions intact when agents query your platform
How to let AI agents query enterprise data without leaking it: why a shared service account is the core risk, how agents can act as the user on Databricks and Snowflake, tool access, audit, and a rollout checklist.
Regulation
BCBS 239 and data lineage: what supervisors expect from bank data platforms
BCBS 239 sets 14 principles for risk data aggregation and risk reporting. Why data lineage sits at the centre of compliance, what the ECB's 2024 guide expects, and how to build lineage on a modern bank data platform.
Comparison
Dask vs Ray: scaling Python for data and machine learning
Dask and Ray compared for distributed Python: pandas and NumPy-style collections versus tasks, actors and AI libraries, data processing, training and inference, GPUs, deployment, ecosystem and when Spark is the better answer.
Guide
FinOps for data platforms: a practical guide for Databricks, Snowflake and Microsoft Fabric
How to apply FinOps to data platforms: getting cost data out of Databricks, Snowflake and Fabric, tagging and allocation, showback versus chargeback, unit costs, the optimizations that matter, guardrails, KPIs and a 90-day plan.
How-to
How to set up Databricks compute policies to control cost
A step-by-step guide to Databricks compute (cluster) policies: the seven rules that matter most, policy JSON examples for auto-termination, DBU caps, node types and required tags, one policy per environment, and how to roll them out.
Databricks
Databricks well-architected assessment: the seven pillars and what to check
How to run a well-architected review of a Databricks lakehouse against the seven pillars of Databricks' framework: governance, interoperability, operational excellence, security, reliability, performance and cost, with the evidence to collect from system tables.
Comparison
Delta Lake vs Apache Iceberg: how they differ, and how to choose
Delta Lake and Apache Iceberg both add ACID transactions to Parquet files in cloud storage. How their metadata differs, which platforms favour each, how UniForm, OneLake and Snowflake now bridge them, and a practical way to choose.
Regulation
DORA: what it means for bank data platforms
What the EU Digital Operational Resilience Act asks of bank and financial data platforms: the five pillars, the register of information, incident evidence, restore testing, cloud and vendor exit plans, and a practical checklist for data teams.
Regulation
HIPAA on Databricks and Snowflake: how to run protected health data on a cloud platform
What HIPAA's Security Rule asks of a healthcare data platform, what Databricks and Snowflake require before you load PHI (BAA, compliance security profile, Business Critical edition), de-identification for analytics and AI, and a checklist.
AI
How to assess AI readiness: seven data checks before an AI use case goes live
A practical way to assess whether your data is ready for a specific AI use case: seven checks covering the use case, data, access, quality, freshness, meaning and lineage, how to score them, and what to fix first.
How-to
How to size a Microsoft Fabric capacity: a step-by-step guide
How to choose the right Microsoft Fabric F SKU: what capacity units mean, measuring on a trial with the Capacity Metrics app, the SKU Estimator, smoothing and the four throttling stages, the F64 licensing threshold, and when to scale, split or pause.
Comparison
MLflow vs Kubeflow: experiment tracking or a Kubernetes ML platform?
MLflow and Kubeflow compared: what each actually does, tracking and model registry versus Kubernetes pipelines, training and serving, GenAI tracing in MLflow 3, managed options, operating effort and how teams combine them.
Regulation
PCI DSS on data platforms: keeping card data out of your lakehouse
How PCI DSS v4.0.1 applies to data warehouses and lakehouses, why tokenizing card numbers before ingestion keeps the platform out of scope, and what Requirement 3 demands if you must store them, including keyed hashes under 3.5.1.1.
Comparison
Prefect vs Airflow: which orchestrator should a data team use?
Prefect 3 and Apache Airflow 3 compared for data pipelines: how you write workflows, scheduling and event triggers, dynamic work, deployment and managed options, observability, ecosystem and when each one fits.
Comparison
SageMaker vs Databricks: which platform for data and machine learning on AWS?
Amazon SageMaker and Databricks compared for machine learning and data on AWS: SageMaker AI versus the next-generation SageMaker with Unified Studio, the lakehouse and table formats, MLOps, governance, multi-cloud, cost and when each fits.
AI
Semantic layers for AI: why agents need one definition of your metrics
What a semantic layer is, why AI assistants and agents give wrong answers without one, what it should contain, and the options on Databricks, Snowflake, dbt and Power BI, with an approach to building it.
How-to
How to set up Snowflake resource monitors: a step-by-step guide
How to set up Snowflake resource monitors to cap warehouse credit spend: account versus warehouse monitors, choosing quotas from metering history, NOTIFY and SUSPEND thresholds, SQL examples, finding unmonitored warehouses, and what monitors do not cover.
Migration
Snowflake to Microsoft Fabric migration: mirroring, T-SQL and what changes
How to move from Snowflake to Microsoft Fabric, or run both: mirroring and Iceberg interoperability, converting Snowflake SQL to Fabric Warehouse T-SQL, VARIANT, streams and tasks, Snowpark, security, and a domain-by-domain cutover.
Migration
SQL Server to Databricks migration: T-SQL conversion, collation and CDC
How to migrate a SQL Server data warehouse to Databricks: Lakebridge assessment and conversion, Lakeflow Connect CDC ingestion, case-insensitive collation versus Databricks defaults, T-SQL differences, SQL Agent jobs, SSIS and reconciliation.
Comparison
Strategy Mosaic vs Databricks metric views: universal semantic layer or native metrics?
Strategy Mosaic and Databricks Unity Catalog metric views compared: scope across platforms, how metrics are defined, governance, which BI and AI tools can use them, pricing model and when each one fits.
Migration
Azure Synapse to Snowflake migration: dedicated pools, T-SQL and the collation trap
How to migrate Azure Synapse Analytics to Snowflake on Azure: dedicated SQL pool tables via ADLS and COPY INTO, SnowConvert for T-SQL, distributions and indexes, case-insensitive collation, workload management, pipelines, serverless views and Power BI.
How-to
How to upgrade from Hive metastore to Unity Catalog: a step-by-step guide
A practical, six-step guide to moving Databricks workspaces from the legacy Hive metastore to Unity Catalog: UCX assessment, metastore and storage setup, groups, SYNC versus DEEP CLONE for tables, code changes and cut-over.
Explainer
What is data mesh? The four principles, and when it fits
Data mesh moves ownership of analytical data to business domains, which publish data products on a shared self-serve platform under federated governance. The four principles, how it differs from a central data team, prerequisites and common failure modes.
Explainer
What is data observability? The five pillars, and how it differs from testing
Data observability is continuous monitoring of data health across pipelines: freshness, volume, distribution, schema and lineage. How it differs from data quality tests, what to monitor first, and the tool options on Databricks, Snowflake and dbt.
Explainer
What is reverse ETL? How it works, use cases and pitfalls
Reverse ETL syncs modelled data from your warehouse or lakehouse back into operational tools such as CRM, marketing, support and finance systems. How it works, common use cases, tools, and the governance and reliability issues to plan for.
Migration
Alteryx to Databricks migration: moving analyst workflows to the lakehouse
How to migrate Alteryx Designer workflows and Alteryx Server schedules to Databricks: inventory tools from workflow XML, map Alteryx tools to Spark SQL, replace macros and predictive tools, and bring analysts along.
Migration
Google BigQuery to Databricks migration: a practical guide
How to migrate BigQuery to Databricks: exporting tables to Delta, converting GoogleSQL to Spark SQL (UNNEST, STRUCT, SAFE functions), partitioned and clustered tables, scheduled queries, BigQuery ML and Lakehouse Federation.
Migration
Google BigQuery to Snowflake migration: a practical guide
How to migrate BigQuery to Snowflake: exporting to Cloud Storage and loading with COPY INTO, converting GoogleSQL (UNNEST, STRUCT, SAFE functions), partitioning and clustering, scheduled queries, authorized views and BigQuery ML.
Cost
Databricks pricing explained: DBUs, compute types and what you really pay
How Databricks pricing works: what a DBU is, list prices by compute type, classic versus serverless, cloud VM and storage costs, worked examples, discounts and how to see your own spend.
Comparison
Databricks vs BigQuery: lakehouse or serverless warehouse?
Databricks and Google BigQuery compared: architecture, who each suits, data engineering and machine learning, pricing models, open formats, Google Cloud integration and when to use both.
Comparison
Databricks vs Microsoft Fabric: how to choose in 2026
A practical comparison of Databricks and Microsoft Fabric: architecture, who each suits, pricing models, governance, AI, Power BI integration and when to use both.
Comparison
Databricks vs Snowflake for healthcare and life sciences
How healthcare providers, payers and life sciences companies should choose between Databricks and Snowflake: HIPAA and PHI, clinical and claims data, genomics and imaging, AI, data sharing and cost.
Comparison
Databricks vs Snowflake for retail and ecommerce
How retailers and ecommerce companies should choose between Databricks and Snowflake: demand forecasting, personalization, clickstream, POS and loyalty data, supplier data sharing and retail media, cost and team skills.
Migration
IBM DataStage to Databricks migration: converting jobs, stages and sequences
How to move IBM InfoSphere DataStage parallel jobs and sequences to Databricks: export and inventory, map each stage to Spark SQL and Delta, rebuild sequences as Lakeflow Jobs and reconcile.
Migration
IBM DataStage to Microsoft Fabric migration: jobs, stages and sequences
How to migrate IBM DataStage parallel jobs and sequences to Microsoft Fabric: export and inventory, map stages to Spark notebooks, Warehouse SQL or Dataflow Gen2, and rebuild sequences as Data Factory pipelines.
Comparison
dbt vs Lakeflow Declarative Pipelines on Databricks: which should you use?
dbt and Databricks Lakeflow Declarative Pipelines (formerly Delta Live Tables) compared: batch modelling vs streaming and CDC, testing and expectations, portability, orchestration, cost and how teams use both.
Migration
Oracle Exadata to Databricks migration: replacing the hardware refresh
How to move a data warehouse off Oracle Exadata to Databricks before the next hardware refresh: what replaces Smart Scan, HCC and partitioning, moving tens of terabytes, and keeping OLTP workloads separate.
Migration
Greenplum to Databricks migration: a practical guide
How to migrate a Greenplum MPP warehouse to Databricks: converting PostgreSQL-based SQL and PL/pgSQL, replacing DISTRIBUTED BY and append-optimized tables, gpfdist loads, MADlib analytics, and reconciliation.
Migration
Greenplum to Snowflake migration: a practical guide
How to migrate a Greenplum warehouse to Snowflake: PostgreSQL-dialect SQL conversion, DISTINCT ON and arrays, PL/pgSQL to Snowflake Scripting, replacing DISTRIBUTED BY and gpfdist, MADlib to Snowpark, and reconciliation.
Migration
Hadoop to Databricks migration: moving HDFS, Hive, Spark and Oozie
How to migrate a Hadoop (Cloudera or Hortonworks) estate to Databricks: HDFS to cloud storage, Hive tables to Delta, Hive metastore and Ranger to Unity Catalog, Spark and Oozie jobs to Lakeflow Jobs.
Migration
Hadoop to Microsoft Fabric migration: HDFS, Hive and Spark to OneLake
How to migrate a Hadoop (Cloudera or Hortonworks) cluster to Microsoft Fabric: HDFS to OneLake, Hive tables to Lakehouse Delta tables, Spark jobs to Fabric Spark, Oozie to Data Factory, Kafka to Real-Time Intelligence.
Migration
Hadoop to Snowflake migration: moving Hive, Spark and HDFS data
How to migrate a Hadoop (Cloudera or Hortonworks) cluster to Snowflake: HDFS data to Snowflake or Iceberg tables, Hive SQL conversion, Spark jobs to SQL or Snowpark, Oozie to Tasks, Ranger to Snowflake access policies.
Cost
Hidden costs on Databricks, Snowflake and Microsoft Fabric (and how to fix each)
Fifteen costs that grow data platform bills without anyone noticing: five each on Databricks, Snowflake and Microsoft Fabric, why they happen and the fix for each.
Migration
Informatica PowerCenter to Databricks migration: a practical guide
How to move Informatica PowerCenter mappings, sessions and workflows to Databricks: export and inventory the repository, map each transformation to Spark SQL and Delta, rebuild workflows as Lakeflow Jobs, and reconcile every target before switching off the licence.
Migration
Informatica PowerCenter to Microsoft Fabric migration: a practical guide
How to move Informatica PowerCenter mappings and workflows to Microsoft Fabric: choosing Dataflow Gen2, Warehouse SQL or Spark notebooks per mapping, transformation mapping, pipelines for workflows, and reconciliation.
Migration
Informatica PowerCenter to Snowflake migration: options and a practical plan
How to move Informatica PowerCenter ETL to Snowflake: keep Informatica and push down, or rebuild in dbt, Snowflake SQL and Snowpark. Transformation mapping, orchestration, CDC and reconciliation.
Cost
Microsoft Fabric pricing explained: capacity SKUs, the F64 threshold and OneLake storage
How Microsoft Fabric pricing works: F SKUs and capacity units, pay-as-you-go versus reservations, pausing, the F64 Power BI licensing threshold, OneLake storage, smoothing and throttling, with a worked example.
Comparison
Microsoft Fabric vs Azure Synapse: should you move, and when?
Microsoft Fabric and Azure Synapse Analytics compared: what changed, SaaS versus PaaS, capacity pricing, OneLake, Warehouse versus dedicated SQL pools, Power BI, and how to decide when to migrate.
Migration
Netezza to Databricks migration: from appliance to lakehouse
How to migrate IBM Netezza to Databricks: converting NZSQL and NZPLSQL to Spark SQL, replacing distribution and organizing keys with liquid clustering, loading with Auto Loader, governance in Unity Catalog and cutover.
Migration
Netezza to Snowflake migration: a practical guide
How to migrate IBM Netezza to Snowflake: inventory, converting NZSQL and NZPLSQL, replacing distribution keys and zone maps, loading with COPY INTO, parallel runs and switching off the appliance.
Migration
Oracle data warehouse to Databricks migration: a practical guide
Moving an Oracle data warehouse to Databricks: PL/SQL conversion, the Oracle behaviours that break results (empty strings, DATE, NUMBER), change data capture, parallel runs and licence decommissioning.
Migration
Oracle to Microsoft Fabric migration: PL/SQL, data types and Power BI
How to migrate an Oracle data warehouse to Microsoft Fabric: Warehouse or Lakehouse, converting PL/SQL to T-SQL or Spark, empty strings and DATE semantics, change data capture, Power BI and cutover.
Migration
Oracle to Snowflake migration: converting PL/SQL and avoiding the classic gotchas
How to move an Oracle data warehouse to Snowflake: PL/SQL to Snowflake Scripting, empty strings and DATE semantics, sequences, CONNECT BY, materialized views and triggers, change data capture and reconciliation.
Migration
How to convert Pentaho (PDI) jobs to Databricks: a step-by-step migration guide
Moving Pentaho Data Integration (Kettle) transformations and jobs to Databricks: how to inventory .ktr and .kjb files, map each PDI step to Spark SQL, Delta and Lakeflow, and prove the results match.
Migration
Amazon Redshift to Databricks migration: a practical guide
How to move Amazon Redshift to Databricks on AWS: unloading to S3 as Delta, Spectrum tables into Unity Catalog, converting SQL and procedures, Lakehouse Federation for a gradual cutover, and replacing WLM with SQL warehouses.
Migration
Amazon Redshift to Snowflake migration: a practical guide
How to migrate Amazon Redshift to Snowflake: SQL and stored procedure conversion, identifier case and VARCHAR length gotchas, DISTKEY and SORTKEY, WLM to warehouses, Spectrum, loading via S3 and reconciliation.
Migration
SAS to Databricks migration: converting DATA steps, PROCs and macros
How to migrate SAS analytics and ETL to Databricks: inventory programs, convert DATA steps and PROC SQL to Spark SQL and PySpark, replace PROCs, macros and formats, revalidate models and retire SAS licences.
Migration
SAS to Snowflake migration: moving SAS analytics and ETL into Snowflake
How to migrate SAS workloads to Snowflake: pushing processing down with SAS/ACCESS as an interim step, converting PROC SQL and DATA steps to SQL and Snowpark, replacing macros, formats and PROCs, and validating models.
Cost
Snowflake pricing explained: credits, warehouses, storage and the rules that drive the bill
How Snowflake pricing works: credits and editions, warehouse sizes and per-second billing, auto-suspend, serverless features, the cloud services 10% rule, storage, Time Travel and data transfer, with worked examples.
Migration
Snowflake to Databricks migration: when and how to consolidate
How to move workloads from Snowflake to Databricks: deciding what to move, federating Snowflake into Unity Catalog, unloading to Delta, converting Snowflake SQL, Streams and Tasks, Snowpark and governance policies.
Comparison
Snowflake vs BigQuery: how to choose a cloud data warehouse
Snowflake and Google BigQuery compared for data leaders: architecture, clouds, pricing models (warehouses vs bytes scanned and slots), performance and concurrency, data sharing, AI features and when each fits.
Comparison
Snowflake vs Microsoft Fabric: which data platform fits your team?
Snowflake and Microsoft Fabric compared for data leaders: architecture, SQL experience, Power BI, pricing models, data sharing, governance and when to combine them.
Migration
SQL Server and SSIS to Microsoft Fabric migration: a practical guide
How to move a SQL Server data warehouse and its SSIS packages to Microsoft Fabric: Warehouse or Lakehouse, T-SQL compatibility, rebuilding SSIS in Fabric Data Factory, Power BI on OneLake and cutover.
Migration
SQL Server to Snowflake migration: T-SQL conversion and the collation trap
How to migrate a SQL Server data warehouse to Snowflake: converting T-SQL procedures, case-insensitive collation versus case-sensitive Snowflake, data types, temp tables, SQL Agent jobs, SSIS and reconciliation.
Migration
SSIS to Databricks migration: how to convert SQL Server Integration Services packages
Moving SSIS packages to Databricks: inventory .dtsx files, map data flow components to Spark SQL and Delta, rebuild control flow and SQL Agent schedules as Lakeflow Jobs, and reconcile before cutover.
Migration
SSIS to Snowflake migration: replacing SSIS packages with ELT in Snowflake
How to move SQL Server Integration Services packages to Snowflake: land raw data, rebuild data flows as SQL and dbt, replace OLE DB Command with MERGE, Script Tasks with Snowpark, and SQL Agent with Snowflake Tasks.
Migration
Azure Synapse to Databricks migration: dedicated pools, Spark and pipelines
How to migrate Azure Synapse Analytics to Azure Databricks: dedicated SQL pools to Delta and Databricks SQL, T-SQL conversion, Synapse Spark notebooks, serverless views over ADLS, pipelines and Power BI.
Migration
Azure Synapse Analytics to Microsoft Fabric migration: a practical guide
How to move Azure Synapse dedicated SQL pools, pipelines, Spark pools and serverless SQL to Microsoft Fabric: Warehouse and Lakehouse targets, T-SQL differences, data movement and cutover.
Migration
Talend to Databricks migration: converting Talend jobs to Spark SQL and Lakeflow
How to migrate Talend Studio jobs, including Talend Big Data Spark jobs, to Databricks: inventory components, map tMap and other components to Spark SQL, rewrite tJava code, and replace contexts and scheduling with Lakeflow Jobs.
Migration
Talend to Microsoft Fabric migration: replacing Talend jobs with Fabric pipelines
How to move Talend Studio jobs to Microsoft Fabric: inventory components from job XML, choose Dataflow Gen2, Warehouse SQL or Spark notebooks, map tMap and other components, and replace contexts and schedules with Data Factory pipelines.
Migration
Talend to Snowflake migration: replacing Talend jobs with dbt and Snowflake SQL
How to move Talend Studio jobs into Snowflake: inventory components, map tMap and other components to SQL, replace contexts and scheduling, handle tJava code, and reconcile before switching off Talend.
Migration
Teradata to Microsoft Fabric migration: a practical guide
How to move a Teradata warehouse to Microsoft Fabric: Warehouse or Lakehouse, converting Teradata SQL to T-SQL, QUALIFY and case-sensitivity differences, BTEQ and FastLoad replacements, Power BI and cutover.
Migration
Teradata to Snowflake migration: a practical guide
How to move a Teradata warehouse to Snowflake: inventory, SQL and BTEQ conversion, the behaviour differences that break reports, loading, parallel runs and decommissioning before the renewal.
Migration
Vertica to Databricks migration: a practical guide
How to migrate a Vertica analytics warehouse to Databricks: exporting to Parquet, replacing projections and segmentation, flex tables, Vertica-specific SQL such as TIMESERIES, UDx and in-database ML, and reconciliation.
Migration
Vertica to Snowflake migration: a practical guide
How to migrate Vertica to Snowflake: exporting to Parquet and loading with COPY INTO, replacing projections, flex tables to VARIANT, rewriting TIMESERIES with time spines and ASOF JOIN, UDx and in-database ML.
Explainer
What is a data lakehouse? A plain-English explanation
A data lakehouse combines the low-cost, open storage of a data lake with the transactions, governance and fast SQL of a data warehouse. How it works, how it differs from a warehouse and a lake, and when it fits.
Explainer
What is change data capture (CDC)? How it works and when to use it
Change data capture copies only the inserts, updates and deletes from a source database instead of reloading whole tables. Log-based, query-based and trigger-based CDC, common tools, applying changes with MERGE, and pitfalls.
Explainer
What is data lineage? Why it matters and how to capture it
Data lineage records where data comes from and everything that transforms it on the way to reports and models. Table and column lineage, how it is captured, tools, and its uses: impact analysis, root-cause analysis and regulation.
Explainer
What is medallion architecture? Bronze, silver and gold explained
Medallion architecture organizes a lakehouse into bronze (raw), silver (cleaned and conformed) and gold (business-ready) layers. What each layer holds, rules that keep it working, common mistakes and how to build it.
Explainer
What is Unity Catalog? Databricks governance explained
Unity Catalog is Databricks' governance layer for data and AI: one place for permissions, row filters and column masks, lineage, auditing and discovery across workspaces. How it is organized, what it governs and how to adopt it.
Data quality
Data contracts for data leaders: catching bad data before the board does
What data contracts are, why they stop upstream changes from breaking reports and models, and how to introduce them without a multi-year governance program.
Assessment
What a data platform assessment should include: a 30-point checklist
A checklist for data leaders commissioning a data platform assessment: architecture, pipelines, cost, governance, AI readiness and the roadmap you should get at the end.
Platform choice
Databricks vs Snowflake for banks and insurers: how to choose
A practical way for financial institutions to choose between Databricks and Snowflake by workload, team skills, governance and cost model, not by vendor benchmarks.
AI-ready data
RAG on enterprise data: how to keep document permissions intact
Why enterprise RAG assistants leak data they shouldn't, and the data engineering patterns that keep access controls attached from source document to answer.
Migration
Teradata to Databricks migration: a wave-by-wave plan that avoids the big bang
How to move a Teradata estate to the Databricks lakehouse in waves (inventory, code conversion, parallel runs and decommissioning) without losing the business's trust in the numbers.
Cost optimization
Where data platform costs hide: 12 places to look on Databricks and Snowflake
Twelve common sources of avoidable spend on Databricks and Snowflake, from idle compute to full refreshes, and how to find each one before your next renewal.
Engineering guides
Illustrated deep dives into database and search internals
Indexes, reads and writes, search engines and concurrency control.
Subscribe with any feed reader: RSS feed
Have 30 minutes? Get a senior architect’s view of your data platform.
No sales deck. You talk to the engineer who would lead the work.
Book an architecture review