ETL & ELT pipelines that survive contact with production
We build ingestion and transformation systems across source systems, warehouses, and cloud providers — batch, streaming, and change data capture. This page covers how we decide what to build, and what we have shipped.
Ingestion patterns
The ETL-versus-ELT question is really a question about where transformation compute should live and who owns the transformation logic. We default to ELT: land raw data in the warehouse first, transform it there with SQL under version control. Modern warehouses are cheaper at transformation than a separate compute tier, raw data stays replayable when requirements change, and transformation logic becomes reviewable by analysts rather than locked inside pipeline code.
We reach for ETL in three cases: when data must be masked or dropped before it is allowed to land, when the source emits a format the warehouse cannot parse economically, and when the volume reduction from pre-aggregation is large enough that landing raw data would dominate cost. Those are real cases — they are just not the default.
Batch vs. streaming: the decision framework
Streaming is not a maturity level, it is a latency requirement with a cost attached. We start from what the business actually does with the data and work backwards.
| When | Choose | Because |
|---|---|---|
| Decisions are made on daily or weekly cadence | Batch | Scheduled loads are cheaper, simpler to reason about, and trivially replayable. |
| Dashboards refresh hourly; freshness matters but minutes do not | Micro-batch | Incremental models on a 15-60 minute cadence get most of the benefit at batch cost. |
| An operational system reacts to the data automatically | Streaming / CDC | Sub-minute latency is a functional requirement, not a nicety. |
| Volume is spiky and mostly idle | Event-driven serverless | Pay-per-invocation beats an always-on cluster that idles between bursts. |
The last row is worth dwelling on. Teams often reach for an always-on streaming cluster when event volume does not justify continuous idle compute. Moving that workload to event-driven serverless functions is one of the most reliable cost wins available — see the near real-time ingestion case study below.
Change data capture
For CDC we capture mutations at the source rather than polling for changes. The design questions that actually matter are not which CDC tool to use — they are what happens when a record fails to land, and whether a replay produces duplicates.
- Dead-letter queues are mandatory. Transient insert failures and schema mismatches must divert to a DLQ with a retry handler on exponential backoff. Silent data loss is far worse than temporary latency.
- Idempotency by construction. Merge on a stable business key so a replayed event updates rather than duplicates. This is what makes recovery safe.
- Schema validation before the warehouse. Reject malformed payloads at the edge, where the error is cheap and attributable, not three models downstream.
- Late and out-of-order arrivals are normal. Partition by event time, not ingestion time, and make incremental models tolerate a lookback window.
Orchestration and transformation
Orchestration handles ingestion, dependency order, retries, and alerting; dbt handles transformation as version-controlled SQL. Keeping that boundary clean is what lets analysts own business logic without touching pipeline infrastructure.
At volume, incremental models stop being an optimisation and become a requirement — full refreshes blow through compute budgets and miss SLA windows. Partitioning by event date and clustering by the primary lookup key is usually the difference between a five-second query and a five-dollar one. DAGs are written to be idempotent so a retry is always safe.
Data quality and testing
Every pipeline run executes tests: not-null and uniqueness on keys, referential integrity across models, accepted-value checks on enumerations, and freshness assertions against source tables. Versioned data contracts define the schema each consumer can rely on, so a breaking upstream change fails a build rather than a dashboard.
Alerting is tied to SLAs rather than to individual task failures. A retried task that still lands inside its window is not an incident; a task that succeeded but produced zero rows usually is.
Cost model
Warehouse spend is dominated by bytes scanned and idle compute. We attack both: partition and cluster so queries prune aggressively, use incremental models so each run touches only new data, and move spiky workloads off always-on infrastructure onto event-driven execution. Where users run ad-hoc queries, a dry-run byte estimate before execution prevents a single careless query from becoming a line item.
Selected data engineering work
Pulled from the case study catalogue by capability — this list updates itself as we publish.
SAP ECC to Databricks Lakehouse: Enterprise CDC Architecture on Azure
Client: Global Industrial Manufacturer
Architected an enterprise Lakehouse solution migrating 5M daily CDC records from SAP ECC (Oracle) to Azure Databricks. Features ODP/OData ingestion, automated PySpark schema evolution & drift alerting, multi-tier data quality quarantine, and Serverless compute optimization.
Near Real-Time Data Ingestion
Client: Global E-Commerce Platform
Architected a high-throughput near real-time streaming ingestion pipeline syncing NoSQL document stores to a cloud analytical warehouse via event-driven triggers.
Large-Scale Traceability System
Client: Global Logistics Enterprise
Developed a distributed data traceability system with micro-batch incremental transformation models, versioned data contracts, and automated SLA alerting rules.
Explore the catalogue
Every filter below is a shareable URL. These are generated from what is actually in the catalogue, so they never point at an empty page.
Have a pipeline problem?
Tell us the source systems, the volume, and the freshness the business actually needs. We will tell you whether it is a batch problem or a streaming one.