Data platforms · AI engineering
Technical Lead Data Engineer

AI amplifies judgment. The engineer is the edge.

I run a global logistics company's Databricks lakehouse end to end, and I build with AI agents that carry the half of the work that doesn't need me.

Email Teams Power BI Web Fabric Wrike Databricks GitLab ROUX THE EDGE

My judgment, built into each workflow.

650
tables
~700
GB / day
900+
workloads
1 019
steps in one unattended AI run
  • Cloudflare
  • Azure Databricks
  • Microsoft Azure
  • Claude
  • Claude Code
  • Power BI
  • Astro
  • GitLab
  • Microsoft Fabric
  • Resend
  • Unity Catalog
  • Playwright
  • VS Code
  • Python
Shipped

Built, tested, shipped.

Two systems on different stacks, built the same way. Each number below comes from code, logs or direct measurement.

The company's data platform

One platform, one loader per layer, nothing hand‑written

Exhibit A
Scope
data platform, owned end to end since 2023
Scale
16 source schemas, 650 tables, 12 scheduled jobs
Result
two workspaces, one metastore, every load metadata-driven
A global freight and logistics company

The company's central data platform (a "lakehouse"), where every number comes from, plus the reports built on top of it. A delivery partner stood it up; I built it into what the company runs on today.

Every table loads the same way: one control row per load, one generic loader per layer, nothing hand-written per table. Twelve scheduled jobs run it every day.

900+ active control rows across 8 control tables; about 700 GB a weekday, measured on the live cluster

  • Took over a platform a delivery partner had stood up and built it into the one the company runs on: the framework, the jobs, the structure, the rules. More than 900 metadata-driven workloads move about 700 GB a weekday across 16 source schemas and 650 tables.
  • Moved the analytics workloads off SQL Server onto the lakehouse. Phases one and two of the round trip are in production and the legacy objects are specified for retirement. Each layer reads only the layers before it.
  • Two workspaces under one metastore, production bound read-only to test. Every change goes through GitLab CI, where a validator checks each merge request for a layer reading upward, and secrets live in Key Vault, never in code.
  • The reports on top are the small part. A 34-page operations report was rebuilt as eleven thin ones in eight working days; refresh went from 92.9 minutes to 4.1.

Azure Databricks · Unity Catalog · Fabric · Power BI · GitLab CI

How every table loads

One daily run reads the rows and fans out

Exhibit B
Scope
the load framework: control tables, generic loaders, one job graph
Scale
8 control tables, 900+ active rows, 12 scheduled jobs
Result
one control row per load, no hand-written code per table
Metadata-driven framework

Every load on the platform is a row in a control table, run by one generic loader per layer. The main daily job reads the rows, fans out, and runs them in parallel.

Extraction from the sources over JDBC runs 10 wide; Delta-to-Delta steps run up to 100 wide. Adding a source table is one new row.

900+ active rows across 8 control tables, one per layer plus custom, API, file and SharePoint loads; for-each fan-out with per-group concurrency

  • Three generic loaders, one per layer: silver mirrors the source with types, warehouse keys every row to where it came from, gold builds what BI reads. A table's behaviour lives in its control row, not in code.
  • Change data from the busiest source arrives as an audit table of change events, ingested inserts-only behind an LSN watermark.
  • Each layer reads only the layers before it. A validator checks every merge request for a layer reading upward, and the rule is written down where the loaders are.

Databricks Workflows · for-each tasks · Delta Lake · JDBC · Python

Two workspaces, one metastore

Production bound read‑only to test, no secret in code

Exhibit C
Scope
workspaces, network, storage, identity, secrets
Scale
2 VNet-injected workspaces, 2 storage accounts, 1 metastore
Result
every secret in a Key Vault-backed scope, none in code
Azure landing zone and identity

Two Databricks workspaces, production and test, each injected into its own virtual network, under one Unity Catalog metastore. The production catalog is bound to production and read-only from test.

Storage is two ADLS Gen2 accounts, production and test, each with a landing and a lake container, reached through external locations and one managed identity via an access connector.

Classic compute is VNet-injected; the corporate firewall allows only the two classic-plane egress addresses, so serverless compute cannot reach the on-prem sources.

  • Identity is Entra groups synced into the workspaces, admins and developers, plus service principals for the jobs, Power BI and the linked server. Scheduled work runs as a service principal, never as a person.
  • Secrets live in Key Vault-backed secret scopes. Zero literals in code.
  • On-prem SQL Server and PostgreSQL are reached through the firewall over ExpressRoute or VPN. Fabric mirrors the gold layer for Power BI.

Azure Databricks · Unity Catalog · ADLS Gen2 · Entra ID · Key Vault

konradvanstaden.com, the homepage

konradvanstaden.com

19 pages, self-hosted end to end

Exhibit E
Scope
author's site rebuilt with booking
Scale
19 pages, one for each of 18 books
Result
19 static pages, fonts and scripts all self-hosted
Author's website, outside work

An author's website, rebuilt from the ground up with a booking system behind it.

The old site was an ageing Joomla install. The new one is nineteen light pages with every font, script and image served from his own domain. A content change is one rebuild and one scripted deploy.

19 static pages, self-hosted assets; the only outside call on load is the host's analytics beacon

  • A light homepage plus a page for each of his 18 books, and nothing on them is fetched from anyone else's servers apart from the host's own analytics ping. Rebuilt from a legacy Joomla site into 19 static pages. Self-hosted fonts and scripts; social embeds load only on click.
  • The booking system went live quietly. Until its last piece was in place the site kept using the old email link, so nothing ever broke for a visitor. Booking backend with no flag day: ships dark, falls back to yesterday's mailto until its key lands.
  • The switch to the new site was scripted, with a five-minute way back if anything went wrong. Checked on the real address before calling it done.

Astro · Cloudflare Pages · Pages Functions · Resend

    The work splits in two. One half runs to the end without me. The other half comes back to me in ordinary words before it ships.

    The way I work is the product.

    The machine’s half

    What runs without me.

    The longest single run went 1 019 steps before it needed me again. 328 unattended runs of 50+ steps, 114 of 100+, 29 of 200+. Counted from every transcript, 2026-07-02 to 2026-09-03.

    What can you do without me?

    I split the work into what needs me and what does not, then let it run the second pile to the end without stopping.

    Training wheels, then autonomy

    New automations run with me watching. After a few clean runs they run alone. Anything touching money, sending to the public, or deleting never runs alone.

    Bull in a china shop, then a diamond

    Get the rough version working fast, then cut it down until it is small and hard to break.

    The same setup does the dull work too. It sorts mail, handles tickets and writes up meetings. One session got into my home router, worked out why the line was bad, and filed the fault with the internet provider.

    My half

    What has to come through me.

    The gate is plain language. If it cannot be said in ordinary words, it does not ship. 437 of 5 020 typed turns across every transcript asking for it in plain terms, in one sentence, in dummy terms. Counted 2026-07-02 to 2026-09-03.

    Replay it back first

    Before anything gets built, the AI says the plan back to me in plain words. If the plan is wrong it dies there, before any code exists.

    Check the result, not the message.

    I don't trust "deploy succeeded". I open the live site and check the version on screen is the one I just shipped. Never trust a name, a dashboard or a green status. Read the rule back. Assert the deployed commit hash. Hit the live site with a fresh, uncached request.

    Audits never return zero

    A check that finds nothing is suspect. One or two small things per run is normal. Five new things every run means the thing being checked is changing under it.

    Two strikes before a rule

    The first time something goes wrong, I note it. The second time, it becomes a written rule, and the tools enforce it so I don't have to remember.

    One route
    contact@kosieroux.com

    Kosie Roux · Technical Lead Data Engineer · The machine runs what doesn’t need me. I check what does.

    UNITY CATALOG one metastore · two workspaces · production bound read-only to test SOURCES ERPWMSSQL ServerFiles & APIs two instances · JDBC3 versions · PostgreSQLon-prem and Azure SQLSharePoint · SFTP · REST 16 source schemas RAWSILVERWAREHOUSEGOLD files as they arrive the landing layer typed mirror of every source 16 schemas 650 tables star schema surrogate keys every row keyed to its source tables and views read by BI every number starts here READ BY Power BIFabric via SQL warehousesDirect Lake over a mirror CONTROL PLANE 8 control tables · 900+ active rows · 12 scheduled jobs · for-each fan-out one control row per load · one generic loader per layer · nothing hand-written per table each layer reads only the layers before it, never upward

    Sources → raw → silver → warehouse → gold, under Unity Catalog; read by Power BI and Fabric; a control plane of eight tables drives every load. Sanitised: no names, only counts already on this page.
    ONE ROW PER LOAD a table's behaviour lives in its control row, not in code · adding a source table adds a row CONTROL TABLES silverwarehousegoldcustomfiles & APIsrandom integrations one per source tableone per fact or dimensionone per BI tablehand-tuned loadsCSV, SFTP, SharePoint, RESTthe ones nobody planned 8 tables · 900+ active rows DAILY JOB one scheduled job for-each groups reads the rows, fans out, runs them one of 12 scheduled jobs ONE GENERIC LOADER PER LAYER silver loader warehouse loader gold loader from the sources · JDBC, 10 wide Delta to Delta · up to 100 wide Delta to Delta · up to 100 wide SILVER WAREHOUSE GOLD FAN-OUT 10 wide over JDBC from the sources · up to 100 wide Delta to Delta each for-each group carries its own concurrency, set per source in the control row the busiest source also feeds a change-event audit table, ingested inserts-only behind a watermark

    Eight control tables, 900+ active rows; one daily job reads the rows, fans out in for-each groups and runs one generic loader per layer. Sanitised: no names, only counts already on this page.
    ENTRA ID groups synced into the workspaces · service principals run the jobs, Power BI and the linked server scheduled work runs as a service principal, never as a person · named platform accounts for platform work ON-PREM SQL Server PostgreSQL ExpressRoute / VPN FIREWALL only the two classic-plane egress addresses PRODUCTION workspace · VNet-injected job clusters, Photon one all-purpose cluster JDBC pulls: the all-purpose cluster warehouse and gold: Photon jobs TEST workspace · VNet-injected same layout, same loaders reads production read-only through the catalog binding promotion SERVERLESS warehouses · health checks cannot reach on-prem KEY VAULT no secret in code secret scopes in both workspaces ADLS GEN2 two accounts, production and test · landing + lake containers · external locations one managed identity through an access connector backs every external location UNITY CATALOG · ONE METASTORE production catalog bound to production, read-only from test · test catalog open a separate catalog for AI-gateway work: production grants cannot be narrowed later FABRIC mirrors gold

    Two VNet-injected workspaces under one Unity Catalog metastore; two storage accounts behind one managed identity; Entra groups and service principals; Key Vault for secrets; Fabric mirrors gold. Sanitised: no region, groups or addresses.