← Back to projects Personal project

Dynamic-ETL. An AI platform, three engines deep.

Turn any document or messy data source into clean, structured records — with AI doing the heavy lifting and a human making every decision that matters. Four industry modules are built on the same three engines.

Type
Independent personal project
Backend
Python · FastAPI · SQLAlchemy · Postgres
AI
Gemini (default) · OpenAI failover
Frontend
React · TypeScript · Vite · Tailwind
3core engines
4industry modules on top
12safe operations the AI may choose from
0records stored without a human decision

Architecture

Build the platform once. Add industries on top.

Every industry module reuses the same engines, the same model gateway and the same human-review discipline. Adding the fourth module needed almost no new machinery — the reuse was the point.

Industry modules built only from the layers below
  • RealEstateOS AI sales agent answering leads from published content only
  • Healthcare AI Administrative workflows, with clinical content blocked in code
  • LogisticsOS Daily route plans reviewed stop by stop before dispatch
  • HospitalityOS Guest conversations, reservations and complaints
Engines
  • 1 · AI DocumentOS Documents in, validated records out — every record reviewed by a person
  • 2 · AI Data Agent Designs schemas and data mappings, tests them on real samples, repairs them
  • 3 · AI DataOps Approve a pipeline once, run it daily, hold anything anomalous
Foundation every model call goes through it
  • Model Gateway One place that knows the vendor; routes by task; refuses forbidden content
  • Human review console Every pending decision across products, in one queue
  • Audit log Append-only record of every stage and decision
  • Postgres + REST API One process serves the API and the React console

Engine 1 · AI DocumentOS

Confidence picks the lane — never whether a human looks.

Documents flow through eight stages. The model extracts and validates; a confidence router decides whether a record goes to a quick check or a detailed review. Either way, nothing reaches the data store without a person accepting it, and every stage writes to an append-only audit log.

  1. 1Ingest
  2. 2Classify
  3. 3Parse
  4. 4Extract
  5. 5Validate
  6. 6Confidence router
  7. 7Human review
  8. 8Commit
Step 1 · Deterministic

Ingest

Documents arrive by upload or API — PDFs, scans, spreadsheets — and each one gets an audit-log entry from the very first moment.

Step 2 · AI step

Classify

The model identifies what kind of document this is, which decides the schema it will be read against.

Step 3 · Deterministic

Parse

Text is read natively where the file has it, and through OCR where it is a scan.

Step 4 · AI step

Extract

The model fills the schema's fields from the text, page by page, with confidence for each value.

Step 5 · Check

Validate

Rule checks (types, formats, ranges) plus a semantic check catch values that look right but aren't.

Step 6 · Check

Confidence router

High-confidence records go to a fast-lane check; uncertain ones get a detailed review. Confidence picks the lane — never whether a human looks.

Step 7 · Human decision

Human review

A person accepts or rejects every record. Nothing is auto-approved.

Step 8 · Deterministic

Commit

Accepted records are written to the data store, with the full trail of who decided what, and when.

Engine 2 · AI Data Agent

The AI picks from a menu. It never writes code.

Point the agent at sample rows from a messy CSV, spreadsheet or JSON source and it produces a mapping into a clean schema. The mapping is a recipe built only from twelve safe operations — the system's own interpreter carries it out, and anything outside the menu is rejected before it can run.

{ "target_field": "unit_price",
  "source_path":  "items[0].price",
  "ops": [ {"op": "strip"},
           {"op": "regex_extract", "pattern": "[0-9.]+"},
           {"op": "to_number"} ] }
stripupperlowerto_stringto_numberto_integerto_booleanregex_extractsplit_takedate_parseconstdefault

Why a recipe instead of code?

If a model writes Python that then runs on real data, a crafted file or a manipulated model response becomes code running with the system's own permissions — so you need a sandbox around it. Here there is nothing to sandbox: "read a file" or "call the internet" simply can't be expressed in the recipe language.

It is also cheaper. Once a human approves a mapping, a 50,000-row file runs through it deterministically — no model call per row.

Understand → generate → test → repair → human decides

Generateschema or mapping from samples
→
Teston the real samples, through the production engine
→
Repaironly the failing field, with the failure as context
→
Promotea person accepts or rejects

A field the agent can't map correctly is left explicitly unmapped rather than shipped as a silent bad guess.

Engine 3 · AI DataOps

Approve the logic once. Watch every run.

✅

Approve once, run daily

A pipeline is generated and tested by the Data Agent, then approved by a person as logic — not re-reviewed on every run.

📈

Statistical anomaly checks

Each run is compared with the last clean baseline: row-count drift, null-rate drift and schema drift. Plain statistics, not a model's opinion.

🧭

Hold, don't hide

Anomalous runs keep their rows in a holding area until a person dismisses or reloads them. Clean runs flow to the warehouse with column-level lineage.

Industry modules

Four industries, one set of engines.

RealEstateOS

An AI sales agent for WhatsApp and web chat with three paths per message: answer straight away from published content, ask a person for a fast live approval when a reply would commit the brokerage (with a deadline and a safe fallback message), or batch low-stakes CRM updates for a daily skim. Whether a reply is grounded is checked in code, not by asking a second model.

Healthcare AI

Administrative workflows only — scheduling, prior authorisations, patient messages — reusing RealEstateOS's approval machinery. A conservative classifier flags clinical content, and the Model Gateway refuses flagged input by type, before any model is reached. Every patient-record read is itself logged.

LogisticsOS

Daily route plans from a deterministic bin-packing optimiser — no model call at all, because the task doesn't need one. Hundreds of stops are reviewed in a fast lane plus per-stop detail, and dispatch happens only when every stop is resolved.

HospitalityOS

Guest conversations reuse the real-estate routing unchanged. New here: card numbers are scrubbed from messages before they are stored or modelled, and the gateway refuses payment-card data for every task — a second compliance domain on the same refusal mechanism.

Key design decisions

What made it hold together.

Safety

Least-powerful output wins

  • AI chooses operations from a menu instead of writing code
  • Forbidden content is refused at the gateway, by type — not by a prompt a model could be talked out of
  • Card numbers redacted before anything is stored or modelled
Judgement

Use AI only where it earns its place

  • Route optimisation: a deterministic algorithm
  • Eligibility rules: a table lookup
  • Anomaly detection: statistics against a clean baseline
  • Grounding checks: code, not a second model grading the first
Trust

Humans decide; machines route

  • Confidence chooses the review lane, never whether review happens
  • Live approvals carry deadlines and safe fallbacks
  • One vendor-neutral gateway, with an offline provider so everything is testable without keys
  • An append-only audit log behind every stage

One honest lesson from reuse: the real-estate "viewing request" detector didn't transfer to hotels — guests asking about a "sea view room" would have triggered a property-viewing approval. Routing transferred cleanly; intent detection needed a per-industry switch.

Want to talk about it?

Happy to walk through the architecture and trade-offs.