Introduction

Every data engineer who has run dbt run –full-refresh on a 400-million-row table at 2 a.m. knows a specific dread: watching warehouse credits burn while reprocessing everything, when only a fraction of the data actually changed.

dbt’s answer — incremental models, materializations, snapshots, exposures — is a decade of hard-won wisdom about one question: when something changes, how much of the world do you actually need to redo?

Here’s what nobody’s really written about yet: AI engineers are relearning this exact lesson right now, one expensive re-embedding job and one forgetful chatbot at a time. RAG pipelines, agent memory, and prompt versioning are all quietly reinventing dbt’s incremental philosophy — usually without knowing a mature vocabulary for it already exists.

This post is that bridge: dbt’s core concepts, translated into AI-native equivalents, with working code.

The Problem AI Systems Are Rediscovering

A very typical (and expensive) RAG pipeline:

Fix one typo in one of 5,000 documents, and this still re-embeds all 5,000 — the RAG equivalent of dbt run –full-refresh because one row changed. The same pattern shows up in agent memory (re-summarizing an entire conversation history every turn) and in evals (re-running the full suite for every prompt tweak, instead of only what’s affected). None of this is a knock on AI engineers — it’s a young field re-discovering an old, already-solved problem.

Translating dbt Concepts Into AI Systems

Same insight as dbt’s is_incremental() macro: filter first, process second.

One caveat worth knowing before you ship this: updated_at timestamps only catch changes to source content. If your chunking strategy or embedding model changes, the source document can look “unchanged” while its vectors are stale. Store the content hash alongside chunking_version and embedding_model_version, and treat any of the three changing as a reason to reprocess. The hash catches content changes; the version fields make pipeline and model changes explicit.

Incremental Strategies and Their AI Counterparts

Use “append” where you needed “merge,” and you get duplicates — in dbt, duplicate rows; in an agent, a context window with three contradictory versions of the same fact.

🧭 Quick self-check: Does the same fact ever get updated more than once? No → Append. Yes, and you only care about the current value → Merge. Yes, and you need to know what it used to be → you need SCD-2, not a simple merge (next section).

Practical Implementation Lessons

Two production details matter once incremental embeddings are running:

• • Delete handling: If a source document is removed, its chunks and vectors should be deleted or tombstoned too. Otherwise, retrieval can continue surfacing content that no longer exists.
  • Idempotent re-embedding and upserts: When the embedding model or chunking version changes, reprocess the affected documents and upsert using a deterministic key such as document_id + chunk_id + chunking_version + embedding_model_version. A retry should update the same vector rather than create duplicates.

Bringing Slowly Changing Dimensions to AI Memory

The single most underused idea in AI system design: not all memory should behave the same way when it changes.

SCD-1 facts (overwrite, no history needed): a corrected phone number, a fixed CRM typo. There’s no value — and real risk — in an agent “remembering” the old, wrong version.

SCD-2 facts (history matters): evolving product preferences, a patient’s medication history, a client’s risk tolerance over time. Treat these as SCD-1 and you lose the ability to explain why the AI is behaving the way it is now — an audit and compliance liability in regulated industries. Treat everything as SCD-2, on the other hand, and you get context-window bloat from facts that never needed history.

dbt’s snapshot pattern (dbt_valid_from / dbt_valid_to) maps directly onto an AI “memory snapshot”:

When an agent is wrong, “knowing what it believed and when helps identify the root cause”—easy with snapshot history, difficult with a single mutable blob.

Bringing dbt Governance Patterns Into AI

• • Pre/Post-hooks → guardrails: dbt runs SQL before/after a model executes. AI pipelines need the same two moments: pre-hooks (PII scrubbing, prompt-injection checks, before the LLM call) and post-hooks (safety classification, schema validation, memory writes, after the call). Treating guardrails as first-class pipeline steps — not an afterthought bolted on post-incident — is the whole point.
  • Source freshness → context freshness: dbt checks if raw data is stale; AI systems rarely check if retrieved context is stale — yet a RAG system citing three-week-old pricing will do so with exactly the same fluent confidence as correct information. A simple warn_after / error_after contract per source (docs: warn at 3 days; pricing: error at 6 hours) catches this before a user does.

# context_sources.yml

sources:
    - name: pricing_docs
         freshness:
            warn_after: {count: 3, period: hour}
            error_after: {count: 6, period: hour}

  - name: product_docs
         freshness:
             warn_after: {count: 3, period: day}
             error_after: {count: 7, period: day}

• • Exposures → dependency declarations: dbt exposures document what downstream thing depends on a model, so a change is traceable before it ships. Most AI stacks have no equivalent — nobody knows what breaks if you swap an embedding model until something breaks. A simple ai_exposures.yml (which prompts/models/indexes feed which features, owned by whom) turns “who broke the chatbot?” into “oh, I broke it” — days earlier and much cheaper.

# ai_exposures.yml

exposures:
    - name: support_chatbot_widget
         type: application
         owner: support-eng
         depends_on:

- prompt: refund_policy_V3
- index: kb_articles_embeddings
- model: text-embedding-3-large

• • dbt Jobs → CI for prompts: The highest-leverage idea to steal: run a golden-set eval suite against any prompt or retrieval-config change before it merges, exactly like dbt’s CI job tests SQL changes before merge.

Case Study: Building a More Reliable TicketMind

A support-bot team ships v1 with: full ticket history stuffed into every prompt, a nightly full re-embed of 12,000 articles, customer facts as one mutable JSON blob, and no eval gate on prompt changes.The failures arrive on schedule: a 45-minute nightly job re-processing 11,950 unchanged articles for 50 new ones (Week 3); an unanswerable refund dispute because facts were overwritten with no history (Week 6); a “make it concise” prompt tweak that silently drops safety disclaimers, caught only after a customer complains (Week 9); and degrading answer quality as 40+ tickets of irrelevant history crowd out the context window (Week 11).

The fix is incremental, fittingly: switch to incremental (merge) embedding, split customer facts into SCD-1 (overwrite) vs. SCD-2 (snapshot) tables, scope context to current merged facts instead of full history, add a CI eval gate on the refund-policy prompt, declare an exposure naming compliance as owner, and set a freshness contract on the knowledge base. None of it required new infrastructure — just applying discipline the data team already had.

A Practical Guide to Getting Started

• • Audit your “full refresh” points — fix the most expensive one first.
  • Classify your top 10 facts as SCD-1 or SCD-2.
  • Write down a freshness contract per data source.
  • Declare exposures for your top 3 production AI features.
  • Add one CI eval gate on your most-changed prompt.
  • Formalize pre/post-hooks instead of scattered if checks.
  • Document your memory schema like a dbt model.

None of this needed a new framework or a research paper — just borrowing ten years of data engineering discipline and pointing it at a problem AI teams are currently re-discovering the expensive way. Next time you’re designing agent memory or debugging a RAG system that “doesn’t feel fresh,” ask the dbt question instead of the AI question: not which model should I use, but what actually changed?

Happy building (incrementally)!