What nobody tells you about putting an LLM in front of SQL

The demo that sells itself

Here's the demo. Someone types "what's my audience overlap between Amazon and TikTok last quarter?" into a chat box. A second later, valid SQL appears, runs against a privacy-safe clean room, and a chart comes back. No SQL knowledge required. No analyst in the loop. The room goes quiet in a good way.

I've given that demo. It works. It is genuinely magic the first time you see it.

It is also the easiest 10% of the problem, and almost everything written about "AI that writes SQL" stops exactly there — at the demo. This post is about the other 90%, the part that took me months, because I think it's the most honest thing I can tell you about building with LLMs in production right now.

The first query that broke

The platform I'm building puts a conversational agent in front of AWS Clean Rooms and Amazon Marketing Cloud — privacy-safe analytics across data that belongs to different companies, where nobody is allowed to see anyone else's raw rows. The agent's job is to turn a plain-English question into SQL that the clean room will actually accept and run.

The model I use is good. Claude Sonnet writes SQL that would pass code review at most companies. The trouble is that a clean room is not most companies, and clean room SQL is not the SQL the model learned from the entire internet.

The first time a real user asked a slightly non-trivial question, the agent produced a textbook-perfect query — a common table expression to stage the data, a couple of ROUND() calls to tidy the numbers, a NULLIF to dodge a divide-by-zero. Exactly what you'd write in Postgres. Exactly what I'd have written.

The clean room rejected all of it.

The dialect wall

Here's the thing nobody warns you about: the LLM is fluent in general SQL, and general SQL is the wrong language. Every restricted execution environment — clean rooms, AMC, federated query engines — speaks a deliberately crippled dialect, crippled on purpose for privacy or performance reasons. And the model has barely seen that dialect, because almost nobody writes it, because almost nobody has access.

A few of the walls I hit, specifically:

• No common table expressions. The model loves a WITH clause to structure a query readably. The clean room I target doesn't support them. Every multi-step query the model produced had to be flattened into nested sub-selects or template-built from scratch.
• You can't wrap aggregates. Write ROUND(SUM(x), 2) and it's rejected — wrapping an aggregate in ROUND or CAST trips the analysis rules. The model does this constantly because it's correct everywhere else.
• NULLIF and friends are out. Half the defensive tricks an experienced SQL author reaches for to handle edge cases simply aren't allowed.
• Aggregation thresholds. Privacy rules mean a query that would return a result set with too few users gets blocked entirely — and the model has no idea this rule exists, so it writes queries that are syntactically perfect and semantically forbidden.

None of these are bugs in the model. The model is doing exactly what it should: writing good, normal SQL. The environment just refuses good, normal SQL. The gap between "correct SQL" and "SQL this specific engine will run" is the entire problem, and it's invisible until you're standing in front of it.

What I actually had to build

The naive mental model is: "the LLM writes the SQL, we run it." The real architecture has the LLM in the middle of a pipeline that spends most of its energy keeping the model inside the lines.

A knowledge base of the rules. Before the model sees a question, it gets injected with a curated set of the dialect's restrictions, the available tables, and the column schemas for whichever of the data partners are connected. This is retrieval-augmented generation, but the retrieval isn't fetching documents to sound smart — it's fetching the constraints that stop the model embarrassing itself. Eleven partner schemas, each with its own quirks, each needing to be in context at the right moment.

Sanitizers that rewrite the output. This is the part I didn't expect to need. After the model writes a query, it passes through a sanitizing layer that strips or rewrites the things the clean room won't accept — unwrapping aggregates, removing CTEs, swapping disallowed functions for permitted equivalents. The model proposes; the sanitizer enforces. I have a separate sanitizer per engine, because AMC and Clean Rooms disagree about what's legal in different ways.

Template builders for the queries I can't trust the model to free-hand. For the highest-stakes query shapes, I don't let the model write raw SQL at all. It fills slots in a known-good, clean-room-compliant template — a flat SELECT I've already proven the engine accepts. Less elegant, far more reliable. The model decides what to ask; the template decides how it's allowed to be asked.

A fallback chain, because one model is a single point of failure. Bedrock is primary. If it times out or throttles, it falls through to the Anthropic API directly, then to OpenAI, then — if everything is on fire — to a keyword-based path that at least returns something useful. Production AI is less "call the model" and more "call the model, and have three plans for when you can't."

By the time you add all that up, the LLM is maybe a fifth of the system. The other four-fifths is the unglamorous scaffolding that turns a brilliant, overconfident generalist into something that behaves inside a narrow, unforgiving box.

The lesson: the model was never the hard part

This is the part I want to land, because I think it generalises far beyond clean rooms.

When people imagine building with LLMs, they imagine the model is the product and everything else is plumbing. In my experience it's the reverse. The model is a commodity — a very good commodity, getting better every few months, and largely interchangeable in my stack with three lines of config. The value, and the work, is in everything wrapped around it:

• The retrieval that gives it the right constraints at the right moment.
• The validation and sanitizing that catches its confident mistakes before they reach production.
• The templates that remove its freedom exactly where freedom is dangerous.
• The fallbacks that keep the system standing when the model isn't.
• The domain knowledge — knowing that clean rooms reject wrapped aggregates — that no amount of model scale will give you, because it's locked behind access nobody trained on.

That last one is the moat. Anyone can call an LLM. The hard, durable, valuable knowledge is what the model gets wrong in your specific domain, and how you catch it. That's not a prompt. That's months of watching it fail in ways the demo never showed you.

Where this leaves me

I'm more bullish on agents than ever, and more allergic to the demos than ever — which I've decided is the correct way to feel.

The demo is real. The agent really does turn a sentence into a chart. But the distance between that demo and a thing you can put in front of a paying customer with other companies' sensitive data flowing through it is measured in failure modes you can't see from the stage. CTEs that don't exist. Aggregates you can't round. Privacy thresholds that block a perfect query. A model that is wrong with total confidence, at scale, unless something downstream is built to stop it.

So when you read the next "AI writes your SQL / your code / your contracts" launch, hold one question in your head: what happens on the query the demo didn't show? That question is the whole job. Everything before it is the easy 10%.

The magic is real. It's just that the magic is the cheap part — and the expensive part is the only part that matters.