Building an AI document reader with a human in the loop

The problem

A few months ago I built a small private app for my family. Important documents arrive the way they always do now: as photos taken on a phone, forwarded on WhatsApp, often handwritten, sometimes at an angle, occasionally upside down. We needed them turned into something structured — searchable, comparable over time, and impossible to lose.

The obvious answer in 2026 is "give it to a model". The less obvious part is everything around the model. This is a teardown of how it works, what went wrong, and what I'd tell anyone building something similar.

The architecture

It's deliberately small and serverless:

1. Upload. The browser asks for a presigned URL and uploads the file straight to a private, encrypted S3 bucket. The app records the document first, and rolls that record back if the upload fails — so there are never "phantom" documents.
2. Trigger. The new object fires an S3 event, which invokes a Lambda function.
3. Read. The Lambda sends the PDF or image to Claude on Amazon Bedrock, which reads documents and images natively — no separate OCR step. The prompt asks for a strict JSON structure covering five kinds of information, plus a confidence level and a list of anything it was unsure about.
4. Draft. Every extracted item is written to DynamoDB with reviewed: false, grouped under a batch ID for that document, alongside a summary card.
5. Review. In the app, each document becomes a card. Every row is editable; you can add things the model missed and delete things it got wrong. Save & approve commits the batch.
6. Use. Only approved records appear in the rest of the app — the history, the charts, the reminders.

Steps 3 and 4 take around ten to thirty seconds. The page polls while it waits, so the review card simply appears.

The model is a first draft, not a final answer

Here's the honest accuracy story. On a clear printed document, extraction is close to perfect. On handwriting, a good run gets most items right — and every so often it confidently reads the wrong page, or turns a scrawl into a plausible-but-wrong word.

The temptation is to keep tuning the prompt. I tried; it plateaus fast. What actually worked was accepting that the model drafts and a human approves, and then making the human step as fast as possible:

• Every row is editable in place, with the original document one click away.
• + Add buttons for every type of record, because a missed item is more common than a wrong one.
• One click to discard a bad read entirely, which deletes the stored file too.

Designing for review turned an 80%-accurate model into a 100%-accurate system. That's the most transferable lesson in this whole post.

Giving the model context it can't see

Handwriting is ambiguous; context resolves ambiguity. The prompt is given a short list of names the family has already confirmed in previous documents, so a scribble that could be three different words gets matched to the one that's actually relevant. It's also told to ignore printed form labels, which it otherwise loves to extract as if they were data.

This is retrieval-augmented generation at its most humble — no vector database, just "here's what we already know" — and it made a bigger difference than any wording change.

Everything around the model

Most of the code isn't about AI at all. It's about not losing or corrupting data:

• Dates are hostile. In India, 12/05 means 12 May; the model sometimes read it as 5 December — dating records months in the future. Model-read dates are now validated and clamped, and the filename date (YYYY-MM-DD at the start) is used as a fallback.
• Long documents overflow. A dense multi-page document once produced more output than the token limit, and the truncated JSON failed to parse. The limit went up, and a truncated response now throws an explicit error that shows on the review card instead of failing silently.
• Writes fail partially. DynamoDB batch writes can return unprocessed items. They're retried with backoff, and anything still left becomes a visible "could not read" card — never silently missing rows.
• Approval order matters. The approve step confirms every row before deleting the summary card. If it fails halfway, the batch is still there to retry, instead of leaving half-confirmed records that nobody can see.
• Duplicates happen. People forward the same photo twice. Approval skips a record that matches an already-confirmed one — but never one a human has edited, because a human edit is a statement of intent.
• Concurrency is real. Two uploads in the same millisecond once risked sharing a batch ID. IDs now carry a random suffix.

None of this is glamorous. All of it is the difference between a demo and something your family trusts.

Guardrails

• Private by default. The storage bucket blocks all public access, is encrypted and versioned; the app sits behind an email login that is checked again on every request.
• Least privilege. The extraction function can call exactly the two models it needs and write to exactly one table.
• Bounded cost. At most three documents are read at once — extra uploads queue — and the API is rate-limited per route.
• Advisory, never prescriptive. Anything the model suggests is a draft for a person to accept, edit or ignore.

What I'd tell anyone building this

1. Design the review step first. Assume the model is wrong some of the time and make correcting it delightful. That's the product.
2. Feed it what you already know. Small, relevant context beats a longer prompt.
3. Validate every value a model returns. Especially dates, numbers and anything that drives an action.
4. Make failures loud. A card that says "could not read this, here's why" beats a silently empty list.
5. Keep it boring underneath. Presigned uploads, events, functions, a table, a bucket. The interesting part should be the experience, not the plumbing.

The model was the easy part. Everything around it is what made it useful.