Case study · Document AI, Instrumental Partners · 2026

AI review of quality clauses in purchase orders

The quality team of a US defense electronics manufacturer reviews every customer purchase order by hand. They find each quality code, look it up in the right revision of that customer's manual, decide whether it applies to the whole order or to one line, and fill in a checklist. A missed code is expensive. I built the tool that does the first pass, at about two cents per order.

Context: Instrumental Partners · client confidential
Cost: $0.022 per order (measured)
Stack: Python · OCR · GPT-4o · FastAPI

The rules the system follows

Quality reviewers need to trust every line, so the design starts from a short set of rules:

Every extracted requirement carries its exact text, and that text must be found word for word on the page.
Every requirement carries its evidence: document, page and position.
Applicability is explicit: whole order, one line item, or ambiguous for a human to decide.
Codes resolve by exact match against one revision of the customer's clause manual.
Whether a code is routine, uncommon or new is computed in SQL from order history. The LLM never decides it.

Two readers that check each other

Orders arrive as native PDFs and as scans. Pages are rendered with PyMuPDF and read twice, by OCR (RapidOCR at two resolutions) and by a vision model (gpt-4o). The two readings are reconciled with a confusion set for characters OCR mixes up, like O and 0 or I and 1. An OpenCV reader measures ink density to detect ticked checkboxes. On native-text pages, vision added no correct codes in testing and invented three, so it is skipped there, which covers about two thirds of all pages.

Each customer writes clause codes in its own format. The system learns those shapes from the customer's documents and verifies matches deterministically. Federal FAR and DFARS clauses are bundled as data. For customers who publish no clause manual (8 of 19 in the archive), an order-only mode still extracts and flags requirements.

Numbers

Measured cost: 19 orders for $0.42, so $0.022 per order. The full backlog of 744 orders comes to an estimated $16.50.
Archive swept: 2,700 PDFs across 19 customers.
A 9-page, 24-line order that used to break OCR now reads 24 of 24 lines with every code.
An evaluation harness scores recall, precision, applicability, source accuracy and false confidence against hand-labelled orders. The targets are recall of at least 0.95 and false confidence of at most 0.05.

Stack

Python 3.12, FastAPI, SQLite, S3-compatible storage, a plain JavaScript interface, pytest and GitHub Actions, deployed with Docker on AWS EC2 behind a login. There is no vector database: retrieval walks the structure of the clause manual, which suits documents organised by code. Every design trade-off goes into a decisions log, which has 169 entries so far.