Audit-ready compliance review needs more than an LLM

AI-enabled OCR and LLM workflows developed at Capital One reduced manual compliance review effort by about 60%. The remaining work included potential violations, high-risk documents and cases with unreliable extraction or unverifiable evidence. Overall accuracy can obscure dangerous misses, making violation recall for each rule a more useful safeguard than an aggregate score.

A proposed production architecture screens OCR quality page by page and routes unfamiliar document layouts to reviewers. Instead of asking for a broad compliance verdict, it evaluates individual rules through structured outputs that allow an insufficient-evidence response. Pass and fail findings require evidence checked against the extracted document, with different evidence requirements for rules assessing whether something is absent.

Only clean, lower-risk documents qualify for automatic clearance. Potential violations, uncertain findings and high-risk cases go to people, accompanied by highlighted evidence and routing reasons. Compliance professionals should label evaluation data, while changes to prompts, models, rules or OCR settings should trigger fresh testing against rule-specific violation recall baselines.

Auditability requires append-only decision records containing document identifiers, system versions, evidence, raw model responses and reviewer outcomes. Blind human sampling of automatically cleared documents can estimate escaped errors in production. Changing requirements remain difficult: affected decisions should return to human review until rule definitions, prompts and evaluation labels are aligned.