legit

The problem

Before a Tzu Chi Foundation department signs a contract, it must judge the risk: was a legal-office template used, is the risk higher or lower, and must the legal office review it? Staff had to find the right template among dozens and apply rules spread over several documents that sometimes contradict each other.

What I built

An internal web service. A conversational risk-assessment agent reads the contract and attachments, shows each tool call as a card and ends with a fixed-format report. An assistant finds the right template and helps fill it in. A case library keeps every submission with its sign-off trail.

How it works

1. A whitelisted extraction layer reads docx, PDF (with OCR for scanned pages), spreadsheets and slides, within size budgets. Anything left out is labelled.
2. The agent first checks that the file is a contract. It then searches 72 templates: an embedding shortlist of 30, reranked by a cross-encoder. When the top two are a known confusable pair, it asks the user one question.
3. Contracts that don’t follow a template are split into clauses. Each clause runs vector and trigram search in parallel over a 20,491-article law database. Quoted law text is read back verbatim from the source.
4. The agent submits a structured assessment. Fixed code turns it into the report and the routing.
5. A person presses submit. The case is stored with an event-sourced sign-off history.

Decisions

• Judgement in the agent, evidence in code. Python tools pass evidence and store results; the model decides. Report format and routing stay deterministic.
• No masking. Masked contracts made the model report false findings. Models run on-premise through the foundation’s own gateway. Logs carry no personal data and cite clause numbers only.
• Recall over speed for law search. The legal office asked that no relevant law be missed. A slower fallback search stays on, and the deadline was raised to 600 s.
• Every retrieval change re-runs its eval. hit@1 may not fall below 0.949 for templates or 0.821 for laws.

Outcome

legit is an internal pilot. The legal office supplied 10 real cases as a golden set (personal data replaced with fictional values), run 3 times each in fast mode (law fallback search off); 9 cases were scored. After reliability fixes, failed runs dropped from 10 to 2 of 27, and run-to-run agreement on cases where all three runs succeeded rose from 56% to 81%. Case-type accuracy by majority vote is still 44% (4 of 9 cases). The hard call, whether a contract is a filled-in template, waits on legal-office rulings. At the 2026-09-03 checkpoint, 1,127 pytest (5 skipped, 1 xfail) and 340 Vitest tests were green. Built with Claude Code in a spec → plan → implement → review loop; every ruling goes into an append-only decision log.