Get in touchContact

All work

Work · Public project

Matchpoint

Agents that investigate invoice exceptions and propose a resolution, inside a harness where deterministic code, not the model, decides what moves money.

Status
Open source · Nov 2026
Stack
  • Claude: Haiku, Sonnet, Opus
  • Python
  • FastAPI
  • SQLite
Disclosure
Public project
Demo
Video
Play on YouTube0:47
Matchpoint demo
Demo: an invoice note tells the AI to change the bank account and skip the callback; the agent holds the payment, a gullible model is blocked, and a large payment waits for a person.

Project videos load from YouTube only when you press play.

  • Matchpoint resolution card: Held for bank verification, because the remit-to bank details do not match the vendor master and the invoice carried instructions aimed at the AI.
  • Matchpoint approval inbox: a controller approval required for a payment above the release limit, with Decline and Approve release buttons.
  • Matchpoint evaluations page: no unsafe action executed across the evaluation runs, with results for live Claude, the reference run, injected faults and a gullible model.
A payment held for bank verification, a controller approval, and the evaluation results.

What it is

Matchpoint handles the invoices that fail automatic matching: mismatched amounts and quantities, duplicates, missing receipts and changed bank details. Six Claude agents triage each one, gather evidence from the ERP, read the payables policy and propose a resolution with an email to the vendor.

A policy engine then checks every proposal against the ERP’s own records rather than the agent’s account of them. Large payments wait for a person to approve, and a model that tries to release a payment the policy forbids is blocked.

Process: what we built

The agents run inside a small state graph of our own: every step is checkpointed, a run can pause for a person’s approval and resume, ERP writes happen exactly once even after a crash, and any error fails safe to a person. Text from vendors is screened for injected instructions and passed to the model as untrusted data.

An evaluation suite of 52 labelled invoices, including fraud and prompt-injection cases, runs offline against a deterministic reference model, with faults injected on purpose, and live against Claude.

  1. 01Triage
    • Haiku sorts and scores risk
  2. 02Investigate
    • Evidence from the ERP and the policy
  3. 03Propose
    • Resolution and vendor email
  4. 04Check
    • Policy engine checks ERP facts (where the system decides)
  5. 05Approve
    • A person signs off on large payments
System schematic, drawn from the project’s facts. The outlined item is where the system decides.

Results and limits

Results

  • Live on Claude, 13 of 13 evaluation cases passed.
  • With 25% of model calls failing on purpose, 99.4% of runs still passed.
  • A deliberately gullible model made 63 unsafe proposals; the policy engine blocked all 63, and none ran.
  • No unsafe action ran in 637 evaluation runs.

Limits

  • The ERP, the vendors and the invoices are simulated demo data.
  • The live Claude evaluation covers the 13 base cases and was run by hand; continuous integration runs the offline suite.

Building something like this for a client?

We can build it under your agency’s brand, for a fixed project fee.

Get in touch contact@neurosmith.co