EBS Invoice Reconciliation¶

Location: inception_recipes/ebs_invoice_reconciliation/
Status: Backend and console both live-validated end-to-end
Stack: React + Vite · Node.js/Express · FastAPI · LangGraph · Oracle ADB · OCI GenAI · OCI Document Understanding · FastMCP SQLcl · FastMCP Object Storage

What it does¶

Extracts supplier invoices detected in an OCI Object Storage inbox, validates them against approved suppliers/purchase orders/goods receipts via a deterministic three-way match, and prepares auditable reconciliation drafts for Finance approval. Never posts to EBS or releases a payment autonomously — every path ends at "drafted, pending human sign-off."

OCI Object Storage inbox ──poll (mcp_os)──► extraction_agent (DUS2.0 + LLM normalize)
                                                    │
                                    lookup_node (oic_client — stubbed OIC boundary)
                                                    │
                                    matching_agent (deterministic, no LLM)
                                          │                        │
                                MATCHED_READY/VARIANCE         EXCEPTION/MISSING_DATA
                                          │                        │
                                    draft_agent                exception_agent
                                          │                        │
                              Finance Approver ───► oic_client   AP Analyst
                              POST /drafts/{id}/approve          POST /exceptions/{id}/resolve
                              (the ONLY call that triggers
                               the stubbed OIC flow)

Built with Inception Skills' recipe-builder — see specs/USE_CASE_INTAKE.md for the filled-in intake form and specs/SPEC.md for the resulting AE-01..15 specification.

Manual workflow (Workflow tab)¶

The pipeline above runs automatically against a polled inbox. The console's Workflow tab gives an AP Analyst a second, on-demand path: upload the three source documents directly, map an invoice to a specific contract, and run the match interactively.

A contract governs an MSA (Master Supply/Service Agreement), not a single PO — confirmed against a real sample document ("Agreement No. MSA-2026-017 ... Associated purchase order: PO-45001982" — the agreement number is the document's real identity; the PO was only one example transaction under it). Many purchase orders can be issued against the same MSA over its term, all priced at the same negotiated line items, so the analyst maps an invoice to an MSA, not a PO — the same MSA can be selected for invoices citing entirely different PO numbers. A PO itself is a lightweight AP_PURCHASE_ORDERS → AP_CONTRACTS association (automatic pipeline only — see below), never a pricing entity.

Upload invoice ──► Object Storage (ap-invoice-inbox), no extraction yet
Upload contract ─► Object Storage (ap-contract-inbox) ──► DUS2.0 extract ──► register into AP_CONTRACTS / AP_SUPPLIERS
Upload receipt ──► Object Storage (ap-receipt-inbox)  ──► DUS2.0 extract ──► register into AP_GOODS_RECEIPTS
                                                                    │
Analyst selects invoice + contract (MSA) ──► POST /match ─────────┘
        extracts the invoice, looks up the selected MSA's supplier/pricing terms plus
        every goods receipt on file for the invoice's own extracted PO number (summed —
        a PO is routinely fulfilled across multiple partial receipts),
        evaluates every rule in matching_agent.evaluate_match_detailed() independently
        (PASSED / FAILED / SKIPPED per rule — not just the first failure)
                                    │
                     MATCHED_READY / MATCHED_VARIANCE
                                    │
                  analyst clicks "Run Reconciliation" ──► POST /reconcile/{invoice_id}
                  same AE-06 semantics as the automatic pipeline's draft_node:
                  submits a PENDING_APPROVAL draft, does NOT itself call the OIC trigger
                                    │
                       draft appears in the existing Approval Queue,
                       same POST /drafts/{invoice_id}/approve as the automatic path

Contracts and receipts extract immediately on upload (the analyst needs them registered before mapping an invoice to one); invoices extract only when a match is run, mirroring the automatic pipeline's own extraction timing. A PO does not need to be pre-registered anywhere for a receipt to be recorded against it, or for a match to succeed — the invoice's own extracted PO number is used purely as a lookup key into AP_GOODS_RECEIPTS at match time.

New agent modules (mirror extraction_agent.py's structure exactly — direct tools.dus2_tool.extract_key_value call, then a single structured-output LLM normalization, honest missing_fields rather than a best-effort guess):

Module	Extracts	Registers into
`contract_agent.py`	MSA/agreement number, supplier name, one or more priced line items	`AP_CONTRACTS` (upsert by MSA number) + `AP_SUPPLIERS` (upsert by name) + `AP_CONTRACT_LINE_ITEMS`
`receipt_agent.py`	Receipt/GRN number, PO number, received quantity	`AP_GOODS_RECEIPTS` (upsert by its own `RECEIPT_ID` — not by PO number; several receipts can exist for the same PO, e.g. partial shipments)

Both routes reject registration outright (422, before touching the database) when extraction leaves any required field in missing_fields — a document that fails to extract an MSA number cannot be upserted as a real contract row under a placeholder value; the analyst re-uploads a clearer scan instead.

matching_agent.evaluate_match_detailed() evaluates all seven rules independently rather than stopping at the first failure, so the Workflow tab can show an analyst the complete picture. evaluate_match() — the automatic pipeline's classifier — now derives its single first-failure MatchResult from this same breakdown, so both views are guaranteed to agree; the 14 existing unit tests in tests/unit/test_matching_agent.py cover evaluate_match()'s output unchanged (updated for the MSA model, not just carried over).

New API routes (reconciliation_api.py): POST /invoices/upload, /contracts/upload, /receipts/upload (multipart), GET /invoices, /contracts, /receipts, DELETE /invoices/{object_name}, /contracts/{object_name}, /receipts/{object_name}, POST /match, POST /reconcile/{invoice_id}. All proxied through the gateway (server/index.js) via multer (memory storage — documents never touch the gateway's own filesystem) and gated behind the AP Analyst IDCS group, same as every other Workflow-tab action. Deleting a contract also un-registers its AP_CONTRACTS/AP_CONTRACT_LINE_ITEMS rows (blocked with a clear 409 if a PO already references that MSA — delete the PO first); deleting a receipt removes only that one AP_GOODS_RECEIPTS row, leaving any other receipts against the same PO untouched.

Binary document uploads bypass mcp_os entirely — see Operational Notes below.

Why EBS is "shielded," not directly integrated¶

This platform's own architecture treats EBS as reachable only through Oracle Integration Cloud workflows (docs/capabilities.md's EBS shielded via OIC rule) — agents never call EBS APIs directly. No OIC integration endpoint exists yet for this recipe, so oic_client.py names the tool boundary a real integration will eventually fill (lookup_supplier_contract_receipt, create_reconciliation_draft), implemented today as real reads/writes against this recipe's own AP_* stand-in tables rather than a hardcoded Python dict — swapping in the real OIC call later is a body-only change to two functions, nothing downstream.

There is deliberately no post/release function anywhere in this module's interface — that capability doesn't exist in this recipe, so there's nothing to accidentally wire up early.

Deterministic Matching Engine¶

matching_agent.evaluate_match() — plain Python, zero LLM calls, evaluates seven rules in order and stops at the first failure:

#	Check	Failure →
1	Supplier match	`SUPPLIER_MATCH`
2	MSA validity — the invoice's governing contract exists, is open, belongs to the supplier	`MSA_VALIDITY`
3	Duplicate invoice (supplier + invoice number + amount)	`DUPLICATE_INVOICE`
4	Required goods receipt on file (summed across every receipt found for the PO)	`REQUIRED_RECEIPT`
5	Billed quantity ≤ total received quantity	`QUANTITY_VARIANCE`
6	Each invoice line's price within tolerance of its matching MSA line-item price	`PRICE_VARIANCE`
7	Invoice total reconciles with line items + tax	`TOTAL_VARIANCE`

MSA_VALIDITY (renamed from PO_CONTRACT_VALIDITY) reflects the MSA-based contract model: a PO is a lightweight AP_PURCHASE_ORDERS → AP_CONTRACTS association with no pricing of its own, not the pricing authority — see Manual workflow above and db/ddl/06_msa_contracts_and_multi_receipts.sql. For the automatic pipeline, a closed PO still fails this check even if its MSA remains open — oic_client._lookup_async only resolves the governing MSA for a PO whose own status is OPEN.

This is also the actual AE-10 prompt-injection defense, not a separate mechanism: the classifier has no code path that reads a line item's free-text description at all, so there is structurally nothing for injected text in an invoice document to influence. The one agent that does see raw document text — exception_agent, drafting the AP Analyst's explanation — only ever explains a decision matching_agent already made; it never decides anything.

An invoice above a configurable high-value threshold, or from a supplier with no prior transaction history, is flagged requires_second_review = true regardless of the rule outcome — a clean match on an unusually large or first-time-supplier invoice still gets a second set of eyes.

Extraction pipeline¶

extraction_agent.py calls tools.dus2_tool.extract_key_value directly, never through a model tool-call loop — whether to run extraction is never a judgment call, so the model never gets the chance to narrate a plausible-sounding result instead of actually calling the tool. A single structured-output LLM call then normalizes DUS2.0's raw output onto the recipe's Invoice schema, honestly reporting which required fields it couldn't confidently determine (missing_fields) rather than guessing.

OCI Document Understanding's output is word-level OCR, not key-value pairs

DocumentKeyValueExtractionFeature() with no custom model returns pages[].words[].text — full-page OCR with per-word bounding-polygon JSON. A single-page invoice's raw payload can run ~28,500 characters, most of it coordinate data the normalization step doesn't need. Collapse it to a plain reading-order text transcript before handing it to the normalization LLM — this is both more token-efficient and avoids truncating away the line-items/totals section, which typically appears later on the page than the header fields.

Tool Governance¶

Tool	Scope	Failure behavior
`tools.dus2_tool.extract_key_value`	Read-only extraction against the configured input bucket	Bounded retries (`DUS2_JOB_MAX_RETRIES`); terminal failure → `EXTRACTION_FAILED` dead-letter state, distinct from `MISSING_DATA`
`mcp_os` (`list_objects`)	The invoice inbox bucket only	Poll failure → logged, retried next interval; never treated as "no new invoices"
`oic_client.py`	`lookup_supplier_contract_receipt` (read) + `create_reconciliation_draft` (write, called only from the Finance-approval route)	Idempotent on `INVOICE_ID` — a retried approval call is a no-op, not a second trigger
`mcp_sqlcl`	This recipe's own `AP_*` schema only — never EBS's own tables	DB failure → the invoice stays `PENDING_REVIEW`, never marked processed

Database¶

AP_* tables live in a shared Oracle ADB schema (SELECT_AI_USER) rather than a dedicated schema of their own — the AP_ prefix on every table name is what avoids naming collisions with other applications in that same schema, not a separate Oracle user per application. AP_INVOICES doubles as the duplicate-detection ledger (no separate dedup table). See db/README.md for the full DDL and the seeded stand-in supplier/PO/receipt data covering every branch of the matching-rules table.

Evaluation (AE-09)¶

Four case files wired to metro.evals.evalFactory (see Continuous Evaluation) — deliberately scoped to what this recipe's tool-free classification path can actually exercise:

File	Layers	Why not others
`cases/matching_rules.yaml`	`assertion`	`matching_agent` is pure Python — free, zero-flake
`cases/extraction.yaml`	`assertion`	Tests the normalization LLM logic against synthetic raw payloads, no live Document Understanding job needed per case
`cases/exception_explanations.yaml`	`assertion` (`reply_contains` on the real cited numbers), `rubric`	No `groundedness` layer — `exception_agent` makes a plain LLM call, not a tool call, so there's no `tool_results` trace for that scorer to check against; `reply_contains` on the real numbers is the stronger, cheaper substitute
`cases/adversarial.yaml`	`assertion`	No `trajectory` layer — there is no tool-calling agent anywhere in the classification path to begin with, which is itself the AE-10 defense (see above)

Operational Notes¶

Integration details worth knowing before extending this recipe:

mcp_os's list_objects response shape. The tool wraps its JSON in a {"result": "<json string>"} envelope, and the object list key is capitalized "Objects". Parse accordingly — a flat-shape assumption on this response reads as zero objects with no error.
mcp_os's upload tools don't fit binary documents. upload_object_remote UTF-8-encodes object_content, which corrupts PDF/PNG bytes; upload_object_file expects a file already on the MCP server's own filesystem, not a remote browser upload. The Workflow tab's document uploads go through src/agents/utils/object_storage.py instead — a direct oci.object_storage.ObjectStorageClient call, same static-config auth pattern as tools/dus2_tool.py. mcp_os is still used for listing, which has no binary-content constraint.
tools.dus2_tool.extract_key_value is scoped to one bucket per process by default. INPUT_BUCKET is read once from the environment and cached; the function now accepts an optional bucket_name argument that overrides it for a single call, building its own ObjectLocation rather than mutating the shared module-level one (needed once this recipe started extracting from three separate inbox buckets instead of one — see contract_agent.py/receipt_agent.py).
DocumentKeyValueExtractionFeature only classifies receipt/invoice/passport/driver-license documents. A generic business document — a purchase-order contract, a warehouse goods-receipt confirmation — can come back FEATURE_NOT_SUPPORTED with every field null, because OCI's built-in document-type classifier doesn't recognize it as one of those four types. Nothing in this recipe ever reads OCI's own structured key-value output anyway (every consumer collapses the response to a word-level OCR transcript and normalizes it with its own LLM call), so tools/dus2_tool.py now requests DocumentTextExtractionFeature instead — OCI's type-agnostic OCR feature, same pages[].words[] output shape, no document-type gate.
A DUS2.0 page with no detected text returns "words": null, not an absent key. page.get("words", []) doesn't catch an explicit None value — use page.get("words") or [].
AP_INVOICES.INVOICE_NUMBER must never receive an empty string. Oracle treats '' as NULL for VARCHAR2 columns, which violates the column's NOT NULL constraint. When extraction can't determine the invoice number, persist a placeholder such as "UNKNOWN" — the same convention already used for the vendor-name field — and rely on missing_fields/the classification outcome to signal the gap, not the placeholder value itself.
mcp_sqlcl renders Oracle DATE columns in the session's display format, not ISO. This ADB's default NLS_DATE_FORMAT returns '20-JUL-26', not '2026-07-20' — date.fromisoformat() raises on it. draft_agent.py's _parse_oracle_date() tries ISO first, then Oracle's DD-MON-RR.
AP_INVOICES.SUPPLIER_ID is only meaningful once something writes it. The column is documented as "resolved by the lookup step; NULL until matched," but the automatic pipeline never actually persisted it — draft_node just keeps lookup.supplier_id in memory for its one process run. The manual workflow's /reconcile route is a separate HTTP request that has to re-read it from the database, so matching_agent.record_match_result() now accepts an optional supplier_id and writes it onto the AP_INVOICES row alongside STATUS — both the automatic and manual paths pass it now.
AP_AUDIT_LOG is a separate write from audit_log(). metro.logging.agentic_logging.audit_log() writes to the structured logger only; the durable compliance table is a distinct database insert (src/agents/utils/audit.py's write_audit_log()), called alongside audit_log() at every workflow transition, not instead of it.
A contract's own agreement number is its real identity, not the PO it happens to mention. A real sample MSA read "Agreement No. MSA-2026-017 ... Associated purchase order: PO-45001982" — the first pass at this recipe extracted the PO number as the contract's primary key, which broke the moment a second PO needed to reference the same agreement. contract_agent.py's normalization prompt now explicitly tells the model to extract the agreement/MSA number and ignore any "associated purchase order" text entirely.
OCI's delete_object is not idempotent on its own. It raises a 404 ObjectNotFound ServiceError for a missing object rather than silently succeeding. src/agents/utils/object_storage.py's delete_document() catches that specific case and treats it as already-deleted, so a retried or duplicate delete click is a no-op, not an error surfaced to the analyst.
A schema-scoped Oracle error and a data-integrity error can look identical from the outside (RuntimeError from mcp_execute_sql) but need very different fixes. ORA-00942 (table/view not found) means the mcp_sqlcl session's schema switch didn't apply — retry, it's transient infra flakiness (see mcp_sqlcl note above). ORA-02291/ORA-02292 (integrity constraint violated) means the referenced row genuinely doesn't exist — check the constraint name in the error text (it includes the owning schema, e.g. SELECT_AI_USER.FK_AP_GOODS_RECEIPTS_PO) before assuming it's the former.

Both outcome paths — a clean match (MATCHED_READY, draft created, Finance approval) and an exception (EXCEPTION / QUANTITY_VARIANCE, a grounded explanation citing the actual billed/received quantities) — are exercised against real OCI Document Understanding, real OCI GenAI, and the shared Oracle ADB, not mocks.

Running locally¶

# 1 — SQLcl MCP server (shared with other recipes)
cd inception_mcp_servers/mcp_sqlcl
.venv_mcp_sqlcl/bin/python -m src.server_token

# 2 — Object Storage MCP server (inbox polling)
cd inception_mcp_servers/mcp_os
.venv_mcp_os/bin/python -m src.server_token

# 3 — Reconciliation API (Exception/Approval queues)
cd inception_recipes/ebs_invoice_reconciliation/ebs_invoice_reconciliation_agents
uv run python -m src.api.reconciliation_api

# 4 — Inbox poller (background job)
uv run python -m src.cli.ebs_invoice_reconciliation

# Or process one object without waiting for a poll cycle:
uv run python -m src.cli.ebs_invoice_reconciliation --once <object_name>

uv run pytest tests/unit    # matching_agent — pure, offline, 14 cases
uv run pytest tests/evals   # AE-09 CI gate — hits live OCI GenAI

What's deliberately not built yet¶

No real OIC integration — the write boundary is stubbed by design (see above). No event-driven invoice detection — polling mcp_os.list_objects, since mcp_os exposes no push mechanism. See specs/SPEC.md's Non-Goals for the complete list.

Console¶

ebs_invoice_reconciliation_ui/ — a single React + Vite console (not a split site; every actor is internal enterprise staff with an IDCS identity) with a Node/Express gateway that proxies every call with the signed-in user's own bearer token. Three tabs, each gated by IDCS group membership — a user in both groups sees all three:

Workflow (AP Analyst) — upload invoices/contracts/receipts, map an invoice to a contract, run the match, review the full per-rule breakdown, and submit for Finance approval. See above.
Exception Queue (AP Analyst) — invoices the automatic pipeline routed here for failing a rule or having insufficient extracted data.
Approval Queue (Finance Approver) — matched drafts awaiting sign-off, fed by both the automatic pipeline and the Workflow tab's "Run Reconciliation" action.

Segregation of duties (AE-12) is an IDCS group-membership control, not something enforced in the UI's own code.

Full Architecture Reference¶

inception_recipes/ebs_invoice_reconciliation/specs/SPEC.md            ← AE-01..AE-15 specification
inception_recipes/ebs_invoice_reconciliation/specs/USE_CASE_INTAKE.md ← the filled-in intake form it was built from
inception_recipes/ebs_invoice_reconciliation/docs/architecture.md     ← node-by-node pipeline detail