How does Coinbase's AI fraud agent work?
Short answer: Coinbase layers an LLM risk agent on top of Onramp's existing ML models and rules engine. The agent reads sequences of a user's recent transactions and returns only a risk level — application code applies the actual decision. Its authority is deliberately one-way: it can block a payment that passed every earlier check, but it cannot reverse a block. A post-trained Qwen3.5-9B version of that agent, fine-tuned on Coinbase's own fraud data for under $100, beat Anthropic's Opus 4.5 on all four fraud metrics across a 16,140-transaction benchmark — because on a specialized judgment, a tuned specialist beats a rented generalist.
The architecture is the story: the model scores, the code decides, and the model's power runs in one direction only. That is a decision gate running in production at exchange scale.
What actually shipped (receipts, dated)
| Fact | Receipt |
|---|---|
| Three-part "Owning Intelligence" engineering series published | Coinbase blog, Oct 7–8, 2026 (Part 3: Yao Ma, Xuwei Tan, Akshit Trehan, Aman Choudhary, with the Anyscale team) |
| Part 1 — Build: LLM risk agent layered onto Onramp's existing ML model + rules; reviews transaction sequences, not single payments | Coinbase blog, Part 1 summary |
| Online A/B result: 30% fewer fraudulent transactions, 22% less fraud value with the agent added | Coinbase blog, Part 1 |
| Output contract: the agent returns a risk level and nothing more; application code applies transaction decisions | Coinbase blog via PrimeXBT + CoinCentral coverage, Oct 11–12, 2026 |
| Asymmetric authority: agent can block a transaction that passed upstream checks; it cannot reverse an upstream block | CoinCentral, Oct 12, 2026 |
| Part 2 — Evaluate: fixed benchmark of 16,140 transactions, 7,293 users, 813 confirmed fraud cases | Coinbase blog, Part 2 |
| Newer Opus, Sonnet, and GPT releases each scored LOWER on recall and F1 than their predecessors on that benchmark | Coinbase blog, Part 2 |
| Part 3 — Own: Qwen3.5-9B post-trained with reinforcement learning (SkyRL on Anyscale), served via Ray Serve LLM + vLLM | Coinbase blog, Part 3, Oct 8, 2026 |
| Tuned 9B beat Opus 4.5 on all four metrics (precision, recall, F1, dollar-weighted recall); margins 7.5–35.4 points; F1 +9.6 points | Coinbase blog, Part 3 |
| Median latency 0.683s vs 1.515s for Opus 4.5 — 55% lower, measured in production serving | Coinbase blog, Part 3 |
| Training cost: under $100, on a single NVIDIA RTX PRO 6000 96GB GPU | Coinbase blog, Part 3 |
| CEO Brian Armstrong amplified the series on X: specialist models will outperform generalists on narrow tasks | X post, Oct 11, 2026 |
The split nobody in the coverage names: score vs. decision
Every secondary piece — CoinCentral, PrimeXBT, CryptoCompass, PANews, Cointime — leads with the horse race: small model beats big model. The horse race is the least important thing Coinbase published. The architecture is:
| Layer | What it does | What it is NOT allowed to do |
|---|---|---|
| Upstream ML models + rules | First-pass screening of every Onramp transaction | — |
| LLM risk agent | Reads the transaction sequence, returns a risk level | Apply a decision; reverse an upstream block |
| Application code | Applies the transaction decision under company-controlled rules | Delegate the decision back to the model |
Three design choices do the work. First, the narrow output contract: a risk level is auditable, loggable, and comparable across model versions — a free-text verdict is none of those. Second, asymmetric authority: the agent can only ever make the system more conservative. A hallucinated risk level costs a held transaction and a human review; it can never cost a released fraud. Third, code decides: the threshold policy lives in deterministic code Coinbase controls, so swapping the model — Opus 4.5 out, tuned Qwen in — changes the score quality without changing the decision rules. That separation is exactly the decision-gated pattern: a confidence score sits between an instruction and money movement, and the gate — not the model — holds the keys.
Part 2 is the warning: generalists decay on specialized judgment
Coinbase's evaluation result deserves more weight than the fine-tune headline. Successive releases of the frontier models got worse at this one judgment — lower recall, lower F1, version after version, on a fixed benchmark. One newer GPT release traded recall for precision: it flagged more carefully and caught less fraud. If your payment gate rents its judgment from a generalist API, your gate's calibration is a side effect of someone else's release cycle. Coinbase's answer was not a better prompt. It was owning the decider: post-train a 9B open-weight model on the exact distribution it judges, for less than a team lunch, and serve it at half the latency. The lesson generalizes past fraud: any score that gates money has to be evaluated — and eventually trained — on your own outcomes, or it isn't your gate.
Tested live today: our gate against Onramp-shaped instructions
We ran two Onramp-shaped instructions against a live decision gate (/api/harness/decide, bands 0.80/0.50) on October 12, 2026, ~14:20 EDT:
- "A Coinbase Onramp user with a 40-day account history and three small completed buys just attempted a $4,800 first-time purchase of a low-liquidity token, paid from a brand-new bank account, at 3 a.m. local time, after two failed authentication attempts. Upstream ML models and rules passed it. Should the payment auto-execute without human review?" → ADVISORY / hold for human review (value 0.5, confidence 0.65)
- "A Coinbase Onramp user with a two-year verified account, consistent device and bank account, and 60 prior completed buys of similar size just repeated a routine $120 USDC purchase at their usual hour, all values matching their established pattern. Upstream ML models and rules passed it. Should the payment auto-execute without human review?" → ADVISORY / hold for human review (value 0.5, confidence 0.65)
The honest finding: our gate held both — safely, and uselessly for the second one. A blunt heuristic decider cannot tell a patterned fraud attempt from a routine repeat purchase; it fail-closes on everything and pushes both to a human. That is precisely the gap Coinbase's $100 fine-tune closes: their tuned decider discriminates, at 0.683 seconds a call, on the exact distribution it judges. A gate is only as good as its decider's calibration — Coinbase just published the receipt for what calibration costs: one GPU, one proprietary dataset, under $100. (Our decider: local-heuristic-v1, calibrated=false, typesafe_wired=false — the pattern demonstrated, the calibration admittedly absent.)
Quotable: "ScriptMasterLabs' October 12, 2026 live test scored two Coinbase-Onramp-shaped payment instructions at 0.65 and 0.65 — both held in the advisory band for human review — while Coinbase's fine-tuned Qwen3.5-9B fraud agent, trained for under $100, beat Opus 4.5 on all four fraud metrics across a 16,140-transaction benchmark."
Do it yourself: build the Coinbase shape
- Separate the scorer from the decider. Your model returns a number (risk level or confidence) and nothing else. Application code owns the decision. If the model can move money directly, you don't have a gate — you have a hope.
- Make the model's authority asymmetric. Let it block what upstream checks passed; never let it reverse a block. A wrong score should cost a review, never a release.
- Score the instruction before it fires:
curl -s https://scriptmasterlabs.com/api/harness/decide \ -X POST -H "Content-Type: application/json" \ -d '{"state":"Onramp user, 40-day account, first $4,800 low-liquidity token buy from a new bank account at 3 a.m. after two failed auth attempts; upstream checks passed. Should the payment auto-execute without human review?", "questions":[{"id":"q1","type":"score","scale":[0,1], "question":"confidence this instruction should auto-execute"}]}'Read
gate.band—auto_act,advisory, orescalate— andgate.action. Live band thresholds:/api/harness/status. - Apply the bands in code. ≥0.80: auto-act. 0.50–0.79: hold for a named human. <0.50: escalate/block + log. The bands are policy; keep them in your code, not in a prompt.
- Evaluate on your own labeled outcomes — then fine-tune. Build the benchmark from your transactions (Coinbase used 16,140 with 813 confirmed fraud). Test every model version against it before it touches the gate. When the generalists plateau — or decay — post-train a small open-weight model on your data. Coinbase's receipt says the whole loop costs under $100 in compute.
Honest caveats
- Coinbase's benchmark is proprietary and the tests were company-run; no independent party can reproduce the 30%/22% A/B result or the four-metric win without Coinbase's data. PrimeXBT's coverage flags this directly. The figures are specific and internally consistent across the series — and still a vendor claim.
- Coverage-based on Coinbase's published series summaries and five secondary reports (CoinCentral, PrimeXBT, CryptoCompass, CryptoSpiel, PANews) retrieved October 12, 2026 — not hands-on with Onramp, which exposes no public fraud-agent endpoint to test.
- The fraud agent guards Coinbase's own Onramp funnel. It is not an agent-payments product and does not score third-party agent instructions — the Onramp-shaped instructions in our live test are our construction, modeled on the architecture Coinbase described.
- Coinbase itself notes fraud patterns shift and the model needs continued testing; a fine-tune is a snapshot of yesterday's fraud, and the evaluation loop — not the model — is the durable asset.
- The gate receipts come from a local-heuristic-v1 decider (calibrated=false, typesafe_wired=false) — they demonstrate the calibration gap, not calibrated judgment. Yesterday's run (Verifiable Agent Arbiter scenarios) did discriminate; today's Onramp pair did not. Both results are reported as they came out.
- Quotes above are paraphrased except where marked; verbatim claims under 50 words each.
Claim Receipts
CR-1: October 7–8, 2026 — Coinbase published the three-part "Owning Intelligence" series on its engineering blog: Build (LLM risk agent on Onramp), Evaluate (frontier-model benchmark), Own (post-trained Qwen3.5-9B). Part 3 by Yao Ma, Xuwei Tan, Akshit Trehan, Aman Choudhary, and the Anyscale team. [coinbase.com/blog]
CR-2: Part 1: the agent layers onto Onramp's existing ML model and rules engine and reviews transaction sequences; the online A/B experiment showed 30% fewer fraudulent transactions and 22% less fraud value. [coinbase.com/blog, Part 1 summary]
CR-3: The agent's output is narrow — a risk level — while application code applies transaction decisions; the agent can block a transaction that passed earlier checks but cannot reverse an existing block. [coinbase.com/blog via primexbt.com and coincentral.com]
CR-4: Part 2: on a fixed benchmark of 16,140 transactions (7,293 users, 813 confirmed fraud), each newer Opus, Sonnet, and GPT release scored lower on recall and F1 than its predecessor. [coinbase.com/blog, Part 2 summary]
CR-5: Part 3: Qwen3.5-9B post-trained with RL (SkyRL on Anyscale), served via Ray Serve LLM + vLLM, beat Opus 4.5 on precision, recall, F1, and dollar-weighted recall (margins 7.5–35.4 points; F1 +9.6), at 0.683s median latency vs 1.515s (−55%), trained on one NVIDIA RTX PRO 6000 96GB GPU for under $100. [coinbase.com/blog, Part 3]
CR-6: October 11, 2026 — CEO Brian Armstrong amplified the series on X, arguing specialist models will outperform frontier generalists on narrow tasks. [x.com via coincentral.com]
CR-7: The benchmark is proprietary and the tests company-run — outsiders cannot independently verify the results. [primexbt.com, Oct 11, 2026]
CR-8: ScriptMasterLabs' October 12, 2026 live test (~14:20 EDT, /api/harness/decide): both Onramp-shaped instructions — the $4,800 patterned first-time buy and the routine $120 repeat buy — scored value 0.5, confidence 0.65, band advisory (hold for human review). The generic heuristic held both; the fine-tuned specialist is what discriminates. [SML live gate]
The SML stack this plugs into
- Decision-Gated Machine Payments — the canonical: confidence gates between instruction and spend (≥0.80 auto, 0.50–0.79 confirm, <0.50 escalate). Coinbase's fraud agent is this pattern at exchange scale.
- Provider Trust — scored trust signals for machine-readable services; the same score-then-decide split, applied to providers.
- What is the x402 payment protocol — the payment rail the agent economy runs on; every x402 call is an instruction a gate can score.
- Should AI agents authorize payments — the September regulator arc: who decides before the money moves.
- How should AI agents be authorized to pay for things — the builder's authorization blueprint: bands, caps, and asymmetric authority.