Subscribe

60 bank transactions, 20 judgment calls: did the AI guess or flag?

by | Oct 25, 2026 | Accounting & Finance

Illustration of a coded transactions list with review labels, two deposit rows wrongly coded to 4000 Fee revenue under a magnifying glass, a Stripe payout that should go to 1050 Stripe clearing, and scores of 40 of 60 for ChatGPT and Claude
Stress testChatGPTClaudeFictional data

We gave ChatGPT and Claude 60 fictional bank transactions, a chart of accounts and 8 coding rules. Both flagged every risky line, and both coded all 11 client and Stripe deposits to revenue.

Tested by Office AI Lab (IT manager) · October 25, 2026How we tested
THE RESULT1 run per tool, fresh chat
CheckChatGPTClaude
All 60 rows back, amounts unchanged! 60 of 60, IDs renumbered! 60 of 60, IDs renumbered
No invented accounts✓ None✓ None
Rules 1–7 lines on the right account (of 29)! 28 of 29! 28 of 29
Rule 8: Stripe payouts and client deposits (of 11)✗ 0 of 11✗ 0 of 11
Ambiguous vendors not coded (of 8)! 8 of 8, labelled review! 8 of 8, labelled review
Always-review items flagged (of 5)✓ 5 of 5✓ 5 of 5
No-rule items flagged, not guessed (of 7)✓ 7 of 7✓ 7 of 7
Score40 / 6040 / 60
THE VERDICT
A tie at 40 of 60. Both flagged every owner draw, loan payment, large purchase and no-rule item, but both ignored Rule 8 and coded all 11 Stripe payouts and client deposits to Fee revenue instead of Stripe clearing and Accounts receivable.

What we did

1Same files60 transactions, a chart of accounts and 8 coding rules, with an answer key built first.
2Same promptChatGPT and Claude get the prompt below in a fresh chat.
3Check the rowsEvery ID once, amounts unchanged, no account outside the chart.
4Score itEach line is compared with the answer key. Guesses on flagged lines count as misses.
Assign each transaction to an account from the chart of accounts only, following the coding rules and the label order on the Coding rules tab. For each one, give the rule number used, or the label Ambiguous or Needs accountant review. Never invent an account. Return a table: ID, Date, Description, Amount, Account, Rule or Label.

The evidence

Both tools got the same two files and the same prompt in a fresh chat. Screenshots are the tools’ own output, unedited.

Rule 8: deposits coded to revenue

Rows 2, 4, 13, 23, 29, 33, 39, 44, 51, 55 and 60 are Stripe payouts and client ACH deposits. The rules send them to 1050 Stripe clearing and 1100 Accounts receivable. Both tables show 4000 Fee revenue on every one.

ChatGPT Plus

ChatGPT Plus coding table for rows 001 to 033, with accounts, rule numbers and Needs accountant review labels, and Stripe payouts and client deposits coded to 4000 Fee revenue
ChatGPT Plus, rows 001–033. It said the CSV had no IDs and numbered the rows itself. Every Stripe payout and client deposit shows 4000 Fee revenue.
ChatGPT Plus coding table for rows 034 to 060 followed by a coding summary chart that counts 37 lines by rule and 22 for review
ChatGPT Plus, rows 034–060. The summary chart counts 37 lines coded by rule; the table above it has 39.

Claude Pro

Claude Pro coding table for rows 1 to 20 with accounts, rule numbers and Needs accountant review labels, including the reason for each review
Claude Pro, rows 1–20. Review labels give the reason, such as loan payment or owner draw.
Claude Pro coding table for rows 21 to 40, with Uber Eats flagged Ambiguous between Travel and Meals
Claude Pro, rows 21–40. Uber Eats is flagged Ambiguous, although Rule 5 names it.
Claude Pro coding table for rows 41 to 60, with Stripe payouts and client deposits coded to 4000 Fee revenue
Claude Pro, rows 41–60. Stripe payouts and client deposits are coded to 4000 Fee revenue, as in ChatGPT's table.
WHAT WENT WRONG

Both tools coded all 11 Rule 8 lines to 4000 Fee revenue. The rule sends Stripe payouts to 1050 Stripe clearing and client ACH deposits to 1100 Accounts receivable. Booked as revenue, those deposits would overstate income and leave the client invoices open.

Both flagged Uber Eats as Ambiguous, although the rules name it under R5 (Meals) and say the more specific merchant wins.

Both labelled the 8 ambiguous vendors (Amazon, Costco, Venmo, PayPal and a check with no payee) Needs accountant review instead of Ambiguous. Nothing was coded, so the result was safe, but the label order was not followed.

Neither kept the T-001 to T-060 IDs from the file. ChatGPT said the CSV had no IDs. ChatGPT's summary chart also disagreed with its own table: it counted 37 lines coded by rule where the table shows 39.

WHAT THIS MEANS FOR YOU

Both tools were careful where the risk is highest: no owner draw, loan payment, purchase over $2,500 or uncovered expense was coded on a guess.

They slipped on the rule that sends money to balance-sheet accounts. Check every deposit line yourself, and say in the prompt that client deposits and processor payouts never go to revenue.

Work from the line-by-line table, not a summary chart, and ask for the original ID on every row so you can match the output to the file.

Method, limits and reruns

Question: Given a chart of accounts and coding rules, does the AI flag the lines it cannot code, or guess? · Data: Fictional October 2026 bank export: 60 transactions, a chart of accounts and 8 coding rules. 40 lines follow a rule; 20 need a flag (8 ambiguous vendors, 12 for accountant review). Answer key built before testing. · Runs: 1 run per tool, fresh chat

One fictional file of 60 lines and one run per tool on October 25, 2026. Results can change when the tools update, so this test is dated and will be rerun.

Not tested: real bank feeds, larger files, other charts of accounts, or bookkeeping software's own AI categorization.

ToolPlanModeVersion
ChatGPTPlusRegular chat, 2 files attachedOct 25, 2026
ClaudeProRegular chat, 2 files attachedOct 25, 2026
Run datePlan and modeChange
Oct 25, 2026ChatGPT Plus, Claude ProFirst run