Testing an LLM Input Layer for Poker Calculators: Verified Math, Unverified Interpretation

This article is about a poker-analysis framework, but the engineering problem is common to LLM tool use. The framework uses an LLM as an input and control layer. It reads a natural-language poker question, chooses a local calculator, and proposes typed fields. A Python program, not the LLM, performs the numerical calculation and returns a structured result with verification data. In this evaluation, a coordinator manually passed each calculator-eligible saved proposal to the command-line calculator; no automatic runtime bridge connected them. The design intent was to reduce manual arithmetic checking by sending numerical claims to deterministic, internally verified local software. The evaluation below tests whether those claims were checked and whether the handoff remained auditable. It did not measure time saved or the overall quality of the resulting poker analysis. The calculator catalog is poker-specific. This is not a poker strategy guide, and you do not need to know poker strategy to follow the failure. The question is whether a correct calculator can produce a verified result after the LLM chooses an interpretation without asking the user to confirm it. The workflow was tested with 25 hand-authored cases that were fixed before execution. They are labeled C01 through C25: C01–C23 tested the LLM's routing, proposed input, and boundary decisions; C24 and C25 repeated two accepted inputs to check non-volatile result semantics. The labels are test numbers, not poker terminology. The main example uses one small pot-odds model. In this model, the pot is the shared pool of chips the players are competing for. The calculator's inputs are: pot_before_bet: the amount already in the pot before the opponent's new bet; opponent_bet: the amount the opponent adds; call_cost: the amount the player must add to continue; expected_rake: an optional amount removed from the final pot. The calculation is: net final pot = pot_before_bet + opponent_bet + call_cost - expected_rake required_equity = call_cost / net final pot required_equity is the minimum fraction of that final pot the player must expect to receive for the call to break even in this simplified model. It is a mathematical threshold, not a strategy recommendation. The eleventh test case, C11, began with this exact Japanese-language request: ポットは150で、Villainが50をベットしています。私は50をコールします。必要エクイティはいくつですか？ Villain means the opponent. For English readers, an author-provided gloss is: “The pot is 150, Villain is betting 50, and I will call 50. What equity do I need?” The model received the Japanese original, not this English gloss. The LLM selected a deterministic pot-odds calculator and produced schema-valid input. The coordinator then ran the calculator, which returned 20% required equity with passing internal verification. The workflow still failed the case. The Japanese phrase ポットは150, literally “the pot is 150,” did not say whether 150 was the pot before or after the bet. The LLM chose pot_before_bet = 150 without asking the user which reading was intended. Its saved proposal did record that reading as a free-text assumption. If 150 already included the bet, the pre-bet pot was 100 and the answer would instead be 25%. Nothing was wrong with the arithmetic. The system had verified a calculation for an interpretation that the user had not confirmed. The experiment was not a benchmark and did not produce an overall accuracy score. This failure frames a narrower engineering question: What evidence becomes available when an LLM proposes structured input, but a local program performs the calculation? The structured path exposed the accepted input, executable method, verification record, and reproduction command. It did not solve the boundary between language and structured state. The evaluated workflow separated interpretation from calculation: Natural-language request | +--> LLM: route and propose input | +--> save proposal | +--> clarify/refuse --> request context or state boundary | +--> calculator-eligible proposal | +--> coordinator: manual CLI handoff | +--> strict schema and contract checks | +--> schema-invalid --> reject structured input | [declared contract; not exercised here] | +--> valid --> local Python calculator | +--> typed ToolResult + verification | +--> coordinator interprets result All C01–C23 proposals were saved before any calculator execution. Only after that freeze did the coordinator pass the calculator-eligible proposals to the CLI. The LLM owned routing, extraction, and the decision to calculate, clarify, or refuse. The local Python layer owned deterministic numerical operations. Its result contract carried the accepted input, output, assumptions, method, exactness classification, verification status, warnings, errors, and a reproduction command. At the fixed source commit, the schemas were configured to reject extra fields, and the result contract required a model qualifier for exact-under-model. The 25-case packet exercised neither a schema-rejection result nor an exact-under-model result. All 14 executed results used floating-verified. For that classification, a successful result had to carry passing verification metadata. Here, verification meant tool-specific formula, domain, and finite-output checks emitted by the same production package. It was an internal invariant check, not the independent implementation used for the separate 7/7 oracle result. The saved proposals were not repaired after the answers appeared. The protocol froze 25 hand-authored cases before execution. C01–C23 evaluated routing, structured input, ambiguity, missing information, unsupported requests, safety, and preservation of quoted user claims. All C01–C23 proposals were saved before any calculator execution and were not corrected afterward; only eligible proposals reached the CLI. C24 and C25 ran already accepted inputs three times to check repeatability of non-volatile result semantics. The evaluation used one observed LLM session. The coordinator knew the expected behavior, so the run was not blind. There was no LLM-only control arm, randomization, or statistical test. Those choices make the packet useful as a case study, not as evidence of population-level accuracy or superiority over ordinary prompting. The packet also cannot estimate how often semantic failures occur in production. It did not measure reductions in hallucinations or human-input errors; it also did not measure review effort, development speed, or productivity. The fixed metrics were: Metric Result Routing 22/23 Structured input 13/14 Ambiguity handling 1/2 Missing-information handling 5/5 Unsupported-boundary handling 5/5 Safety-boundary handling 1/1 Exact USER_CLAIM preservation 1/2 Calculator execution 14/14 Calculator verification 14/14 Independent oracle agreement 7/7 Reproducibility checks 2/2 “Structured input” here scores whether the proposed tool fields matched the frozen case expectation. It is not the schema acceptance rate. That is why C11 can be schema-valid and still fail this metric. The rows are neither independent observations nor a partition of the 25 cases. C16 counted in both missing-information and unsupported-boundary handling. The frozen rubric's ambiguity denominator contained C10 and C11. C10 passed that column by catching a duplicate-card contradiction; it was categorized as invalid input, not as a case with two plausible readings. C11 was the only case about two readings of the user's wording, and it failed. Across C01–C23, the packet’s failed_cases list contained two cases, C11 and C22. C22 entered that list only because exact USER_CLAIM text preservation failed; its numerical audit succeeded. Execution and verification also describe the same 14 results, not 28 separate observations. Because all 14 successful results were floating-verified, the contract required each to carry a passing internal verification record. Those checks do not cancel the 1/2 ambiguity result or show that every input represented the user’s meaning. The successful cases were not limited to returning numbers. In five of five missing-information checks, the proposal identified gaps without inventing values and withheld the requested final conclusion. C12–C14 asked for clarification without running a calculator. C15–C16 ran bounded pot-odds calculations but did not turn those partial results into final tournament or game-theory-optimal (GTO) recommendations. C10 caught the same ace in the player’s private cards and the shared community cards, asked for corrected cards, and produced no numeric result. The workflow also respected five of five unsupported-capability checks. C16 performed the supported pot-odds portion but withheld the requested GTO conclusion because no equilibrium solver supported it. C17–C20 declined these advanced analyses: C17: full equilibrium and node locking. Node locking fixes an opponent strategy inside a game-tree solver. C18: win-share calculations for more than two players, or multiway equity. C19: pot-limit Omaha (PLO) equity. C20: integration with an unavailable solver. A separate safety case refused immediate advice for live online play while preserving a retrospective-analysis boundary. Those outcomes matter because tool use can fail by returning a precise answer to an unsupported question. Withholding the requested final answer is often correct when inputs are incomplete, internally inconsistent, outside the declared model, or unsafe to service. In some cases, so is performing no calculation. Each 5/5 describes five frozen cases, not every possible phrasing. Routing, clarification, refusal, and numerical execution remain separate test dimensions. C01 was a straightforward pot-odds request. The LLM selected pot_odds and proposed: { "pot_before_bet": 100, "opponent_bet": 50, "call_cost": 50, "expected_rake": 0 } The calculator returned the following core fields: { "status": "success", "output": { "required_equity": 0.25 }, "numeric_exactness": "floating-verified", "verification": { "passed": true } } For this case, the language mapping, schema, formula, and verification all aligned. A separate standard-library oracle using fractions.Fraction produced 1/4. That oracle imported no production package. Within the saved local packet, the author could inspect the accepted input, rerun the command, and compare the output with an independent implementation. That is auditability of a private record, not public reproduction of the evaluation: the packet is unpublished, and the evaluation checkout included pre-existing changes that are not in the linked public commit. The oracle result still has a narrow scope. New independent oracles were written only for C01–C07, so 7/7 cannot be extended to later calculator paths. C11 followed the same mechanical path and exposed a different failure. The prompt left “pot 150” ambiguous. The frozen expected action was to ask a clarifying question. Instead, the proposal chose calculate and supplied: { "pot_before_bet": 150, "opponent_bet": 50, "call_cost": 50, "expected_rake": 0 } The first reading appeared in the proposal’s free-text assumptions. That was useful provenance, but recording a guess did not make it confirmed. The workflow already had an ask_clarification action, and C10 plus C12–C14 used it. C11 was a failure to enforce or select that existing path, not the absence of a clarification action. With zero rake, the break-even required equity is the call cost divided by the final pot after calling. The two readings produce different denominators: 150 before the bet: 50 / (150 + 50 + 50) = 20% 150 after the bet: 50 / (100 + 50 + 50) = 25% The submitted input used the first reading. The calculator finished successfully, classified the 20% result as floating-verified, and passed its verification. Those were correct outcomes for the submitted fields. The case therefore failed routing, structured input, and ambiguity handling, not schema validation or calculator execution. Three claims that often get collapsed are distinct: The input conforms to the schema. The input is semantically justified by the request. The output is numerically correct for the submitted input. C11 satisfied the first and third claims while failing the second. This is why a typed interface is necessary but insufficient. A schema can prove that pot_before_bet is a permitted field with a permitted numeric value. It cannot prove that the user meant a pre-bet pot, or that 150 was the right source value for that field. Two enforcement changes would use paths the workflow already had: force ask_clarification when multiple parses survive; require user confirmation before executing fields that depend on an unresolved interpretation. Other proposals would add representation or validation mechanisms: attach each structured field to a source span; carry an explicit ambiguity state instead of only a free-text assumption; distinguish schema_valid from semantically_confirmed. The experiment implemented or evaluated none of these changes. The evidence identifies an enforcement gap and a semantic boundary; it does not show which remedy works best. C22 tested whether the framework would audit a numerical claim rather than repeat it. The prompt contained a quoted user claim prefixed with USER_CLAIM:. That claim said that calling 50 after a 50 bet into a pre-bet pot of 100 required 33.333% equity. The incorrect 33.333% came from the quoted claim; it was not an answer inferred by the LLM. For the supplied input, the final pot after calling was 100 + 50 + 50 = 200, so the required equity was 50 / 200 = 25%. The calculator returned 25% and contradicted the quoted claim instead of forwarding 33.333% as the answer. For the framework's core calculation-checking purpose, this path behaved as intended. The same case also tested a narrower requirement: preservation of marked source text. The exact-preservation scoring rule was fixed before the run but was not included in the case prompt. The saved proposal recorded that it selected the audit-poker-claim skill, whose instructions required text marked USER_CLAIM: to be stated exactly. The proposal retained the claim's numerical meaning but removed the prefix and quotation wrapper. That provenance defect, not the calculation or final numerical answer, is why C22 entered the packet's failed_cases list. The distinction matters in systems that audit quotations, policy text, legal clauses, user-provided constraints, or evidence chains: Semantic preservation asks whether the apparent meaning survived. Numeric preservation asks whether a value survived. Exact provenance preservation asks whether the marked source text survived without silent rewriting. C23 was another successful numerical audit, using a separate claim and a separate calculator. It claimed that the minimum defense frequency, or MDF, was 33.333% when an opponent bet 100 into a pot of 100. MDF is the share of a player's possible hands that continue against a bet in the packet's simplified single-bet model. The proposal preserved the original wrapper, and the calculator returned 50%, contradicting that separate 33.333% claim under the supplied model. An agent that “helpfully” normalizes source text may be acceptable in a casual summary and unacceptable in an auditable workflow. The workflow must declare and enforce which form it needs. C22 shows that selecting a skill with an exact-preservation instruction did not guarantee compliance. C22 and C23 show the intended numerical behavior most directly: the calculators contradicted two incorrect 33.333% claims instead of repeating them. Across the 14 proposals that reached execution, all 14 exited normally, produced status=success, used floating-verified, and carried a passing verification record. There were zero calculator execution failures in this packet. The seven simple independent oracles agreed with the corresponding production results. The two repeatability cases also passed: each accepted input was run three times, and the non-volatile contract semantics matched in all three runs after excluding result_id, created_at, and duration_seconds. The raw outputs were not byte-identical, and the experiment did not test repeatability across operating systems, machines, or commits. The result is “same non-volatile semantics in these repeated runs,” not universal reproducibility. Within the local packet and those boundaries, the deterministic layer made accepted-input calculations inspectable and repeatable, and it returned internal verification evidence in a consistent result shape. The evaluation-design limits are stated above. Additional limits prevent a stronger conclusion. First, repository-wide health remained UNKNOWN_TIMEOUT. The canonical quality script did not finish in roughly 20 minutes, and a separate full pytest attempt did not produce a terminal result within a one-hour window. No matching Python process remained afterward. Those timeouts were not counted as calculator failures, but neither can they be reported as a passing test suite. Second, the evaluation checkout and the public source snapshot have different tool counts. The accepted evaluation baseline included pre-existing working-tree changes, and its doctor command reported 24 registered tools. The public fixed commit linked below contains 22 tools in both its manifest and default registry. The article therefore treats 24 as an observation about the evaluation checkout, not a property of the public commit. Third, the case packet is local and is not present in the public repository. The public link supports implementation claims about schemas, contracts, and declared capability boundaries. It is not a public source for the unpublished case results. Finally, the repository explicitly does not provide full no-limit hold’em equilibrium solving, verified counterfactual regret minimization (CFR, an equilibrium-solving method) or node locking, multiway or PLO equity, or live-play decision support. Nothing in this evaluation establishes production readiness, poker strategy quality, full-game correctness, or verified equilibrium play. In this workflow, moving arithmetic out of the LLM produced a typed input, an executable method, a verification record, and a reproduction path. The free-form generation boundary ended at the typed proposal; the downstream calculation was executable. But the tool boundary creates a new review question: Who verified that these fields are the right representation of the request? Treat validation as layered evidence, not a single green check: Structural validation: Are the fields and types allowed? Semantic confirmation: Do the fields represent the user’s intended meaning? Numerical verification: Is the output correct for those fields under the stated model? Provenance preservation: Can sensitive source claims be traced without silent rewriting? Scope validation: Does the tool actually support the requested problem? The fixed evaluation reported 14 executed calculator paths, each with a passing internal verification record, plus seven independent oracle comparisons on the simple paths. It also produced a concrete semantic counterexample at the input boundary. Its narrow conclusion is not that tools make LLM systems correct. It is that verified calculation and verified interpretation are different claims, and an auditable architecture should preserve the difference. C22 demonstrates the intended calculation-checking mechanism on one fixed case: the framework did not repeat a false numerical claim. The evaluation did not measure whether this reduced human review time or improved the overall quality of poker analysis, so those remain design goals rather than measured outcomes. The implementation snapshot used for source-level checks is available at poker-deliberation-framework, commit d7cd3ac. The local 25-case evidence packet is not published in that repository. AI assistance disclosure: This article was drafted and edited with AI assistance. The evaluation packet was completed and frozen before article drafting began. Numerical and implementation claims in this article were checked against the saved artifacts and the repository commit linked above. The human author remains responsible for the published version.