Migration decision report / October 13, 2026
Migration decision.
12 measured model configurations on reused public Banking77 cases. Validation evidence, not rollout approval.
01 / Customer
Banking77 Demo Bank
Illustrative name.02 / Identified workflow
Banking support intent classification
77 intents. Plain prompt, no retrieval.03 / Current model
o4-mini
Prior measurements retained unchanged.04 / Recommended model
GPT-6 Luna
Conditional. Recommended for workflow-specific validation.
Default weights: correctness 50% · consistency 20% · cost 20% · p95 time 10%.
Weighted preference score
43 corrections · 12 regressions · 31 fewer primary errors than o4-mini.
Challenge: 28/30 correct vs 25/30 incumbent.
GPT-6 Luna leads GLM-5.3-Flash by 0.39 preference points; this does not establish statistical superiority. Gemini 3.8 Flash has the fewest primary errors: 82/500.
How priorities change the choice.
Four illustrative scenarios set after collection. The observations, eligibility rules and published default recommendation stay unchanged.
| Priority | WeightsCorrectness / consistency / cost / p95 time | Preferred model | Versus default | Score / 100 |
|---|---|---|---|---|
| DefaultOriginal weights | 50% / 20% / 20% / 10% | GPT-6 Luna | Baseline | 82.45 |
| CorrectnessIllustrative scenario | 80% / 10% / 5% / 5% | Gemini 3.8 Flash | Changed choice | 81.24 |
| ConsistencyIllustrative scenario | 30% / 50% / 10% / 10% | Gemini 3.5 Flash Lite | Changed choice | 82.90 |
| CostIllustrative scenario | 30% / 10% / 50% / 10% | GLM-5.3-Flash | Changed choice | 87.59 |
| LatencyIllustrative scenario | 30% / 10% / 10% / 50% | Gemini 3.5 Flash Lite | Changed choice | 83.23 |
- Gemini 3.8 Flash: 3/500 strict-format violations; intent scoring still accepts valid fenced JSON. Validate the output contract before use.
- Gemini 3.5 Flash Lite: 132/500 strict-format violations; intent scoring still accepts valid fenced JSON. Validate the output contract before use.
Consistency means matching incumbent intent labels, including shared errors; the incumbent matches itself. The remaining weights still affect the choice. Scores compare models within each row, not across weight sets. These scenarios are not confidence estimates or new model evaluations.
05 / Other models evaluated
12 models. 500 primary cases each.
| Model | Consistency | Errors | Cost / 1,000 cases | Response timep50 / p95 HTTP time | Score / 100 |
|---|---|---|---|---|---|
| o4-miniCurrent model | 100.0%500/500 | 126/500 | $1.7612 | 2.07 s / 5.87 s | 72.40 |
| DeepSeek V4 FlashUnranked: Unresolved or missing outcomes; Incomplete charges | Unavailable437/500 confirmed; 1 unresolvedPossible range: 87.4% to 87.6% | 125/5001 invalid output1 additional outcome unresolved | Unknown499/500 cases fully priced | 5.65 s / 24.16 s | Unranked |
| Qwen3.7-FlashEvaluated alternative | 84.4%422/500 | 123/5001 invalid output | $0.0735 | 4.47 s / 14.57 s | 76.65 |
| Nemotron 3.5 LightningUnranked: Incomplete charges | 83.8%419/500 | 120/5001 invalid output; 2 provider failures | Unknown499/500 cases fully priced | 3.69 s / 13.23 s | Unranked |
| GLM-5.3-FlashEvaluated alternative | 84.4%422/500 | 111/500 | $0.0282 | 1.32 s / 3.03 s | 82.06 |
| Gemini 3.5 Flash LiteEvaluated alternative | 85.0%425/500 | 117/500 | $0.2319 | 0.65 s / 0.97 s | 81.56 |
| Gemini 3.8 FlashEvaluated alternative | 84.0%420/500 | 82/500 | $1.0833 | 2.09 s / 4.38 s | 76.71 |
| GPT-6 LunaRecommended for validation | 84.8%424/500 | 95/500 | $0.0842 | 2.52 s / 4.08 s | 82.45 |
| GPT-6.1 SolEvaluated alternative | 83.4%417/500 | 86/500 | $1.3176 | 2.07 s / 4.53 s | 75.17 |
| Claude Haiku 5.5Unranked: Incomplete charges | 81.8%409/500 | 115/5007 invalid outputs; 4 provider failures | Unknown496/500 cases fully priced | 0.98 s / 2.83 s | Unranked |
| Claude Sonnet 5.5Unranked: More primary errors than incumbent | 73.6%368/500 | 154/500109 invalid outputs | $2.2768 | 1.68 s / 3.72 s | Unranked |
| Grok 4.7Evaluated alternative | 88.0%440/500 | 98/500 | $3.2315 | 2.81 s / 9.59 s | 68.65 |
Consistency: identical valid intent labels / all planned cases, including shared errors. Cost: recorded USD charges, normalized per 1,000 cases. Time: summed HTTP attempts, including failures/retries, excluding pauses. Separate collection windows make timing descriptive.
Weights: correctness 50% · consistency 20% · cost 20% · p95 time 10%. Score is not confidence; Conditional is an editorial rating.
06 / Current model issues
o4-mini
- 126 wrong labels.
- 0 invalid outputs · 0 provider/unusable responses · 0 unresolved outcomes.
07 / Recommended model issues
GPT-6 Luna
- 95 wrong labels: 12 regressions, 62 shared errors, 21 different wrong answers.
- 0 invalid outputs · 0 provider/unusable responses · 0 unresolved outcomes.
Matching and non-matching examples.
Recorded outputs with publisher reference labels.
500 pairs with valid labels out of 500 primary cases.
0 invalid, failed or unresolved pairs excluded from these five categories.
One example per outcome; counts cover all valid pairs.
Other outcomes
Agreement can also preserve an error, and a different answer can correct one. These outcomes are separate from the regression example above.
Showing candidate regression for GPT-6 Luna, case banking77:test:274.
Non-matching
Incumbent correct · candidate wrong
Banking77 input Case banking77:test:274
How man currencies can I hold?
Publisher reference fiat_currency_support
Incumbent
o4-mini
Actual visible response
{"intent":"fiat_currency_support"}Scored intent fiat_currency_support
Candidate
GPT-6 Luna
Actual visible response
{"intent":"supported_cards_and_currencies"}Scored intent supported_cards_and_currencies
How these examples were selected
One example per observed outcome and candidate: 55 recorded comparisons across 11 candidates. Each is the lowest numeric case ID in its category among the 500 primary cases. These examples illustrate outcomes; their equal representation does not describe how often each outcome occurs.
Both outputs must contain a valid intent. Invalid outputs, failed requests and unresolved outcomes remain in the report’s aggregate results but cannot supply two comparable intent labels here. Labels are scored against the unchanged publisher reference, not a customer review.
Matching means the same parsed intent label, not identical response text. The primary matching example has two correct answers; the non-matching example is a regression. Shared wrong answers and the other types of disagreement are available under Other outcomes.
Responses are shown as recorded. The scored intent is parsed separately; hidden reasoning is not shown. A successful policy-allowed retry supplies the final answer where applicable, with first attempts retained in the report’s recovery, cost and timing totals.
Dataset: Banking77, test split (dataset license). Revision sha256:d12d6e3bc4c3103966ae786dc435913c0c563dfa328f5a3646d0e62cfeeb474d.
Incumbent model openai/o4-mini, route openai. Response receipt SHA-256:b4127726699f600fce2293fbc3eba31bfb310e9297d5cd9941f43e92aa6bb32e
Candidate model openai/gpt-6-luna, route openai. Response receipt SHA-256:ca3de7c03d33d000a3d44e7bd537c2ce80b9d73f713cba133f90622a65acf701
Comparison source SHA-256:3b41e33295c60f5b80807797c9446aed1ff644c022af72442246dbcb770f91ca
08 / Next action and evidence
Validate before migration.
Test GPT-6 Luna on a fresh held-out workflow sample against an agreed acceptance policy, including observed regressions and challenge failures.
Technical evidence: coverage, recovery, accounting and sources
Collection history
| Segment / status | Initial / retry calls | Collection window (UTC) | Known charges | Held unknown-charge reserve |
|---|---|---|---|---|
| Operational prefixReadiness gate failed | 35 / 0 | to | $0.04103170 | $0 |
| Stopped continuationStopped after provider rejection | 1740 / 0 | to | $2.08393090 | $0.001792 |
| Untouched continuationCompleted; charges remain unknown | 1935 / 0 | to | $2.28802765 | $0.005376 |
Prior responses and charges are retained. Continuations cover only untouched requests; rejected requests are not replayed. Held reserves are not measured charges.
Confirmed rejections with unknown charges
- Claude Haiku 5.5 · Case banking77:test:2602: HTTP 429, confirmed no answer. Charge unknown; $0.001792 remains reserved.
- Claude Haiku 5.5 · Case banking77:test:2606: HTTP 429, confirmed no answer. Charge unknown; $0.001792 remains reserved.
- Claude Haiku 5.5 · Case banking77:test:2608: HTTP 429, confirmed no answer. Charge unknown; $0.001792 remains reserved.
- Claude Haiku 5.5 · Case banking77:test:2612: HTTP 429, confirmed no answer. Charge unknown; $0.001792 remains reserved.
A confirmed no-answer response is a delivery failure. Its missing charge remains unknown and prevents ranking.
Inspect collection source identities
Continuation amendment:7b9e5e5933782f87c18368a75c52d341e778b5a1ab27ba630a159d4483007014
Operational prefix
Plan:7d754c636ae5406ccd0ed86dee815bcc7b1e112167c68dedc9df0d05ae8368f4
Journal:9a6a559f8bfe814bacb318498cfc7b81a5c6b6bec80436ef7dfb9feb6743c85d
Prefix review:5b6889ddf760f78012cf73f374ecc96e88823651fec1733a6b1ad98fb7580ddf
Stopped continuation
Plan:e1cbd00495afb9c5aa74e2b024c39d0034eb1c70130dbdc55109b1d79661075b
Journal:a2069c72f60b76f91dd1badf8b46e6900cd2522823e7b3b1fe75766efef65ad8
Stopped result:2a19b2b06747ff660401ac72a54cb257ed4194c6391eb8c36b9ed84db2f2ec20
Untouched continuation
Plan:d0e53b2ccbc823fb7a960bf5cf31e12c3c95b2cfc7465691725175e81e7e649c
Journal:e2e0b650a36824e09db0d4544ad45d7dc8c42149aea7ec89906baf3421e68a1b
Completed result:9629be29651ded2dc859fbbc6efb2ca0dd65e9a79d91831066da94286cb20563
Claude Haiku 5.5 · banking77:test:2602
Original receipt:3e779bd93881982832879d4175c760b5de39ab51ea75225b781857d073480ce5
Original response:659d11b962d28a9e94f6a85b3e36706522f07c7e3ade1100f0996b8baf95b1a4
Claude Haiku 5.5 · banking77:test:2606
Original receipt:e80f6b7da176ea0b1806f0a325e8653c74bf164164bf5fe9f8e9ba3d2bee5c77
Original response:f2f5403d6211249b3d22d3c459e0df158979d608ba6979f8a664c2bf8659c2ad
Claude Haiku 5.5 · banking77:test:2608
Original receipt:8c3796ec2ee0bdd2947fc8ef542631fa4b8533f504c1444823feea9e591144f5
Original response:c937e2ba8a5d6d4a29bfc6e0d75951b0547f828048b344c85836eee640156405
Claude Haiku 5.5 · banking77:test:2612
Original receipt:0661b90d0edb9af0033279c028c530a3d7fb72294c001737ce4f798f437e0cc8
Original response:33f9c90ff70fcf71e001931dcd9da3a862fba9594098d73d785dd3f57946e16c
Collection and model configurations
Retained measurements are unchanged. Models collected later use the same public cases and references. Collection windows and provider reasoning settings differ; low-effort labels do not establish equal compute.
| Model | Route and reasoning | Output cap / retry cap | Evidence origin and collection |
|---|---|---|---|
o4-miniopenai/o4-mini | openaiEnabled | 2,048 / 4,096 | Retained history2026-10-13 01:45:24 UTC to 2026-10-13 06:29:05 UTC |
DeepSeek V4 Flashdeepseek/deepseek-v4-flash-0731 | streamlake/fp8Low effort | 8,192 / 16,384 | Retained history2026-10-13 01:45:24 UTC to 2026-10-13 06:29:05 UTC |
Qwen3.7-Flashqwen/qwen3.7-flash | alibaba/us-east-11,024 reasoning tokens | 2,048 / 4,096 | Retained history2026-10-13 01:45:24 UTC to 2026-10-13 06:29:05 UTC |
Nemotron 3.5 Lightningnvidia/nemotron-3.5-lightning | deepinfra/bf16Enabled | 8,192 / 16,384 | Retained history2026-10-13 01:45:24 UTC to 2026-10-13 06:29:05 UTC |
GLM-5.3-Flashz-ai/glm-5.3-flash | z-ai/fp8Low effort | 2,048 / 4,096 | Retained history2026-10-13 01:45:24 UTC to 2026-10-13 06:29:05 UTC |
Gemini 3.5 Flash Litegoogle/gemini-3.5-flash-lite | google-ai-studioLow effort | 2,048 / 4,096 | New collection2026-10-13 16:22:28 UTC to 2026-10-13 19:11:44 UTC |
Gemini 3.8 Flashgoogle/gemini-3.8-flash | google-ai-studioLow effort | 2,048 / 4,096 | New collection2026-10-13 16:22:29 UTC to 2026-10-13 19:11:46 UTC |
GPT-6 Lunaopenai/gpt-6-luna | openaiLow effort | 2,048 / 4,096 | New collection2026-10-13 16:22:31 UTC to 2026-10-13 19:11:47 UTC |
GPT-6.1 Solopenai/gpt-6.1-sol | openaiLow effort | 2,048 / 4,096 | New collection2026-10-13 16:22:33 UTC to 2026-10-13 19:11:49 UTC |
Claude Haiku 5.5anthropic/claude-haiku-5.5 | anthropicLow effort | 2,048 / 4,096 | New collection2026-10-13 16:22:36 UTC to 2026-10-13 19:11:37 UTC |
Claude Sonnet 5.5anthropic/claude-sonnet-5.5 | anthropicLow effort | 2,048 / 4,096 | New collection2026-10-13 16:22:37 UTC to 2026-10-13 19:11:39 UTC |
Grok 4.7x-ai/grok-4.7 | xaiLow effort | 2,048 / 4,096 | New collection2026-10-13 16:22:40 UTC to 2026-10-13 19:11:43 UTC |
Eligibility
- o4-mini: Complete outcomes, charges, timing and model identity, with no more primary errors than the incumbent.
- DeepSeek V4 Flash: Unresolved or missing outcomes; Incomplete charges
- Qwen3.7-Flash: Complete outcomes, charges, timing and model identity, with no more primary errors than the incumbent.
- Nemotron 3.5 Lightning: Incomplete charges
- GLM-5.3-Flash: Complete outcomes, charges, timing and model identity, with no more primary errors than the incumbent.
- Gemini 3.5 Flash Lite: Complete outcomes, charges, timing and model identity, with no more primary errors than the incumbent.
- Gemini 3.8 Flash: Complete outcomes, charges, timing and model identity, with no more primary errors than the incumbent.
- GPT-6 Luna: Complete outcomes, charges, timing and model identity, with no more primary errors than the incumbent.
- GPT-6.1 Sol: Complete outcomes, charges, timing and model identity, with no more primary errors than the incumbent.
- Claude Haiku 5.5: Incomplete charges
- Claude Sonnet 5.5: More primary errors than incumbent
- Grok 4.7: Complete outcomes, charges, timing and model identity, with no more primary errors than the incumbent.
Primary answer coverage
| Model | End-to-end consistencyAll planned cases | Valid-answer agreementObserved both-valid pairs | Valid-output rateAll planned cases |
|---|---|---|---|
| o4-mini | 100.0%500/500 | 100.0%500/500 observed both-valid pairs | 100.0%500/500 planned cases |
| DeepSeek V4 Flash | Unavailable437/500 confirmed; 1 unresolvedPossible range: 87.4% to 87.6% | 87.8%437/498 observed both-valid pairs | Unavailable498/500 planned cases; 1 unresolvedPossible range: 99.6% to 99.8% |
| Qwen3.7-Flash | 84.4%422/500 | 84.6%422/499 observed both-valid pairs | 99.8%499/500 planned cases |
| Nemotron 3.5 Lightning | 83.8%419/500 | 84.3%419/497 observed both-valid pairs | 99.4%497/500 planned cases |
| GLM-5.3-Flash | 84.4%422/500 | 84.4%422/500 observed both-valid pairs | 100.0%500/500 planned cases |
| Gemini 3.5 Flash Lite | 85.0%425/500 | 85.0%425/500 observed both-valid pairs | 100.0%500/500 planned cases |
| Gemini 3.8 Flash | 84.0%420/500 | 84.0%420/500 observed both-valid pairs | 100.0%500/500 planned cases |
| GPT-6 Luna | 84.8%424/500 | 84.8%424/500 observed both-valid pairs | 100.0%500/500 planned cases |
| GPT-6.1 Sol | 83.4%417/500 | 83.4%417/500 observed both-valid pairs | 100.0%500/500 planned cases |
| Claude Haiku 5.5 | 81.8%409/500 | 83.6%409/489 observed both-valid pairs | 97.8%489/500 planned cases |
| Claude Sonnet 5.5 | 73.6%368/500 | 94.1%368/391 observed both-valid pairs | 78.2%391/500 planned cases |
| Grok 4.7 | 88.0%440/500 | 88.0%440/500 observed both-valid pairs | 100.0%500/500 planned cases |
Valid-answer agreement is conditional on both outputs being valid. It excludes failed, invalid and unresolved pairs and can include shared wrong labels. Read it alongside end-to-end consistency and valid-output coverage; unresolved outcomes make complete rates unavailable.
Separate challenge cohort
| Model | Correct | Confirmed errors | Unresolved |
|---|---|---|---|
| o4-mini | 25/30 | 5/30 | 0 |
| DeepSeek V4 Flash | 25/30 | 5/30 | 0 |
| Qwen3.7-Flash | 24/30 | 6/30 | 0 |
| Nemotron 3.5 Lightning | 25/30 | 5/30 | 0 |
| GLM-5.3-Flash | 23/30 | 7/30 | 0 |
| Gemini 3.5 Flash Lite | 27/30 | 3/30 | 0 |
| Gemini 3.8 Flash | 27/30 | 3/30 | 0 |
| GPT-6 Luna | 28/30 | 2/30 | 0 |
| GPT-6.1 Sol | 27/30 | 3/30 | 0 |
| Claude Haiku 5.5 | 24/30 | 6/30 | 0 |
| Claude Sonnet 5.5 | 23/30 | 7/30 | 0 |
| Grok 4.7 | 25/30 | 5/30 | 0 |
Challenge cases remain outside the primary score. Unresolved cases are unknown, not confirmed wrong answers.
| Model | End-to-end consistencyAll planned cases | Valid-answer agreementObserved both-valid pairs | Valid-output rateAll planned cases |
|---|---|---|---|
| o4-mini | 100.0%30/30 | 100.0%30/30 observed both-valid pairs | 100.0%30/30 planned cases |
| DeepSeek V4 Flash | 93.3%28/30 | 93.3%28/30 observed both-valid pairs | 100.0%30/30 planned cases |
| Qwen3.7-Flash | 96.7%29/30 | 96.7%29/30 observed both-valid pairs | 100.0%30/30 planned cases |
| Nemotron 3.5 Lightning | 93.3%28/30 | 93.3%28/30 observed both-valid pairs | 100.0%30/30 planned cases |
| GLM-5.3-Flash | 90.0%27/30 | 90.0%27/30 observed both-valid pairs | 100.0%30/30 planned cases |
| Gemini 3.5 Flash Lite | 93.3%28/30 | 93.3%28/30 observed both-valid pairs | 100.0%30/30 planned cases |
| Gemini 3.8 Flash | 93.3%28/30 | 93.3%28/30 observed both-valid pairs | 100.0%30/30 planned cases |
| GPT-6 Luna | 90.0%27/30 | 90.0%27/30 observed both-valid pairs | 100.0%30/30 planned cases |
| GPT-6.1 Sol | 86.7%26/30 | 86.7%26/30 observed both-valid pairs | 100.0%30/30 planned cases |
| Claude Haiku 5.5 | 83.3%25/30 | 83.3%25/30 observed both-valid pairs | 100.0%30/30 planned cases |
| Claude Sonnet 5.5 | 83.3%25/30 | 100.0%25/25 observed both-valid pairs | 83.3%25/30 planned cases |
| Grok 4.7 | 100.0%30/30 | 100.0%30/30 observed both-valid pairs | 100.0%30/30 planned cases |
Recovery by cohort
| Model / cohort | First attempts | Output-limit failures | Required / completed retries | Recovered valid / correct | HTTP attempts |
|---|---|---|---|---|---|
| o4-miniprimary | 500/500 | 1 | 1 / 1 | 1 / 0 | 501 |
| DeepSeek V4 Flashprimary | 500/500 | 0 | 0 / 0 | 0 / 0 | 500 |
| Qwen3.7-Flashprimary | 500/500 | 0 | 0 / 0 | 0 / 0 | 500 |
| Nemotron 3.5 Lightningprimary | 500/500 | 0 | 0 / 0 | 0 / 0 | 500 |
| GLM-5.3-Flashprimary | 500/500 | 0 | 0 / 0 | 0 / 0 | 500 |
| o4-minichallenge | 30/30 | 0 | 0 / 0 | 0 / 0 | 30 |
| DeepSeek V4 Flashchallenge | 30/30 | 0 | 0 / 0 | 0 / 0 | 30 |
| Qwen3.7-Flashchallenge | 30/30 | 0 | 0 / 0 | 0 / 0 | 30 |
| Nemotron 3.5 Lightningchallenge | 30/30 | 0 | 0 / 0 | 0 / 0 | 30 |
| GLM-5.3-Flashchallenge | 30/30 | 0 | 0 / 0 | 0 / 0 | 30 |
| Gemini 3.5 Flash Liteprimary | 500/500 | 0 | 0 / 0 | 0 / 0 | 500 |
| Gemini 3.5 Flash Litechallenge | 30/30 | 0 | 0 / 0 | 0 / 0 | 30 |
| Gemini 3.8 Flashprimary | 500/500 | 0 | 0 / 0 | 0 / 0 | 500 |
| Gemini 3.8 Flashchallenge | 30/30 | 0 | 0 / 0 | 0 / 0 | 30 |
| GPT-6 Lunaprimary | 500/500 | 0 | 0 / 0 | 0 / 0 | 500 |
| GPT-6 Lunachallenge | 30/30 | 0 | 0 / 0 | 0 / 0 | 30 |
| GPT-6.1 Solprimary | 500/500 | 0 | 0 / 0 | 0 / 0 | 500 |
| GPT-6.1 Solchallenge | 30/30 | 0 | 0 / 0 | 0 / 0 | 30 |
| Claude Haiku 5.5primary | 500/500 | 0 | 0 / 0 | 0 / 0 | 500 |
| Claude Haiku 5.5challenge | 30/30 | 0 | 0 / 0 | 0 / 0 | 30 |
| Claude Sonnet 5.5primary | 500/500 | 0 | 0 / 0 | 0 / 0 | 500 |
| Claude Sonnet 5.5challenge | 30/30 | 0 | 0 / 0 | 0 / 0 | 30 |
| Grok 4.7primary | 500/500 | 0 | 0 / 0 | 0 / 0 | 500 |
| Grok 4.7challenge | 30/30 | 0 | 0 / 0 | 0 / 0 | 30 |
Actual charges and coverage
| Model | Known primary subtotal | Fully priced | Timed | Valid-output pairs |
|---|---|---|---|---|
| o4-mini | $0.8806 | 500/500 | 500/500 | 500/500 |
| DeepSeek V4 Flash | $0.0732 | 499/500 | 500/500 | 498/500 |
| Qwen3.7-Flash | $0.0367 | 500/500 | 500/500 | 499/500 |
| Nemotron 3.5 Lightning | $0.1256 | 499/500 | 500/500 | 497/500 |
| GLM-5.3-Flash | $0.0141 | 500/500 | 500/500 | 500/500 |
| Gemini 3.5 Flash Lite | $0.1159 | 500/500 | 500/500 | 500/500 |
| Gemini 3.8 Flash | $0.5416 | 500/500 | 500/500 | 500/500 |
| GPT-6 Luna | $0.0421 | 500/500 | 500/500 | 500/500 |
| GPT-6.1 Sol | $0.6588 | 500/500 | 500/500 | 500/500 |
| Claude Haiku 5.5 | $0.0543 | 496/500 | 500/500 | 489/500 |
| Claude Sonnet 5.5 | $1.1384 | 500/500 | 500/500 | 391/500 |
| Grok 4.7 | $1.6158 | 500/500 | 500/500 | 500/500 |
Known reported subtotal: $5.5974860649. Held uncertainty reserve: $0.013845. All 12 compared model configurations; primary and challenge cases.
Unknown charges prevent normalized cost and ranking. Known subtotals are not complete totals or invoices; the reserve is not a measured charge. Possible outcome ranges are arithmetic bounds, not confidence intervals.
Formatting diagnostics for newly collected models
| Model / cohort | Strict-format failures | Valid labels inside fences |
|---|---|---|
| Gemini 3.5 Flash Liteprimary | 132 | 132 |
| Gemini 3.5 Flash Litechallenge | 14 | 14 |
| Gemini 3.8 Flashprimary | 3 | 3 |
| Gemini 3.8 Flashchallenge | 0 | 0 |
| GPT-6 Lunaprimary | 0 | 0 |
| GPT-6 Lunachallenge | 0 | 0 |
| GPT-6.1 Solprimary | 0 | 0 |
| GPT-6.1 Solchallenge | 0 | 0 |
| Claude Haiku 5.5primary | 7 | 0 |
| Claude Haiku 5.5challenge | 0 | 0 |
| Claude Sonnet 5.5primary | 109 | 0 |
| Claude Sonnet 5.5challenge | 5 | 0 |
| Grok 4.7primary | 0 | 0 |
| Grok 4.7challenge | 0 | 0 |
The fixed scorer accepts a valid intent inside JSON fences. Formatting diagnostics are separate and do not redefine the label score.
Declared preference profiles
| Profile | Correctness / consistency / cost / p95 time | Preferred eligible models |
|---|---|---|
| default | 50% / 20% / 20% / 10% | GPT-6 Luna |
| quality | 70% / 10% / 10% / 10% | GPT-6 Luna |
| economy | 50% / 10% / 30% / 10% | GPT-6 Luna |
| latency | 50% / 10% / 10% / 30% | Gemini 3.5 Flash Lite |
Cost and time utility is incumbent / (incumbent + candidate); equal values score 0.5. Weights are unchanged. Ties are retained, missing evidence does not redistribute weights, and the incumbent remains an option.
Source identity
Audited summary SHA-256:3b41e33295c60f5b80807797c9446aed1ff644c022af72442246dbcb770f91ca
Retained five-model summary:31d2f1a4ddc87eb9a6682c2c69f51a679d0c51f91bac9279772aa51c3d209b1d
Extension plan:7d754c636ae5406ccd0ed86dee815bcc7b1e112167c68dedc9df0d05ae8368f4
Untouched continuation plan:e1cbd00495afb9c5aa74e2b024c39d0034eb1c70130dbdc55109b1d79661075b
Readiness amendment:c2e3f6e758920525a56cc1e22882a2aab8b562bda1e39195d7cc2e9c57e94b19
Rejection continuation plan:d0e53b2ccbc823fb7a960bf5cf31e12c3c95b2cfc7465691725175e81e7e649c
Rejection amendment:7b9e5e5933782f87c18368a75c52d341e778b5a1ab27ba630a159d4483007014
Final result:9629be29651ded2dc859fbbc6efb2ca0dd65e9a79d91831066da94286cb20563
Independent audit:ee9496dae06a559b90191b577fab02107ed411979801b78523d4479175206cdb
Audited input manifest:9d5b9237701042b222b2ebcccb5754225f7baefaa22e09d828df36343d449b17
Prepared 2026-10-13 19:47:57 UTC. Public reused cases and unchanged publisher labels support descriptive comparison, not customer adjudication or production approval.