October 10 snapshot / five-model benchmark
Five configurations.
One workflow.
Compare actual outputs, errors, cost and response time on the same banking-support cases.
Public Banking77 benchmark. Plain prompt, no retrieval. These previously exposed cases support a descriptive comparison, not a fresh held-out assessment or production approval.
Measured · 500 primary cases + 30 separate challenge cases
Identified workflow
Banking support intent classification
Current model
o4-mini / OpenAI route
Preferred eligible configuration
GLM-5.3-Flash
Under the declared weights, including the option to retain the incumbent.
Weighted preference score
Correctness 50% · end-to-end incumbent agreement 20% · cost 20% · p95 response time 10%. The score reflects these priorities; it is not statistical confidence.
The existing published assessment and its retrieved-Luna configuration remain separate. This example does not change a customer verdict, acceptance policy or rollout authorization.
01 / Primary cohort
The same 500 cases, side by side.
End-to-end consistency counts matching valid intent labels across all planned cases. Failed or unusable answers count against it; shared wrong answers can still match.
| Configuration | End-to-end consistency | Correct labels | All errors | USD / 1,000 | p50 | p95 | Score / 100 |
|---|---|---|---|---|---|---|---|
| o4-miniIncumbent | 100.0%500/500 matches | 75.8%379/500 correct | 121/500 | $1.7566500/500 charges known | 2.26 s | 5.92 s | 72.90 |
| DeepSeek V4 FlashEvaluated candidate | 30.6%153/500 matches | 25.6%128/500 correct | 372/500 | Unknown177/500 charges known | 0.16 s | 15.72 s | Unranked |
| Qwen3.7-FlashEvaluated candidate | 84.2%421/500 matches | 73.6%368/500 correct | 132/500 | Unknown491/500 charges known | 4.04 s | 13.19 s | Unranked |
| Nemotron 3.5 LightningEvaluated candidate | 82.2%411/500 matches | 74.6%373/500 correct | 127/500 | $0.2020500/500 charges known | 3.22 s | 5.28 s | Unranked |
| GLM-5.3-FlashPreferred eligible configuration | 86.0%430/500 matches | 79.6%398/500 correct | 102/500 | $0.0274500/500 charges known | 1.31 s | 3.08 s | 83.27 |
On small screens, scroll the table horizontally to inspect every metric. Costs normalize actual primary charges to 1,000 requests; they are not a measured scaled workload.
Agreement and answer coverage.
Valid-answer agreement isolates cases where both models returned a valid label. Read it alongside valid-output coverage: excluding failed answers can make agreement look stronger.
| Configuration | End-to-end consistency | Valid-answer agreement | Valid-output rate |
|---|---|---|---|
| o4-mini | 100.0%500/500 planned cases | 100.0%500/500 both-valid pairs | 100.0%500/500 planned cases |
| DeepSeek V4 Flash | 30.6%153/500 planned cases | 86.4%153/177 both-valid pairs | 35.4%177/500 planned cases |
| Qwen3.7-Flash | 84.2%421/500 planned cases | 86.1%421/489 both-valid pairs | 97.8%489/500 planned cases |
| Nemotron 3.5 Lightning | 82.2%411/500 planned cases | 83.7%411/491 both-valid pairs | 98.2%491/500 planned cases |
| GLM-5.3-Flash | 86.0%430/500 planned cases | 86.0%430/500 both-valid pairs | 100.0%500/500 planned cases |
A valid output contains an accepted intent label; it may still be wrong. This breakdown is derived from the recorded counts; the original PDF and CSV remain unchanged. Conditional agreement and output coverage do not replace end-to-end agreement in the frozen weighted score or change eligibility.
Why some configurations are unranked
- o4-mini: Complete evidence and no more primary errors than the incumbent.
- DeepSeek V4 Flash: Some primary attempt charges are unknown. Full-run billing is incomplete. More primary errors than the incumbent.
- Qwen3.7-Flash: Some primary attempt charges are unknown. Full-run billing is incomplete. More primary errors than the incumbent.
- Nemotron 3.5 Lightning: More primary errors than the incumbent.
- GLM-5.3-Flash: Complete evidence and no more primary errors than the incumbent.
Eligibility checks all 530 planned cases for outcomes, known attempt charges, timing and model identity. Missing evidence never redistributes the weights or turns an unknown charge into zero.
02 / Incumbent and candidate issues
Where the outputs differ.
Wrong labels are separate from service and output failures. Corrections and regressions below use pairs with valid labels.
| Configuration | Wrong labels | Provider / unusable response | Invalid output | Corrections | Regressions | Shared wrong label |
|---|---|---|---|---|---|---|
| o4-mini | 121 | 0 | 0 | 0 | 0 | 121 |
| DeepSeek V4 Flash | 49 | 323 | 0 | 8 | 11 | 33 |
| Qwen3.7-Flash | 121 | 9 | 2 | 24 | 25 | 77 |
| Nemotron 3.5 Lightning | 118 | 2 | 7 | 24 | 23 | 62 |
| GLM-5.3-Flash | 102 | 0 | 0 | 33 | 14 | 65 |
All planned attempts are accounted for. No missing or unresolved outcomes and no wrong-model responses remain in this collection. Unknown billing for explicit provider rejections is still retained. A provider/unusable-response count can include a billed unusable response; it does not always mean a rate limit.
03 / Separate diagnostic cohort
The 30 challenge cases.
These cases do not enter the primary denominator or weighted scores.
| Configuration | End-to-end consistency | Valid-answer agreement | Valid-output rate | Correct labels | All errors |
|---|---|---|---|---|---|
| o4-mini | 100.0%30/30 planned cases | 100.0%30/30 both-valid pairs | 100.0%30/30 planned cases | 25/30 (83.3%) | 5/30 |
| DeepSeek V4 Flash | 30.0%9/30 planned cases | 90.0%9/10 both-valid pairs | 33.3%10/30 planned cases | 8/30 (26.7%) | 22/30 |
| Qwen3.7-Flash | 93.3%28/30 planned cases | 93.3%28/30 both-valid pairs | 100.0%30/30 planned cases | 25/30 (83.3%) | 5/30 |
| Nemotron 3.5 Lightning | 90.0%27/30 planned cases | 90.0%27/30 both-valid pairs | 100.0%30/30 planned cases | 24/30 (80.0%) | 6/30 |
| GLM-5.3-Flash | 86.7%26/30 planned cases | 86.7%26/30 both-valid pairs | 100.0%30/30 planned cases | 24/30 (80.0%) | 6/30 |
GLM recorded 24/30 correct challenge labels; the incumbent recorded 25/30. Its primary-cohort recommendation is not a claim that it wins on every case or cohort.
04 / Declared priorities
How the recommendation is weighted.
| Profile | Correctness | Agreement | Cost | p95 time | Preferred eligible model |
|---|---|---|---|---|---|
| Default | 50% | 20% | 20% | 10% | GLM-5.3-Flash |
| Quality priority | 70% | 10% | 10% | 10% | GLM-5.3-Flash |
| Cost priority | 50% | 10% | 30% | 10% | GLM-5.3-Flash |
| Latency priority | 50% | 10% | 10% | 30% | GLM-5.3-Flash |
Cost and time utilities are incumbent / (incumbent + candidate). Equal values score 0.5; both-zero utility is also 0.5. The frozen scores and winner lists are displayed as recorded, with ties retained.
05 / Method and provenance
Exact settings. Visible limits.
All arms used the same original intent prompt, no retrieval and a 2,048-token output cap. Requests were serial with rotating model order and no retries. The 2,650 attempts span 3 preserved collection segments. Percentiles use linear interpolation.
Publisher labels are unchanged, not customer adjudication. The cases were previously exposed, so these numbers do not establish fresh held-out confirmation, customer-traffic performance or production savings. Provider route and reasoning settings are part of each tested configuration.
Known reported charges across the full run total $1.0938. A further $0.2685 is reserved for uncertain charges. The reserve is a budget allowance, not a measured bill; the reported subtotal is not a final invoice total.
| Exact model ID | Provider route | Reasoning |
|---|---|---|
openai/o4-mini | openai | Enabled |
deepseek/deepseek-v4-flash-0731 | deepinfra/fp8 | Low effort |
qwen/qwen3.7-flash | alibaba/us-east-1 | 1,024 reasoning tokens |
nvidia/nemotron-3.5-lightning | deepinfra/bf16 | Enabled |
z-ai/glm-5.3-flash | z-ai/fp8 | Low effort |
Inspect source identity and collection windows
Source summary SHA-256: 7687c9fbec7063ae1835e8d44ff9611e51f3bc882d315325c917ca6bc3e54f5e
Prepared October 10, 2026 UTC. The PDF, CSV and curated JSON are the audited October 10 snapshot.
- parent: to . First collection segment.
- continuation: to . Prior pause 653.77 seconds.
- second continuation: to . Prior pause 248.70 seconds.