← All articles

RESEARCH / MULTIMODAL DECISIONS

CMDB-1500Multimodal Decision Model Benchmark

1,200 text questions and 300 image questions, measuring decisions and probability estimates.

Filter
SPX-CDJevGeneral reasoning modelsOther decision models

View exact text results
CMDB-1500 Text results; fixed total 1,200, valid answers counted over the full evaluated scope
RankModel / versionAccuracy · 0–100%Correct / fixed totalValid / evaluated scope
01GPT 6.1 Sol · low80.00%960 / 1,2001,500 / 1,500
02SPX-CD-Pro79.33%952 / 1,2001,500 / 1,500
03Claude Sonnet 5.5 · low78.00%936 / 1,2001,500 / 1,500
04StartLux-Decision-27B · BF1677.92%935 / 1,2001,500 / 1,500
05pplx-decider · v1.1 27B77.83%934 / 1,2001,500 / 1,500
06SPX-CD-Flash77.58%931 / 1,2001,500 / 1,500
07GPT 6 Luna · low76.58%919 / 1,2001,500 / 1,500
08Microsoft-Decision-1 ·9B76.17%914 / 1,2001,200 / 1,200
09GLM 5.3 Flash · low75.75%909 / 1,2001,500 / 1,500
10StartLux-Decision-35B-A3B · BF1675.67%908 / 1,2001,500 / 1,500
11SPX-CD-Omni74.67%896 / 1,2001,500 / 1,500
12Mercury Decide73.92%887 / 1,2001,200 / 1,200
13pplx-decider 27B72.83%874 / 1,2001,500 / 1,500
14Jev72.17%866 / 1,2001,200 / 1,200
15Open-Jev · Zefan 27B71.67%860 / 1,2001,199 / 1,200
16Cloudflare Clef71.50%858 / 1,2001,485 / 1,500
17JPT · 9B71.33%856 / 1,2001,500 / 1,500
18Qwen 3.8 Flash71.08%853 / 1,2001,500 / 1,500
19Kimi K2.671.00%852 / 1,2001,500 / 1,500
20Mapika · Decider 4B v2.169.83%838 / 1,2001,200 / 1,200
21Cloudflare Clef-flash69.67%836 / 1,2001,485 / 1,500
22GPT 6 Luna Decisions69.67%836 / 1,2001,498 / 1,500
23Imajev-9B68.83%826 / 1,2001,478 / 1,500
24Kev · 9B68.67%824 / 1,2001,200 / 1,200
25DeepSeek V4.1 Flash68.58%823 / 1,2001,500 / 1,500
26JPT · 4B67.67%812 / 1,2001,500 / 1,500
27Decision Lux · 9B67.42%809 / 1,2001,200 / 1,200
28Quyet-1.0 · Large 31B67.08%805 / 1,2001,085 / 1,200
29Jet · v6.2 4B66.92%803 / 1,2001,200 / 1,200
30Open-Jev · Zefan 9B66.67%800 / 1,2001,199 / 1,200
31Wity-1 · auto66.67%800 / 1,2001,426 / 1,500
32Nimble · v2 9B66.58%799 / 1,2001,200 / 1,200
33Hopper G · 4B v1.365.92%791 / 1,2001,200 / 1,200
34Malkuth · 4B65.58%787 / 1,2001,200 / 1,200
35Hopper · 4B65.50%786 / 1,2001,200 / 1,200
36Winnow · 12B Q865.33%784 / 1,2001,410 / 1,500
37JevK564.83%778 / 1,2001,200 / 1,200
38Lev · 4B64.58%775 / 1,2001,200 / 1,200
39Kev ·4B API64.33%772 / 1,2001,200 / 1,200
40Mica · 4B BF16 GGUF64.25%771 / 1,2001,200 / 1,200
41Kev · 4B64.17%770 / 1,2001,200 / 1,200
42Jev-Omni · 12B63.58%763 / 1,2001,493 / 1,500
43APUS OpenJev · 9B 3000 High61.42%737 / 1,2001,085 / 1,200
44OpenDecider · Small61.25%735 / 1,2001,200 / 1,200
45APUS OpenJev · 9B 5949 High61.08%733 / 1,2001,085 / 1,200
46OpenDecider · Small TD60.67%728 / 1,2001,200 / 1,200
47APUS OpenJev · 4B 5949 High60.08%721 / 1,2001,085 / 1,200
48Ateve Jev59.75%717 / 1,2001,087 / 1,200
49Bocha Jev58.92%707 / 1,2001,087 / 1,200
50Winnow · E4B Q858.92%707 / 1,2001,410 / 1,500
51Reflex · Stable 4B58.50%702 / 1,2001,387 / 1,500
52Plumb · v5.2 4B58.08%697 / 1,2001,085 / 1,200
53FLock THIS/THAT57.67%692 / 1,2001,156 / 1,200
54Kev · 0.8B51.75%621 / 1,2001,200 / 1,200
55Bosun v3.1 · 1.7B49.75%597 / 1,2001,200 / 1,200
56OpenDecider · Nano46.75%561 / 1,2001,143 / 1,200
57CalDec · Laya46.08%553 / 1,2001,189 / 1,200
58Bosun v3.1 · 0.6B45.58%547 / 1,2001,200 / 1,200
59Laya · Typed Decisions44.58%535 / 1,2001,189 / 1,200
60CalDec · GLiNER43.17%518 / 1,2001,200 / 1,200
61Laya · English41.00%492 / 1,2001,189 / 1,200
62Laya · Multilingual40.25%483 / 1,2001,192 / 1,200
63GLiNER2.5-Decide38.58%463 / 1,2001,200 / 1,200
64Julia-135.75%429 / 1,2001,079 / 1,200
65Liquid AI · d1-omni-600M35.50%426 / 1,2001,128 / 1,500
66CLM v0.1 · 8B34.92%419 / 1,2001,200 / 1,200

The multimodal filter shows models evaluated on image questions.

Pooled ECE calibration leaderboard

Taller bars mean lower error ↑
View exact ECE results
Complete ECE results
RankModel / versionECE ↓
01Microsoft-Decision-1 ·9B2.35%
02SPX-CD-Pro2.46%
03SPX-CD-Flash2.49%
04StartLux-Decision-27B · BF162.65%
05StartLux-Decision-35B-A3B · BF162.71%
06pplx-decider 27B3.08%
07Reflex · Stable 4B3.33%
08pplx-decider · v1.1 27B3.42%
09JPT · 9B3.47%
10Plumb · v5.2 4B3.53%
11Quyet-1.0 · Large 31B3.64%
12Imajev-9B3.74%
13GLiNER2.5-Decide3.98%
14Mapika · Decider 4B v2.14.08%
15Kev · 9B4.09%
16OpenDecider · Small TD4.21%
17Decision Lux · 9B4.36%
18Malkuth · 4B4.43%
19Winnow · E4B Q84.56%
20OpenDecider · Small4.58%
21Laya · Typed Decisions4.86%
22Jev4.99%
23OpenDecider · Nano5.00%
24JevK55.05%
25SPX-CD-Omni5.11%
26GPT 6 Luna Decisions5.24%
27Kev · 0.8B5.27%
28Nimble · v2 9B5.28%
29Kev ·4B API5.62%
30Kev · 4B5.63%
31JPT · 4B5.97%
32Ateve Jev6.09%
33Hopper · 4B6.23%
34Bocha Jev6.57%
35Lev · 4B6.78%
36Cloudflare Clef-flash7.90%
37Jet · v6.2 4B7.91%
38Jev-Omni · 12B8.08%
39Open-Jev · Zefan 9B8.62%
40Mica · 4B BF16 GGUF8.66%
41Winnow · 12B Q89.07%
42CalDec · Laya9.24%
43Hopper G · 4B v1.39.29%
44CalDec · GLiNER9.40%
45Cloudflare Clef9.40%
46Wity-1 · auto9.48%
47Bosun v3.1 · 0.6B10.67%
48Mercury Decide10.77%
49APUS OpenJev · 9B 3000 High12.68%
50Bosun v3.1 · 1.7B13.01%
51Laya · English14.42%
52APUS OpenJev · 4B 5949 High14.86%
53APUS OpenJev · 9B 5949 High15.45%
54Liquid AI · d1-omni-600M16.01%
55Open-Jev · Zefan 27B16.69%
56Laya · Multilingual19.74%
57FLock THIS/THAT30.09%
58CLM v0.1 · 8B31.16%
59Julia-140.55%

ECE uses 10 probability bins and valid choice and yes/no responses with native probabilities. Labels show the actual error. Text and image coverage varies by model.

I/C index leaderboard

0–100; higher is better ↑

View exact I/C results
Complete I/C results
RankModel / versionI ↑C ↑
01pplx-decider · v1.1 27B68.2981.27
02Microsoft-Decision-1 ·9B65.6983.84
03SPX-CD-Pro65.5583.22
04StartLux-Decision-27B · BF1663.8882.36
05Mercury Decide63.2573.80
06Quyet-1.0 · Large 31B63.0280.88
07Winnow · 12B Q861.4769.93
08StartLux-Decision-35B-A3B · BF1658.9481.34
09Cloudflare Clef58.5678.61
10JPT · 9B58.1880.48
11Wity-1 · auto56.3265.92
12Cloudflare Clef-flash56.2981.63
13pplx-decider 27B55.4979.01
14Plumb · v5.2 4B55.0975.19
15SPX-CD-Flash54.7082.81
16Jev53.4777.26
17Open-Jev · Zefan 9B53.3480.66
18APUS OpenJev · 4B 5949 High52.2458.81
19Imajev-9B52.1881.45
20APUS OpenJev · 9B 3000 High51.9164.61
21Malkuth · 4B51.1477.74
22JPT · 4B50.8278.54
23GPT 6 Luna Decisions50.5072.95
24Jev-Omni · 12B50.5060.94
25Jet · v6.2 4B50.4378.25
26FLock THIS/THAT49.3943.26
27APUS OpenJev · 9B 5949 High49.1855.21
28SPX-CD-Omni45.8379.02
29Ateve Jev45.0674.22
30JevK544.9677.16
31Hopper G · 4B v1.344.9273.12
32Hopper · 4B44.7477.20
33Kev ·4B API43.6478.89
34Kev · 4B42.8879.36
35Decision Lux · 9B42.1977.47
36Mapika · Decider 4B v2.142.0980.32
37Lev · 4B40.3970.04
38Kev · 9B40.3078.53
39Bocha Jev40.1271.95
40Nimble · v2 9B38.5082.47
41Mica · 4B BF16 GGUF37.5872.13
42OpenDecider · Small37.2880.11
43Winnow · E4B Q833.8376.34
44Open-Jev · Zefan 27B32.5075.21
45OpenDecider · Small TD32.3379.73
46Reflex · Stable 4B31.3475.34
47Liquid AI · d1-omni-600M28.7667.09
48Bosun v3.1 · 1.7B28.1165.82
49Laya · English23.3965.18
50Bosun v3.1 · 0.6B23.3269.70
51OpenDecider · Nano18.5775.58
52Laya · Typed Decisions17.7977.76
53Laya · Multilingual16.8352.01
54CalDec · Laya13.7477.75
55Kev · 0.8B12.6272.20
56Julia-111.4327.67
57CLM v0.1 · 8B5.3158.54
58CalDec · GLiNER0.0066.63
59GLiNER2.5-Decide0.0074.78

I · Intelligence: measures decision accuracy relative to random guessing.C · Calibration: measures how well predicted probabilities match the correct answers.

I/C uses the same 648 text questions for every model, applying the JevBench v1.5 formulas to CMDB results. ECE and I/C compare models that provide native candidate probabilities.

Add your model

Want your model evaluated on CMDB-1500?

Contact us ↗

CMDB-1500 combines language, image, and action selection tasks in four formats: single choice, yes/no, ordered scoring, and multi-select. The dataset contains 1,500 questions and 322 distinct source images.

What makes up the 1,500 questions?
1,500All questions

Select a category to see its share; select it again to reset.

Questions are sampled across domains, with answer options and source images preserved. Accuracy is calculated over 1,200 text questions, 300 image questions, or all 1,500 questions. Refusals and missing answers count as incorrect. Models evaluated only on text appear in the text leaderboard.

The tasks cover language understanding, business decisions, image understanding, tool use, and action selection.

Language, knowledge & judgment450 questions

Questions from JevBench test knowledge, intent recognition, evidence verification, language understanding, sentiment, and response quality.

ARC-Challenge · MMLU · Banking77 · CLINC150 · MASSIVE · BoolQ · MNLI · ChaosNLI · FEVER · PAWS · SST-5 · STS-B · StrategyQA · Civil Comments · Measuring Hate Speech · SMS Spam · HelpSteer2

Professional & business decisions300 questions

Questions from Atlan Decision Bench test tool selection, task completion checks, citation verification, SQL selection, and business routing.

The collection includes tasks adapted from BFCL, AgentDojo, τ-bench, MT-Bench, and Spider. Models choose an answer from the supplied context.

Visual decisions300 questions

Thirty questions from each of ten visual benchmarks test visual mathematics, image and text understanding, scientific diagrams, and fashion classification. Questions that require multiple images retain all source images.

MathVista · MMMU · MMMU-Pro · AI2D · ScienceQA · Fashion200k · GQA · VQAv2 · TextVQA · VizWiz

GQA, VQAv2, TextVQA, and VizWiz contribute yes/no question subsets.

Safety & adversarial tasks100 questions

Questions from Aegis, PhishNChips, and JevAdvBench test content safety, phishing email and URL detection, and decisions under adversarial perturbations.

Adversarial questions retain their source labels, some of which are not human annotations.

Game actions100 questions

Choose an action from a given game state. OpenJev provides scenarios from racing, Snake, tic-tac-toe, platformers, runner games, and ViZDoom.

Scores measure agreement with reference actions at fixed game states.

Multi-select decisions100 questions

Fifty questions each from MultiRC and GoEmotions ask models to select all correct reading-comprehension answers or identify multiple emotions in a text.

Embodied actions50 questions

ALFRED , ALFWorld, and VirtualHome provide 50 questions about choosing the next action from a task, observations, and action history.

Reference answers come from expert trajectories and evaluate individual action choices.

Chinese transcript selection50 questions

Select the transcript closest to the reference from ChineseHP AISHELL-4 candidates. Answers are determined by character edit distance.

Inputs contain transcript candidates, without audio.

Tool use25 questions

Adapted from BFCL v3, these questions ask whether a user request requires a tool call.

Spatial & complex rules25 questions

Nineteen complex-rule questions and six spatial questions from THIS-THAT test decisions under conflicting rules and spatial relationships. Reference answers come from rules or simulators.

Surd AI

SPX-CD · Native decision models

SPX-CD-Pro, Flash, and Omni return structured choices and candidate probabilities. All three are evaluated with effort=2.

SPX-CD-Flash, SPX-CD-Pro, and SPX-CD-Omni are open source on Hugging Face, with LoRA weights, inference code, and evaluation results.

TypeSafeStartLuxHugging Face

Other native decision models

Jev, StartLux, Mercury Decide, pplx-decider, JPT, Winnow, and Kev use decision or probability interfaces. The leaderboard distinguishes model sizes, versions, and quantization formats.

GPTClaudeGLM

General reasoning models

GPT, Claude, GLM, Qwen, Kimi, and DeepSeek are included in the accuracy comparison as general reasoning models.

SPX-CD-Pro · Overall #281.20%1,218 / 1,500 correct
SPX-CD-Flash · Overall #579.53%1,193 / 1,500 correct

Text and image accuracy are 79.33% / 88.67% for Pro and 77.58% / 87.33% for Flash.

When a model assigns 80% probability to a set of answers, are about 80% actually correct? ECE measures the gap between predicted confidence and observed accuracy; lower is better. Pooled ECE is 2.46% for Pro and 2.49% for Flash.

Explore CMDB-1500 on Hugging Face ↗