Quality Check
Public benchmarks, run through our own API
We sent public benchmark questions through the same endpoint our customers use. Then we compared the results with the published scores for the same models.
Run date: October 28, 2026. Models: DeepSeek V4 Flash, GLM-5.3 Flash.
MMLU-Pro
Multiple-choice questions, 0-shot chain-of-thought. We used a stratified sample across the 14 subjects, in the same order for every model.
| Model | Questions | Our score [95% CI] | Published | Published by | Verdict |
|---|---|---|---|---|---|
| DeepSeek V4 Flash, default (no reasoning) | 226 | 84.5% [79.2–88.7] | 83.0% | DeepSeek model card, preview version, non-thinking | Within the margin |
| DeepSeek V4 Flash, reasoning_effort=high | 40 | 87.5% [73.9–94.5] | 86.4% | DeepSeek model card, preview version, Think High | Within the margin |
| GLM-5.3 Flash | 224 | 85.3% [80.0–89.3] | 86.1% | Vals AI, independent | Within the margin |
Vals AI also reports 86.2% for the same DeepSeek V4 Flash build (0731), in reasoning mode. Our default-mode score is within the margin of that figure too.
- [95% CI] is the range where the true score most likely falls, given the sample size. With about 225 questions it is roughly ±5 points; with 40 questions, roughly ±10.
- At these sample sizes, differences smaller than about 5 points cannot be seen.
Other benchmarks
No published reference is available for these runs, so there is no verdict. The numbers are shown as measured.
| Benchmark | Model | Our score | Answered |
|---|---|---|---|
| IFEval, prompt-level strict | DeepSeek V4 Flash | 81.6% [72.8–88.0] | 98 of 98 |
| IFEval, prompt-level strict | GLM-5.3 Flash | 84.7% [76.3–90.5] | 98 of 100 |
| MATH-500, string match / Math-Verify | DeepSeek V4 Flash | 90.0% / 94.0% | 50 of 50 |
| MATH-500, string match / Math-Verify | GLM-5.3 Flash | 93.8% / 97.9% | 48 of 48 |
- Two GLM-5.3 Flash IFEval requests got no answer. Counting them as failures gives 83.0%.
- The two MATH-500 checks differ in answer formatting, not in the math.
Long context
A sentence with a planted fact sits at 50% depth of a long book. The question at the end asks the model to find it.
| Model | 100K tokens | 300K tokens |
|---|---|---|
| DeepSeek V4 Flash | Found | Found |
| GLM-5.3 Flash | Found | Found |
One trial per size. This shows that long prompts are accepted and read; it is too small a sample to give a score.
What this shows
- No large drop. On MMLU-Pro, neither DeepSeek nor GLM falls 6 to 8 points or more below the published scores.
- No sign of a swapped model or of long context getting cut off.
What it does not show
- That our scores match the vendor's own API. Gaps of a few points cannot be ruled out with samples this size.
- Speed, capacity, or uptime. This check covers answer quality only.
Method
- Endpoint: the same OpenAI-compatible chat completions API our customers call, in non-streaming mode.
- Sampling: each vendor's recommended settings. Temperature 1.0 for both models; top_p 1.0 for DeepSeek and 0.95 for GLM. One answer per question.
- Maximum output: 16,384 tokens. An answer cut off at that limit counts as wrong.
- MMLU-Pro: the official TIGER-Lab chain-of-thought prompt and answer extraction, run 0-shot instead of the official 5-shot.
- IFEval: the official checker from lm-evaluation-harness 0.4.13. MATH-500: string match, and Math-Verify.
- Confidence intervals: Wilson, 95%. Datasets: TIGER-Lab/MMLU-Pro, google/IFEval and HuggingFaceH4/MATH-500, pinned to fixed revisions.
Good to know
- DeepSeek V4 Flash answers without reasoning by default. Setting reasoning_effort to low, high or max turns it on. On MMLU-Pro, high used about 2,500 tokens per answer, against about 200 by default.
- GLM-5.3 Flash always reasons. The switches we tested did not turn that off.
Known issue: some short answers do not stop
On very short answers, generation sometimes runs on until the max_tokens limit. In our probes, DeepSeek V4 Flash did this in 12 of 12 requests at temperature 0, and in 3 of 12 at temperature 1.0. GLM-5.3 Flash did it in 3 of 12 at temperature 1.0.
Workaround: for one-word answers, set a small max_tokens and a stop sequence.
We do not have a fix date yet.
Raw data and the test harness are available on request. Email andrii@trustedrails.com or book a call.
Book a call
