--- title: "DeepSeek V4.1-Flash-1017: Vendor Says 42.3 on Terminal-Bench 4.0, Vals.ai Measures 31.8" date: 2026-10-25 tags: ["deepseek","open-weight","benchmarks","terminal-bench","models"] categories: ["AI Tools"] summary: "DeepSeek's V4.1-Flash-1017 checkpoint (Oct 17, MIT) claims 77.0 on DeepSWE v1.1 and 42.3 on Terminal-Bench 4.0. Vals.ai's independent run puts Terminal-Bench 4.0 at 31.82%, rank 19 of 55. Parameter counts also disagree across sources (552B vs 763B)." --- ![DeepSeek V4.1-Flash-1017: Vendor Says 42.3 on Terminal-Bench 4.0, Vals.ai Measures 31.8](/images/deepseek-v4-1-flash-1017-vendor-vs-vals-numbers.png) When a lab ships a "minor update" under the same API endpoint and the same price, the interesting question is whether anyone outside the lab can reproduce the gains. For DeepSeek V4.1-Flash-1017, released October 17, we now have a first answer, and it is not flattering on the benchmark that matters most for agentic coding. ## What shipped DeepSeek released [V4.1-Flash](https://thenextweb.com/news/deepseek-v4-1-flash-launch-v4-pro-retired-price-cut) on September 10 as an MIT-licensed open-weight model with weights on Hugging Face. TNW reports a 552B-parameter model with roughly 8B active parameters on input and 16B on output, a 1M-token context window and native image input. It also reports that from September 14, V4-Pro requests were routed to V4.1-Flash at Flash prices until a V4.1-Pro appears. Off-peak output was listed at $0.60 per million tokens. The 1017 checkpoint swaps in new weights. [AIWeekly's write-up](https://aiweekly.co/fr/alerts/deepseek-v41-flash-1017-763b-sous-mit-426-sur-hle-sans-outils) says the model card calls it a minor update and cites improvements in reasoning, coding, agentic and visual benchmarks. The same piece gives the license as MIT and the model size as 763B. That is the first discrepancy. TNW says 552B for the September model; AIWeekly says 763B for the October one. Either the architecture changed more than "minor" suggests, or one of the sources is wrong. I could not read the Hugging Face model card, so I'm not going to pick a side. If you plan to self-host, check the card before sizing GPUs. ## The vendor numbers Per AIWeekly's transcription of the model card (all vendor-reported): | Benchmark | Previous | 1017 | Comparison in the card | |---|---|---|---| | DeepSWE v1.1 | 74.2 (TNW, Sept) | 77.0 | Opus 5: 74.0, GPT-5.6 Sol: 73.0 | | Terminal-Bench 4.0 | 31.2 | 42.3 | not given | | HLE, no tools | 36.8 | 42.6 | Opus 5: 56.3, GPT-5.6 Sol: 44.5 | | Agents' Last Exam | n/a | 35.4 | Opus 5: 28.6 | | Automation-Bench v1.0.6 | n/a | 60.5 | Opus 5: 50.3 | Read generously, that is an open model leading both US frontier models on the agentic tests DeepSeek chose to list, while trailing on HLE. Read carefully, notice who chose the table, who ran the harness, and which flagship rows are missing. Our earlier coverage of [Terminal-Bench 4.0](/terminal-bench-4-0-replaces-unverified-3-0/) is the context for why that last point matters. ## The independent number [Vals.ai lists the 1017 checkpoint](https://www.vals.ai/models/deepseek_deepseek-v4.1-flash-1017) with a release date of October 17 and token prices of $0.30 input / $1.20 output. Its measurements: - **Vals Index:** 61.26% (±1.05), rank 9 of 27 - **Terminal-Bench 4.0:** 31.82% (±2.62), rank 19 of 55 - **Vibe Code Bench v1.1:** 87.73% (±2.75), rank 11 of 119 - **Code Migration v1.1:** 67.52% (±2.76), rank 7 of 28 - **Tax Agent Bench:** 44.35%, rank 6 of 76 The Terminal-Bench gap is the headline: 42.3 claimed against 31.82 measured, a difference of about 10 points and far outside Vals's stated margin. Notably, 31.82 is close to the *previous* checkpoint's vendor-reported 31.2. That coincidence is suggestive, not proof. Harness configuration, tool budgets, timeouts and sampling settings all move Terminal-Bench scores, and Vals may simply run a different setup than DeepSeek. But it means the claimed jump has not been reproduced here. The other Vals results are respectable. Ninth of 27 on a broad index, at those prices, is a strong value proposition. This is a good model; the open question is only how good on terminal-style agent work. ## What this means for your stack **Price-performance is real.** Even on the independent numbers, a model in the top ten of a broad index at $0.30/$1.20 is hard to ignore for high-volume, low-stakes work: bulk code migration, test generation, first-pass triage. **Don't route autonomous work by the vendor table.** Terminal-Bench is a proxy for long-horizon shell-driven tasks, which is exactly what unattended agents do. A 10-point shortfall between claim and measurement is the difference between an agent that finishes and one that stalls. Run your own eval on your repos before switching an agent workflow to any model, open or closed. **Self-hosting needs the real spec.** With sources disagreeing on size (552B vs 763B) and weight formats listed as BF16, F32, F8_E4M3 and I8, budget from the model card, not from a news article. **Spec-driven workflows make this testable.** If your acceptance criteria live in specs and executable tests, swapping the model behind an agent is an experiment you can score in an afternoon. If they live in someone's head, you are choosing by leaderboard. ## Caveats This article is built entirely from secondary sources plus Vals.ai's public listing. I did not run the model, did not read the model card directly, and the AIWeekly transcription of vendor numbers is unverified against the original. TNW noted for the September release that DeepSeek's figures were not independently verified, and the same holds here for everything except the Vals rows. DeepSeek's technical report, per TNW, also describes reward hacking during training, with agents exploiting vulnerabilities and deleting files in test environments. That is worth remembering when an agent benchmark score is the only evidence offered. ## The takeaway DeepSeek keeps compressing the price of capable coding models, and the open-weight MIT license makes that pressure real. But the gap between its own Terminal-Bench figure and Vals.ai's is a reminder of why benchmarks need independent harnesses. Treat 1017 as a strong cheap model with an unproven agentic ceiling, and let your own evals decide. ## Sources - [TNW: DeepSeek launches V4.1-Flash and retires V4-Pro](https://thenextweb.com/news/deepseek-v4-1-flash-launch-v4-pro-retired-price-cut) - [AIWeekly: DeepSeek V4.1-Flash-1017](https://aiweekly.co/fr/alerts/deepseek-v41-flash-1017-763b-sous-mit-426-sur-hle-sans-outils) - [Vals.ai: DeepSeek V4.1 Flash 1017](https://www.vals.ai/models/deepseek_deepseek-v4.1-flash-1017)