Devstral Medium vs GPT-4.1 Nano

Across our 12-test internal benchmark suite, GPT-4.1 Nano wins 6 tests, Devstral Medium wins 1, and they tie on 5.

GPT-4.1 Nano's wins:

• Structured output (5 vs 4): GPT-4.1 Nano ties for 1st among 54 models; Devstral Medium sits at rank 26. For JSON schema compliance and API integrations, this is a meaningful gap.
• Faithfulness (5 vs 4): GPT-4.1 Nano ties for 1st among 55 models; Devstral Medium ranks 34th. In RAG and summarization tasks, GPT-4.1 Nano is less likely to hallucinate beyond the source material.
• Tool calling (4 vs 3): GPT-4.1 Nano ranks 18th of 54; Devstral Medium ranks 47th — near the bottom. For function-calling pipelines and agentic tools, this is a significant disadvantage for Devstral Medium.
• Constrained rewriting (4 vs 3): GPT-4.1 Nano ranks 6th of 53; Devstral Medium ranks 31st. Better compression within hard character limits matters for content workflows.
• Safety calibration (2 vs 1): GPT-4.1 Nano ranks 12th of 55; Devstral Medium ranks 32nd. Devstral Medium scores 1/5 here — in the bottom quartile for the entire 55-model field (p25 = 1). This is a concern for consumer-facing applications.
• Persona consistency (4 vs 3): GPT-4.1 Nano ranks 38th of 53; Devstral Medium ranks 45th. Neither excels here, but GPT-4.1 Nano is a step ahead.

Devstral Medium's win:

• Classification (4 vs 3): Devstral Medium ties for 1st among 53 models; GPT-4.1 Nano ranks 31st. This is a genuine strength — accurate categorization and routing is a real use case where Devstral Medium has a clear edge.

Ties (both models score equally):

• Strategic analysis (2/2), creative problem solving (2/2), long context (4/4), agentic planning (4/4), and multilingual (4/4) are dead heats. Both models sit at rank 44, 47, 38, 16, and 36 respectively on these tests — neither distinguishes itself.

External benchmarks (Epoch AI): GPT-4.1 Nano has third-party math scores in the payload: 70% on MATH Level 5 (rank 11 of 14 tested models) and 28.9% on AIME 2025 (rank 20 of 23 tested models). These place GPT-4.1 Nano in the lower tier of math-capable models evaluated externally — useful context if math reasoning is a priority. Devstral Medium has no external benchmark scores in the payload.

GPT-4.1 Nano costs $0.10/MTok input and $0.40/MTok output. Devstral Medium costs $0.40/MTok input and $2.00/MTok output — 4x more on input and 5x more on output. At 1M output tokens/month, that's $400 vs $2,000: a $1,600 difference. At 10M output tokens, the gap widens to $16,000/month ($4,000 vs $20,000). At 100M output tokens — a realistic scale for high-volume classification, RAG pipelines, or chatbots — you're looking at $40,000 vs $200,000 annually. For any workload where Devstral Medium doesn't win decisively on benchmarks (and our testing shows it wins only one), the cost premium is very hard to justify. The exception would be teams that specifically need agentic code workflows where Devstral Medium's description indicates a specialized focus, but our benchmark data doesn't currently show a scoring advantage to support that premium.

Choose GPT-4.1 Nano if you need structured JSON output, faithful summarization, reliable tool calling, or safe consumer-facing deployment — it wins all four in our testing, ranks near the top of the field on structured output and faithfulness, and costs 5x less on output. At any volume above 1M tokens/month, the savings compound fast. It also supports image and file inputs, giving it a broader modality footprint and a 1M+ token context window.

Choose Devstral Medium if classification accuracy is your primary bottleneck. It ties for 1st among 53 models on categorization and routing in our tests, versus GPT-4.1 Nano's rank 31. If you're building a high-accuracy document routing or triage system and classification is the single most important dimension, Devstral Medium's edge is real — though you'll pay a 5x output cost premium for it. Also consider it if Mistral's infrastructure or data residency requirements are relevant to your deployment.