Devstral Small 1.1 vs GPT-4.1

Summary of our 12-test comparison (scores from our testing):

• GPT-4.1 wins (9): strategic_analysis 5 vs 2, constrained_rewriting 5 vs 3, creative_problem_solving 3 vs 2, tool_calling 5 vs 4, faithfulness 5 vs 4, long_context 5 vs 4, persona_consistency 5 vs 2, agentic_planning 4 vs 2, multilingual 5 vs 4. These wins include top-tier ranks: GPT-4.1 is tied for 1st on faithfulness, long_context, persona_consistency, classification, strategic_analysis, constrained_rewriting, and tool_calling — i.e., it sits among the best performers in our pool for tasks requiring accuracy, maintaining character, and retrieval at 30K+ tokens.
• Devstral Small 1.1 wins (1): safety_calibration 2 vs GPT-4.1’s 1 — Devstral ranks 12 of 55 on safety_calibration in our tests while GPT-4.1 ranks 32 of 55. That means Devstral was more likely in our tests to correctly refuse harmful requests while allowing legitimate ones.
• Ties (2): structured_output 4/4 (both rank ~26/54) and classification 4/4 (both tied for 1st with many models). For JSON/schema adherence and routing tasks, both models perform equivalently in our suite.
• Rankings context: Devstral’s low ranks (e.g., persona_consistency rank 51 of 53, agentic_planning rank 53 of 54) indicate it struggles to maintain persona and to decompose goals compared with the field. GPT-4.1’s top ranks on long_context (tied for 1st) and faithfulness (tied for 1st) imply better behavior on long-document retrieval and sticking to source material.
• External benchmarks (Epoch AI): GPT-4.1 scores 48.5% on SWE-bench Verified, 83% on MATH Level 5, and 38.3% on AIME 2025 (these external numbers are from Epoch AI and supplement our internal results). These values show GPT-4.1’s relative standing on third-party coding and math tests but do not change our internal ranking procedure. Practical meaning: pick GPT-4.1 when you need reliable long-context answers, high faithfulness, complex planning, or multilingual parity. Pick Devstral Small 1.1 when you must minimize per-token cost and can accept weaker strategic analysis, persona maintenance, and planning.

Prices (per mTok): Devstral Small 1.1 = $0.10 input / $0.30 output; GPT-4.1 = $2 input / $8 output. Assuming a 50/50 split of input/output tokens: for 1M total tokens/month (1,000 mTok) Devstral ≈ $200, GPT-4.1 ≈ $5,000. At 10M tokens: Devstral ≈ $2,000, GPT-4.1 ≈ $50,000. At 100M tokens: Devstral ≈ $20,000, GPT-4.1 ≈ $500,000. The cost gap matters most for high-volume products (APIs, consumer apps, automation pipelines) where GPT-4.1’s extra capabilities must justify 25x–30x higher monthly bills; small teams, prototypes, and cost-sensitive deployments will favor Devstral Small 1.1.

Choose Devstral Small 1.1 if: you need a cost-efficient model for high-volume text tasks where structured output and basic classification suffice, and you can accept weaker scores on strategic analysis, persona consistency, tool calling, and long-context retrieval. Choose GPT-4.1 if: you need the highest faithfulness, 1M+ token context work, stronger tool calling/agentic planning, persona consistency, or robust multilingual and constrained rewriting capabilities and you can justify the much higher per-token cost.