Devstral Small 1.1 vs Gemini 2.5 Pro

Summary of our 12-test head-to-head (scores are our 1–5 internal tests; rankings reference the tested pool):

• Gemini wins (9 tests): structured_output 5 vs 4 (Gemini tied for 1st of 54), long_context 5 vs 4 (Gemini tied for 1st of 55), faithfulness 5 vs 4 (Gemini tied for 1st of 55), tool_calling 5 vs 4 (Gemini tied for 1st of 54), creative_problem_solving 5 vs 2 (Gemini tied for 1st of 54), strategic_analysis 4 vs 2 (Gemini ranks 27 of 54), persona_consistency 5 vs 2 (Gemini tied for 1st of 53; Devstral ranks 51 of 53), agentic_planning 4 vs 2 (Gemini rank 16 of 54; Devstral rank 53 of 54), multilingual 5 vs 4 (Gemini tied for 1st of 55). Practical meaning: Gemini’s higher scores and top ranks for long_context and tool_calling indicate it will better handle retrieval over 30k+ token contexts and produce correct function selection/arguments in agent workflows. Its faithfulness and creative problem solving scores imply fewer hallucinations and more useful brainstorming for hard problems.
• Devstral wins (1 test): safety_calibration 2 vs 1 (Devstral ranks 12 of 55, tied with 19 others). This means Devstral more often refuses harmful prompts appropriately in our tests.
• Ties (2 tests): classification 4 vs 4 (both tied for 1st with many models), constrained_rewriting 3 vs 3 (both rank ~31). So for straightforward categorization both models perform equally well in our suite.
• External benchmarks for Gemini (supplementary): on SWE-bench Verified (Epoch AI), Gemini scores 57.6%; on AIME 2025 (Epoch AI), Gemini scores 84.2%. We cite Epoch AI for those numbers. These external results align with Gemini’s strengths on coding and math-heavy reasoning tasks in our internal suite.

Price per million tokens (as provided): Devstral Small 1.1: $0.10 input / $0.30 output. Gemini 2.5 Pro: $1.25 input / $10.00 output. If you assume a 50/50 split of input vs output tokens per 1,000,000 total tokens, cost per 1M tokens is: Devstral ≈ $0.20 (0.5*$0.10 + 0.5*$0.30); Gemini ≈ $5.625 (0.5*$1.25 + 0.5*$10.00). Scaling to monthly volumes with a 50/50 split: 1M tokens → Devstral $0.20 vs Gemini $5.63; 10M → Devstral $2.00 vs Gemini $56.25; 100M → Devstral $20.00 vs Gemini $562.50. Who should care: high-throughput services (chatbots, background indexing, analytics pipelines) will see large absolute savings with Devstral; research or mission-critical apps that require Gemini’s higher benchmark performance may justify the higher spend.

Choose Devstral Small 1.1 if: you need the lowest possible inference cost at scale (Devstral costs $0.10 input / $0.30 output per million tokens), you run very high token volumes (10M–100M+/month), or your product prioritizes stricter safety refusals over top-tier long-context reasoning. Choose Gemini 2.5 Pro if: you need best-in-class long-context retrieval, reliable tool calling, higher faithfulness and creative problem solving (Gemini wins 9 of 12 tests and ranks tied for 1st on long_context, tool_calling, and faithfulness), and you can absorb higher cost ($1.25/$10 per million tokens) for higher-quality outputs.