Devstral Small 1.1 vs Llama 4 Scout

Across our 12-test internal suite, Llama 4 Scout wins 3 benchmarks outright and ties the remaining 9. Devstral Small 1.1 wins none.

Long context (5 vs 4): Llama 4 Scout ties for 1st among 55 tested models at 5/5. Devstral Small 1.1 scores 4/5, landing at rank 38 of 55. For retrieval tasks at 30K+ tokens, this is a meaningful gap — and Llama 4 Scout's 327,680-token context window (vs Devstral Small 1.1's 131,072) reinforces this advantage structurally. If you're summarizing long documents, processing codebases, or doing multi-document analysis, Llama 4 Scout has the clear edge here.

Creative problem solving (3 vs 2): Llama 4 Scout scores 3/5 (rank 30 of 54), while Devstral Small 1.1 scores 2/5 (rank 47 of 54). Neither scores near the top of the field — the median across models is 4/5 — but Devstral Small 1.1's score puts it in the bottom eighth of tested models. This matters for tasks requiring novel, feasible ideas rather than pattern-matched responses.

Persona consistency (3 vs 2): Llama 4 Scout scores 3/5 (rank 45 of 53), Devstral Small 1.1 scores 2/5 (rank 51 of 53). Both are below the median of 5/5 on this dimension, but Devstral Small 1.1 is near the bottom. For chatbot or roleplay applications requiring stable character, neither model excels — but Llama 4 Scout is substantially less problematic.

Ties across 9 benchmarks: The two models are indistinguishable on structured output (4/5 each, rank 26 of 54), tool calling (4/5 each, rank 18 of 54), faithfulness (4/5 each, rank 34 of 55), classification (4/5 each, tied for 1st among 53 models), multilingual (4/5 each, rank 36 of 55), constrained rewriting (3/5 each, rank 31 of 53), strategic analysis (2/5 each, rank 44 of 54), agentic planning (2/5 each, rank 53 of 54), and safety calibration (2/5 each, rank 12 of 55).

The shared weak spots are notable: both models score 2/5 on agentic planning (near the bottom of our tested set at rank 53 of 54) and 2/5 on strategic analysis (rank 44 of 54). For complex agent pipelines requiring goal decomposition and failure recovery, neither model is a strong candidate based on our testing.

Both models charge $0.30/M output tokens, making output costs identical at any volume — $0.30 at 1M tokens, $3.00 at 10M, and $30.00 at 100M. The only pricing difference is on input: Devstral Small 1.1 costs $0.10/M input tokens vs Llama 4 Scout's $0.08/M. That $0.02/M gap is negligible at low volumes ($0.02 per 1M input tokens) but adds up at scale — at 100M input tokens per month, Devstral Small 1.1 costs $10 more in input fees. For most workloads, output tokens dominate total spend, so this difference is unlikely to drive a decision. Neither model sits at the premium end of the market — both are well below the $5/M input ceiling in our tracked range.

Choose Llama 4 Scout if you need a general-purpose AI for tasks involving long documents (its 327K context window and 5/5 long context score lead the field), image inputs (it supports text+image->text modality), or creative and conversational tasks where its higher persona consistency and creative problem solving scores matter. At identical output pricing, there is no cost reason to accept the performance gap.

Choose Devstral Small 1.1 if you are specifically building software engineering agents and the model's stated specialization for that domain is relevant to your stack — though note that our internal benchmark suite does not include a dedicated coding task, and both models score similarly on the closest proxies (tool calling 4/5, structured output 4/5). Its smaller context window (131K vs 327K) and text-only modality are real constraints to weigh. At $0.10/M input vs $0.08/M, it also costs slightly more on the input side for no demonstrated benchmark advantage.