Devstral Small 1.1 vs Grok 4

Summary of our 12-test suite (scores shown as Devstral Small 1.1 vs Grok 4):

• strategic_analysis: 2 vs 5 — Grok wins and is tied for 1st on the ranking (tied with 25 others). This matters for tasks that require nuanced tradeoff reasoning with real numbers. Devstral's 2 (rank 44/54) signals weaker performance on those analyses.
• constrained_rewriting: 3 vs 4 — Grok wins (rank 6/53). Use Grok for tight character-limit compression; Devstral performs worse (rank 31).
• creative_problem_solving: 2 vs 3 — Grok wins (rank 30/54). Grok generates more non‑obvious, feasible ideas in our tests.
• faithfulness: 4 vs 5 — Grok wins and is tied for 1st (rank 1/55). For tasks that must stick to sources and avoid hallucination, Grok is measurably better.
• long_context: 4 vs 5 — Grok wins and is tied for 1st on long-context (rank 1/55). This aligns with its 256k context window versus Devstral's 131k; Grok is stronger on retrieval at 30K+ tokens.
• persona_consistency: 2 vs 5 — Grok wins (tied for 1st). In our tests Grok better resists persona injection and maintains character.
• agentic_planning: 2 vs 3 — Grok wins (rank 42/54). Grok shows stronger decomposition and failure-recovery ability in our suite.
• multilingual: 4 vs 5 — Grok wins and ties for 1st (rank 1/55). Grok is best for equivalent-quality non‑English output in our tests. Ties (no clear winner in our tests): structured_output 4/4 (both rank 26), tool_calling 4/4 (both rank 18), classification 4/4 (both tie for 1st), safety_calibration 2/2 (both rank 12). Tool calling parity means both models select functions and arguments similarly in our tests; classification parity means both reach top scores on routing/categorization. Safety calibration is low for both (score 2), so neither is a standout for safety-sensitive refusal/permissiveness. What these scores mean for real tasks: Grok 4 is objectively stronger on high-demand tasks — strategy, faithfulness, long-context retrieval, multilingual output and persona stability — and also offers multimodal inputs per the payload. Devstral Small 1.1 is competent on classification, structured output, and standard tool calling but trails on strategy/faithfulness and persona. The rank positions show Grok often sits among the top performers (multiple rank 1 ties), while Devstral ranks in the lower half for several high-value dimensions (agentic_planning rank 53, persona_consistency rank 51).

Costs are listed per mTok in the payload (per 1,000 tokens). We assume a 50/50 split of input/output tokens to illustrate real usage. Devstral Small 1.1: input $0.1 + output $0.3 per mTok. At 1M tokens (1,000 mToks) that equals $200/month (500 mToks input = $50; 500 mToks output = $150). At 10M tokens it's $2,000/month; at 100M tokens $20,000/month. Grok 4: input $3 + output $15 per mTok. With the same 50/50 split, 1M tokens = $9,000/month (500 mToks input = $1,500; 500 mToks output = $7,500); 10M = $90,000; 100M = $900,000. The payload's priceRatio = 0.02 reflects this gap: Devstral runs at ~2% of Grok's sticker cost in this comparison. High-volume consumers (teams sending millions of tokens/month) and cost-sensitive startups should care most; only teams that need Grok 4's higher scores and multimodal/256k context features should justify the much higher spend.

Choose Devstral Small 1.1 if: you must minimize inference cost (runs ≈ $200/mo at 1M tokens with a 50/50 I/O split), your workloads are text-only, and you need solid classification, structured output, or standard tool calling at low price. Choose Grok 4 if: you need top-tier long-context retrieval, strict faithfulness, stronger strategic reasoning, multilingual parity, or multimodal inputs (image/file support) and can accept the much higher cost (≈ $9,000/mo at 1M tokens with the same I/O mix).