Codestral 2508 vs GPT-4o-mini

Overview: In our 12-test suite Codestral 2508 wins 5 tests, GPT-4o-mini wins 3, and 4 tests tie. Below we walk each test with scores, ranks, and what that means for real tasks.

Wins for Codestral 2508 (scores: Codestral → GPT-4o-mini):

• structured_output: 5 → 4. Codestral is tied for 1st with 24 others out of 54 tested, indicating stronger JSON/schema compliance for tasks that require exact format adherence (APIs, code scaffolding, test-case outputs).
• tool_calling: 5 → 4. Codestral tied for 1st with 16 others out of 54, so it more reliably selects functions and arguments — important for orchestrating toolchains and FIM-style code edits.
• faithfulness: 5 → 3. Codestral is tied for 1st with 32 others out of 55, meaning it sticks to source material more tightly (useful for code transformation and documentation extraction).
• long_context: 5 → 4. Codestral tied for 1st with 36 others out of 55, and combined with its 256k context window (payload) it is better suited for very long repo or spec contexts.
• agentic_planning: 4 → 3. Codestral ranks 16 of 54 (tied) versus GPT-4o-mini lower; it better decomposes tasks and plans failure recovery in our tests.

Wins for GPT-4o-mini:

• classification: 3 → 4. GPT-4o-mini is tied for 1st with 29 other models out of 53, so it routes and categorizes content more accurately in our tests (helpful for triage, intent detection).
• safety_calibration: 1 → 4. This is a large gap: GPT-4o-mini ranks 6 of 55 (tied with 3 other models) while Codestral ranks 32 of 55. GPT-4o-mini more reliably refuses harmful requests and permits legitimate ones in our testing — critical for public-facing assistants.
• persona_consistency: 3 → 4. GPT-4o-mini maintains character better (rank 38 of 53 vs Codestral rank 45), useful for chatbots and branded UX.

Ties (both models scored the same in our suite): strategic_analysis (2), constrained_rewriting (3), creative_problem_solving (2), multilingual (4). That means both models behave similarly on nuanced tradeoffs, tight character-limit rewriting, idea-generation difficulty, and non-English quality in our tests.

External math benchmarks (supplementary): GPT-4o-mini also has external math scores in the payload: MATH Level 5 = 52.6% and AIME 2025 = 6.9% (Epoch AI). Codestral has no external math scores in the payload. These external measures suggest GPT-4o-mini's relative performance on those math datasets but are separate from our 1–5 internal test suite.

Implications: For code generation, tool orchestration, long-repo context and strict JSON outputs, Codestral shows measurable advantages in our tests. For safety-sensitive deployments, classification-heavy flows, persona-based chat, and multimodal inputs (GPT-4o-mini supports text+image+file→text), GPT-4o-mini is the safer, cheaper choice in our benchmarked comparisons.

Raw unit prices from the payload: Codestral 2508 charges $0.30/mTok input and $0.90/mTok output; GPT-4o-mini charges $0.15/mTok input and $0.60/mTok output. Price ratio = 1.5 (payload). Practical examples (1 mTok = 1,000 tokens):

• Output-only scenario (all tokens billed as output): Codestral = $0.90/mTok → $900 per 1M tokens; GPT-4o-mini = $0.60/mTok → $600 per 1M tokens. Difference = $300/1M.
• Input-only scenario: Codestral = $0.30/mTok → $300 per 1M; GPT-4o-mini = $0.15/mTok → $150 per 1M. Difference = $150/1M.
• Even split (50% input, 50% output): Codestral = $0.60/mTok → $600 per 1M; GPT-4o-mini = $0.375/mTok → $375 per 1M. Difference = $225/1M. Scale the above linearly: for 10M tokens multiply by 10, for 100M tokens multiply by 100 (e.g., output-only: Codestral $9,000/10M vs GPT $6,000/10M; output-only 100M: Codestral $90,000 vs GPT $60,000). Who should care: high-volume deployments, SaaS products, or pipelines processing millions of tokens monthly — the $225–$300/1M gap quickly becomes tens of thousands of dollars at scale. Low-volume or multimodal apps may prefer GPT-4o-mini for lower cost and image/file inputs.

Choose Codestral 2508 if: you build code-first or tool-driven systems (Codestral's description highlights FIM, code correction, and test generation) and you need top structured output, tool calling, faithfulness and extreme long-context (256k). Expect to pay ~1.5× for those gains. Choose GPT-4o-mini if: you need lower cost, multimodal inputs (text+image+file→text), stronger safety calibration and classification, or you run very high token volumes where the $225–$300/1M-token gap matters.