Grok 3 vs Grok 4

We ran a 12-test suite and compared scores and ranks from our testing. Summary of wins/ties: Grok 3 wins structured output and agentic planning; Grok 4 wins constrained rewriting; the remaining nine tests tie. Details and implications:

• structured output: Grok 3 = 5 vs Grok 4 = 4. Grok 3 is tied for 1st ("tied for 1st with 24 other models out of 54 tested"). Practical impact: Grok 3 is better at JSON/schema compliance and exact response formats — useful for data extraction, API responses, and systems that must parse strict schemas.

• agentic planning: Grok 3 = 5 vs Grok 4 = 3. Grok 3 ranks "tied for 1st with 14 other models out of 54 tested" while Grok 4 ranks 42 of 54 ("rank 42 of 54 (11 models share this score)"). Practical impact: Grok 3 is stronger for goal decomposition, fallback/failure recovery, and multi-step planning.

• constrained rewriting: Grok 3 = 3 vs Grok 4 = 4. Grok 4 ranks 6 of 53 ("rank 6 of 53 (25 models share this score)") versus Grok 3 at rank 31. Practical impact: Grok 4 is better at tight compression and rewriting to strict character limits (SMS, microcopy, embedded UI text).

• strategic analysis: both = 5 and tied for 1st (Grok 3 display: "tied for 1st with 25 other models out of 54 tested"). Both are good at nuanced tradeoff reasoning.

• creative problem solving: both = 3, rank 30 of 54. Expect similar output for non-obvious ideation prompts.

• tool calling: both = 4, rank 18 of 54 (tied). Both select functions and arguments at similar quality in our tests.

• faithfulness: both = 5 and tied for 1st. Both stick closely to source material in our tests.

• classification: both = 4 and tied for 1st with many models; both are reliable for routing and categorization tasks.

• long context: both = 5 and tied for 1st; both perform well on retrieval at 30k+ tokens, though Grok 4’s payload context window is larger (256k vs 131,072).

• safety calibration: both = 2 and rank 12 of 55 (tied). Both models have the same calibration score in our tests.

• persona consistency and multilingual: both = 5 and tied for 1st (equivalent outputs across personas and languages in our tests).

Context-window and modality differences from the payload: Grok 3 context_window = 131,072; Grok 4 context_window = 256,000 and supports text+image+file->text with reasoning-token quirks. Those capabilities explain why Grok 4 wins constrained rewriting and is valuable when images or extremely long context are required.

Both models share the same pricing in the payload: input_cost_per_mtok = $3 and output_cost_per_mtok = $15. Interpreting 'per_mtok' as the payload unit of 1k-token blocks, combined cost is $18 per 1k tokens. That implies: 1M tokens ≈ 1,000 × $18 = $18,000/month; 10M ≈ $180,000/month; 100M ≈ $1,800,000/month. Because Grok 3 and Grok 4 have identical input/output rates in the payload, cost is not a differentiator — the price burden matters most to high-volume (multi-million-token) customers and teams evaluating model selection vs. hosting/engineering tradeoffs.

Choose Grok 3 if you need: reliable structured outputs (JSON/schema compliance), stronger agentic planning and workflow decomposition, or prioritise extraction and predictable API responses — it scores 5 vs 4 on structured output and 5 vs 3 on agentic planning in our tests. Choose Grok 4 if you need: image inputs, the larger 256k context window, or better constrained rewriting for tight character/size limits (Grok 4 scores 4 vs 3 on constrained rewriting). Pricing is identical in the payload, so choose on capability and context requirements, not cost.