The Field That Changed Faster Than Anyone Could Track
The AI model landscape has evolved so rapidly that anyone who formed strong opinions about which model was best in 2023 is operating on outdated information. The relative rankings of GPT-4, Claude, Gemini, and their successors have shifted with each major release; capabilities that were frontier features in one release became standard across models in the next; and the price-performance frontier has moved substantially as competition has intensified.
The question ‘which AI model is best’ doesn’t have a stable answer that applies across all tasks. Different models have different strengths, different context window sizes, different pricing structures, different safety profiles, and different API characteristics that make them more or less appropriate for specific use cases. The framework for choosing between models is more useful than any specific current ranking.
The Major Providers and Their Positioning
OpenAI (GPT-4o, GPT-4o mini, o1, o3 series) occupies the largest market share and the most established developer ecosystem. The o1 and o3 models introduced ‘reasoning’ capability — extended chain-of-thought processing that produces higher accuracy on complex mathematical, logical, and coding problems at the cost of higher latency and cost. GPT-4o mini provides a cost-effective option for applications that don’t require frontier capability. OpenAI’s API is the most mature in terms of tooling, documentation, and third-party integration support.
Anthropic (Claude 3.5 Sonnet, Claude 3.5 Haiku, Claude 3 Opus and successors) has built a reputation for producing models that excel at long-form writing quality, following nuanced instructions, and producing outputs that require careful attention to context and tone. Claude’s extended context window (200K tokens in Claude 3 Opus, with Claude 3.5 models matching or exceeding this) supports analysis of very long documents. Google (Gemini Ultra, Gemini Pro, Gemini Flash, Gemini Nano) provides the broadest range from on-device to frontier, with native multimodal capability across all sizes.
The Open Model Tier
Meta’s Llama 3 and successors, Mistral’s models, Microsoft’s Phi series, and dozens of community fine-tunes have produced open-weight models capable enough for a wide range of production applications. The Llama 3 70B model runs on consumer hardware with sufficient VRAM and produces outputs that compare favourably to frontier models from 12 months prior on many general tasks. For applications where data privacy, cost at scale, or customisation requirements make proprietary API models inappropriate, the open model ecosystem provides genuine production-ready alternatives.
The performance gap between the best open models and frontier proprietary models has been narrowing consistently: tasks where frontier models were clearly superior in early 2024 were being matched by open models by late 2024 at a fraction of the inference cost. For organisations willing to manage the infrastructure of running open models, the economics are compelling at the volumes where API costs become significant.
Choosing a Model for a Specific Use Case
The dimensions that matter for model selection: task type (coding, writing, analysis, reasoning, multimodal), required context length (short queries vs. long document analysis), latency requirements (real-time applications vs. batch processing), cost constraints (API pricing varies by an order of magnitude between model tiers), privacy requirements (cloud API vs. on-device or self-hosted), and the need for structured output (some models handle JSON schema constraints better than others).
For most general productivity and writing tasks, the mid-tier models (GPT-4o mini, Claude 3.5 Haiku, Gemini Flash) provide adequate quality at significantly lower cost than frontier models. Reserving frontier model access for the tasks that genuinely require it — complex reasoning, nuanced writing that requires extensive revision, long-document analysis — and using cost-efficient models for everything else is the approach that produces both good results and sustainable API budgets.
The Benchmark Scepticism Worth Developing
AI model benchmarks (MMLU, HumanEval, MATH, GPQA) provide useful comparative data that’s also consistently gamed by providers who optimise for benchmark performance rather than general capability. A model that scores highest on MMLU may not perform best on the specific task the buyer cares about; a model that scores highest on HumanEval coding benchmarks may produce code with security issues not measured by the benchmark.
The most reliable evaluation is the one that tests the model on the specific task the buyer intends to use it for, with the specific types of inputs and quality criteria that matter for the application. Building a small evaluation set of representative inputs with expected outputs and testing candidate models against this set produces more actionable selection data than comparing published benchmark scores. This is the approach that production AI teams use; it’s also the approach available to any buyer willing to invest an hour in defining evaluation criteria before model selection.