Topic
Evaluation and models
Choosing models with evidence: evaluation harnesses, cost per successful task, and what changes when you host or switch a model family.
14 articles · Last updated
Articles
Evaluation & models
Decision models compared: OpenAI, Jev, Clef and more
Eight vendors now document models that answer typed questions with probabilities instead of text. This is what each one documents today — endpoint, question types, limits, images and list rate — what none of them publish, and how to choose on your own labelled data.
Comparison12 min read
Evaluation & models
AI updates, September–October 2026: models, prices, shutdowns
Five weeks brought GPT-6 in three tiers, three new Claude models, Gemini 3.8 Flash, Grok 4.7, DeepSeek V4.1-Flash and a model that does not write text. They also brought breaking API changes inside version bumps, the end of Sora 2 and Sonar Chat Completions, and dated price rises. This is what changed, from the vendors' own pages.
Analysis8 min read
Evaluation & models
What is Jev? TypeSafe's System One model, explained
TypeSafe's Jev is not a chat model. You send content and typed questions; it returns probabilities your code can branch on. Here is how the API works, how it is billed, where version 1.13 is weak, and how to evaluate it on your own decisions.
Guide11 min read
Evaluation & models
Claude Fable 5.1, Opus 5.5, Sonnet 5.5, Haiku 4.5 compared
Anthropic's lineup spans a tenfold range in list rate, two context sizes, three thinking modes and retirement commitments more than eleven months apart. Here is what Anthropic publishes about each, and how to decide which runs which task.
Comparison8 min read
Evaluation & models
Claude Fable 5.1 vs GPT-6 Astra: rates, caching and fit
Anthropic and OpenAI price their top models identically per token. The differences that reach an invoice are in cache reads, long prompts and location options, and the differences that reach users are in behaviour you can only measure yourself.
Comparison7 min read
Evaluation & models
Coding models compared: Opus 5.5, Kimi, Qwen3-Coder, MiniMax
Four models their vendors position for coding, compared on published facts only: rates, the prompt-length tiers that change them, context windows, caching and tool calling. Then a test-suite-based method for choosing on your own repository.
Comparison8 min read
Evaluation & models
Frontier models compared: GPT-6, Fable 5.1, Gemini, Grok
Four frontier models, compared only on what their vendors publish: list rates, context windows, output limits, modalities, caching and the long-prompt rules that change a bill. Then a method for choosing between them on your own work.
Comparison8 min read
Evaluation & models
GPT-6 Astra, Sol or Luna: choosing a tier per task
OpenAI's GPT-6 family comes in three tiers with the same context window and output limit and very different rates. The useful question is not which tier is best, but which is the lowest tier that passes each of your tasks.
Comparison7 min read
Evaluation & models
When a provider retires a model: a migration playbook
Model retirements arrive on the vendor's schedule, sometimes with two weeks' notice, sometimes taking a whole API with them. A playbook for finding every caller, qualifying the replacement and moving traffic by configuration instead of by deploy.
Guide8 min read
Evaluation & models
Same open-weight model, different hosts: a consistency test
The same model name on two inference hosts is not always the same behaviour. Precision, chat templates, limits and batching all differ. Here is a test protocol, the harness code, and how to read what it finds — with no results we did not measure.
Methodology8 min read
Evaluation & models
Fast, low-cost models compared: Gemini Flash, GPT-6 Luna
The small-model tier carries most production traffic by volume, and its list rates now differ by a factor of ten. What each vendor publishes about six fast, low-cost models, the dated changes to watch, and how to choose on your own tasks.
Comparison7 min read
Evaluation & models
Structured outputs across providers: JSON schema reliability
A 200 response is not a parsed object. Structured outputs fail by truncation, refusal, dropped parameters, unsupported schema keywords and plain wrong values — and each provider enforces a different subset of JSON Schema. A guide to measuring and containing it.
Guide9 min read
Evaluation & models
How to evaluate LLMs on real production workloads
Benchmarks narrow the field; your workload makes the decision. Build an evaluation set that measures success, cost, latency and operational failure together, and re-run it whenever something changes.
GuideUpdated 9 min read
Evaluation & models
Chinese open-weight models in production: hosting and compliance
DeepSeek, Qwen, Kimi, GLM and MiniMax all publish model weights. The production questions are less about the models than about who hosts them, how, under which license, and what your contracts allow.
GuideUpdated 9 min read