The Dispatch #1, in full

In a production agent with 10-15 model calls per turn, plan for only 1-2 frontier calls doing heavy reasoning; the rest should be cheap classifiers or speculative calls. Design latency and cost budgets around this split from the start.

Classify your use case on the test-time compute scaling spectrum before sizing your inference budget. Factual retrieval barely improves with more compute; open-ended reasoning scales without a clear ceiling.

Model capability is a function of inference budget, so single-number benchmark comparisons are misleading. Always ask how much compute each model was given before drawing conclusions.

Frontier models can continue improving for weeks of inference time on hard benchmarks without plateauing, which means you must set explicit token or cost ceilings rather than running until performance flattens.

Run model calls in parallel wherever possible: pre-fetch knowledge before confirming you need it, and run two transcription models simultaneously with result arbitration to cut latency and reduce single-provider dependency.

Replace generic LLM-judge scalars (helpfulness: 0.4 vs 0.7) with task-specific rubrics tied to your agent's actual success criteria. A rubric failure on a specific task tells you exactly what to fix; a scalar score does not.

Off-the-shelf LLMs make poor synthetic users because they are polite and articulate, while real users send 1-2 word messages. Lyft's 90% offline pass rate collapsed in production; fine-tune your simulated user on real verbatim traffic before trusting any benchmark number.

Running a model N times and taking the best result inflates benchmark scores without any underlying model improvement. Always ask whether a reported eval score controls for compute budget before accepting it.

Structurally separate the team that builds evals from the team that trains models. Adversarial separation of incentives is the only reliable mechanism for preventing benchmark overfitting.

Validate your LLM-as-judge the same way you would validate any ML model: collect human-labeled ground truth from domain experts and check whether the judge's scores agree with them before trusting it as a quality gate.

Close the eval feedback loop automatically: route every trace that fails your LLM judge into an annotation queue, have ops staff label it, then have engineers analyze systematic failure modes. This creates a continuous improvement flywheel.

When your agent behaves incorrectly, the cause is almost always incoherent or conflicting context in the prompt, not model incapability. Treat context debugging as the primary diagnostic step.

Replace hard-coded step-ordering in agent logic with prompt-level policy statements and let the model handle sequencing. This keeps the graph flexible as task complexity grows.

Building a multi-agent system to mirror your org chart almost always destroys value by depriving each agent of context the other holds. Better context engineering solves the same coordination problem without the added complexity.

Use progressive disclosure in prompts: inject context only when it becomes relevant, and be careful when compacting history not to leave incoherent fragments. A mismatched system prompt versus conversation history is a reliable hallucination trigger.

When teaching a model a novel format or DSL, reframe your problem into structures the model already knows well (file systems, Git, grep) or inject context to teach your format explicitly. Using both levers together is the practical default.

Avoid fine-tuning or RL on your own task unless you are genuinely pushing state-of-the-art capability. Frontier models improve fast enough that a fine-tuned model is often obsolete within 3-6 months, making the tradeoff negative for most product teams.

LLM providers are not PCI-certified, so any agent handling payment data needs isolated infrastructure that never routes that data to an external model. Build this separation from day one because retrofitting it is painful.

Never fine-tune a model on data you would not want regurgitated to end users. The risk of proprietary customer data leaking through a fine-tuned model is a hard blocker for enterprise deployments.

Outcome-based pricing (commission on sales driven by the agent) is live in production today. If your agent drives measurable revenue, pricing on outcomes is a tested monetization path.

Simple memory signals (greeting by name, recalling the last call topic, acknowledging a prior frustrating experience) already produce meaningful lifts in resolution rate. Ship basic memory before building a sophisticated ML pipeline for it.

As of mid-2026, native voice-to-voice models are production-ready only for simple English-language journeys. They are roughly 10x more expensive than cascaded approaches, weaker at tool-calling, and unreliable across non-English languages.

For voice agents, spoken placeholders ('hang on while I look up your account') are the primary latency design lever. Some languages still carry 20-plus percent word error rates, requiring ensembling across multiple transcription providers.

“Anytime you think the model's being dumb, it's probably you.” 'Anytime you think the model's being dumb, it's probably you.' The Sierra founder's framing that almost every agent misbehavior traces to incoherent context rather than model failure.

“Our first AI agent was giving us 90% pass rate. Nice, we can launch, right? But it's a rude awakening for us when we actually launch in production.” 'Our first AI agent was giving us 90% pass rate. Nice, we can launch, right? But it's a rude awakening for us when we actually launched.' Lyft's offline-to-production collapse, caused by synthetic users that were too polite and articulate to resemble real traffic.

“It was honestly, it was shocking to see how inefficient I was in retrospect, and they were able to make it like, you know, 10 hundred X faster. And then I was like, okay, can you come up with an algorithm that is better... And it's not able to do it.” 'It was honestly shocking to see how inefficient I was in retrospect, and they were able to make it like 10 hundred times faster.' Noam Brown on AI delivering 100-1000x algorithmic efficiency gains on existing approaches, while novel algorithm synthesis remains out of reach.

“Instead of just running this model once, we're gonna run it five times and take the best of the five responses... you can get much higher scores than that model. It's really easy to make something that looks a lot better on paper.” 'Instead of just running this model once, we're gonna run it five times and take the best of the five responses... you can get much higher scores.' Noam Brown on how benchmark inflation via repeated sampling requires compute-controlled comparisons before any score is trustworthy.

“It would have been possible for somebody to disprove the Erdős unit distance conjecture before we did using a general purpose model and nobody had explored sufficiently of what happens if I put $100,000 worth of compute into 5.5.” 'It would have been possible for somebody to disprove the Erdős unit distance conjecture before we did using a general purpose model.' Noam Brown's claim that latent research-grade capability already exists in current frontier models and is going unexploited, with the conjecture falling to a scaffold estimated at $1k-$100k that no external team attempted.