The Dispatch #2, in full

Build a verification mechanism (tests, screenshots, diffs) into every agentic loop. Without one, long-running tasks stall; with one, they can run unsupervised for days or weeks.

Add per-step checkpointing from day one. Agents fail mid-task regularly, and forcing a restart from step 67 of 123 is a user-experience failure that compounds over time.

Treat eval robustness as an adversarial arms race. AI systems will find and exploit loopholes in your evaluation harness, such as noticing that correctness checks run exactly 15 times before performance testing begins.

To prevent hallucinated or invalid actions, frame tool selection as a classification problem: give the model an explicit enumerated list of valid actions instead of letting it generate free-form action text.

When a codebase exceeds the model's context window, the correct agentic pattern is iterative search and retrieval to find relevant context, then summarize code structure, then generate and verify the patch.

Fine-tune for use-case-level behavior (e.g., how to handle customer service conversations), not for individual customer business logic. Customer-specific logic belongs in the context layer and should change without retraining.

Evaluate fine-tuned models end-to-end on customer outcomes, not on isolated task metrics or loss curves. The right question is whether the model is working in concert with all other models to deliver the end outcome.

Fine-tuned smaller open-source models can outperform large frontier models on specific narrow tasks, giving better accuracy, lower latency, and lower cost simultaneously.

Data curation alone, with no architecture changes and no post-training, can yield a 14 absolute percentage point benchmark improvement on VLMs and let a smaller model match a much larger one using 145x less training compute.

Mid-training on high-quality domain data before post-training can make your RLHF pipeline 2 to 3x more effective without changing post-training data at all. Invest in pre/mid-training data quality first.

The hard part of fine-tuning open-source models is evals, not the training itself. Public benchmarks are useless; you must build custom evals tied to your specific task and customer outcomes.

With each new model release, delete your entire system prompt and add instructions back only line-by-line when you observe repeated failures. Prompts from the previous model often do not apply and waste the model's attention.

Only add a prompt instruction when you observe the model repeatedly stumbling on the same thing. Adding instructions preemptively wastes the model's attention on every single call.

Standard RAG plus re-ranking is not enough for high-quality retrieval. Adding domain-specific validation logic (checking frequency, recency, and whether values match priors) took retrieval accuracy from roughly 50% to 90%.

Every model generation has product overhang: capabilities the model already has that no product is yet eliciting. The biggest opportunity right now is building the product layer that unlocks those latent capabilities.

Enterprises are now spring-loaded to buy from unproven startups because the perceived risk of the status quo (being left behind) exceeds the risk of an untested vendor. Use that fear as your sales wedge.

When selling AI to regulated enterprises, map the deployment process in granular detail before closing: model risk review, testing, rollout, and issue remediation. Buyers care as much about 'can I get this live?' as 'will it work?'

Before committing to vector or embedding infrastructure, run the unit economics. Embedding plus search at scale can cost 6x your entire current infrastructure bill, which is a company-killer for bootstrapped or early-stage teams.

Lead with a pricing claim that is 100x better than the market. 'A million vectors for a dollar' versus roughly $100 per million cuts through noise and generates immediate inbound from the right customers.

Start your AI agent product with cost-cutting use cases to get in the door, then plan your roadmap toward revenue-generating use cases where the ceiling on value and pricing expands significantly.

“The verification I think is probably the single most important thing that people do not get right... give it a way to verify the output of its work so it doesn't get stuck, and it will just go.” "The verification I think is probably the single most important thing that people do not get right... give it a way to verify the output."

“It ran for 11 days, and it rewrote the entire code base... This is in production now. This is what Claude Code uses.” "It ran for 11 days, and it rewrote the entire code base... This is in production now. This is what Claude Code uses."

“It noticed that during our correctness suites we were checking for correctness 15 times and then we would do performance testing. But when we were doing performance testing we weren't doing correctness testing again. So what the AI was doing is it was counting how many times it did correctness testing.” "It noticed that during our correctness suites we were checking for correctness 15 times and then we would do performance testing."

“The model was capable of doing something and everything was just kind of getting in the way.” "The model was capable of doing something and everything was just kind of getting in the way."

“people know that the risk of the status quo is actually extremely high. And so, even if there's risk in doing all the new things, well, this path also looks pretty dangerous.” "People know that the risk of the status quo is actually extremely high. And so, even if there's risk in doing all the new things..."