Methodology

Loop Engineering: Skills That Trigger When They Should

A skill only pays off when the agent invokes it at the right moment, and that invocation decision is where most of a skill library's value quietly leaks away. This is a methodology for treating skill triggering as a measurable behavior. Synthesize an eval set for every skill, run it through the harness you deploy, score compliance straight from the tool log, and raise the number with a /goal prompt and targeted edits. The loop runs until the compliance curve flattens. Skills are the worked example here, and the loop itself generalizes to any behavior you can quantitatively measure.

Agentic harnesses now ship with skills: packaged instruction sets that load on demand. Claude Code has skills and slash commands, Codex has custom prompts and standing instructions, and a homegrown loop usually has some equivalent, a playbook the model is supposed to consult for a class of work.[1] The skill encodes how a team wants that work done. The model decides when it applies. That decision is the weak link. A skill that fires on half the tasks it covers delivers half its value, and the missing half never shows up in a bug report, because the agent still produces plausible output from its priors.

The methodology below treats the invocation decision as a behavior you can measure, attribute, and improve. I call the practice loop engineering: hold an eval set fixed, run the harness, read the number, edit the context the model decides from, and run again. This piece covers the method itself. What to generate, how to score it, where the /goal prompt enters, and what a post-training team can do with the artifacts the loop leaves behind.

01The behavior under test

Mechanically, skill exposure works the same way across harnesses. The model sees a listing of available skills, each reduced to a name and a short description, injected near the start of the session. The trigger moment arrives later, buried in a live task, and the model must map that task back onto a description it read once. Two failure modes fall out. Under-triggering: the model proceeds from its priors and the skill never loads. Over-triggering: the model loads a skill on an adjacent task outside its intent, paying latency and context for nothing. Under-triggering dominates in practice and is the quieter of the two, since the output still arrives and merely omits everything the skill would have enforced.

For a post-training audience the framing is familiar. This is a recognition problem over instructions about instructions, and it sits next to behaviors that post-training already shapes deliberately: system-prompt adherence, tool selection, and knowing when to ask. Compliance here means the joint behavior: on tasks inside a skill's contract the skill fires before substantive work begins, and on tasks outside it the skill stays quiet.

02The loop at a glance

Five stages, run as a cycle. A trigger contract defines ground truth. A synthetic eval set instantiates the contract as runnable tasks. The harness executes them. The tool log yields an exact-match score. Attribution turns misses into edits, and the cycle repeats on the same frozen set so every iteration's number is comparable to the last.

03Write the trigger contract first

Every skill gets one testable predicate: the conditions under which it should fire, and the nearby conditions under which it should hold. Both halves matter. The positive space defines recall, the negative space defines precision, and the pair is the ground truth every downstream grade refers to.

Writing the contract exposes the most common defect in a skill library before any eval runs. Most skill descriptions are capability summaries. They say what the skill does ("formats SQL migrations") and stay silent on when it applies ("fire whenever the diff touches a file under migrations/"). A capability summary asks the model to infer the trigger. A trigger contract states it. The contract also surfaces sibling conflicts early: two skills claiming overlapping conditions will split trigger probability between them, and that shows up later as a ceiling on both skills' compliance.

04Synthesize the eval set

Each skill's contract expands into a task set with a deliberate phrasing gradient, generated by a strong model working from the contract plus a few seed examples.[2] Four tiers do the work.

• Verbatim positives. The task states the trigger in the skill's own vocabulary. This tier establishes the floor; a miss here means the description itself is broken.
• Paraphrase positives. Same intent, none of the skill's vocabulary. This tier tests whether recognition survives a change of words.
• Latent positives. The prompt reads as generic work, and the trigger condition only emerges mid-task, from a file the agent opens or a tool return it reads. This is the deployment-realistic tier and the one that usually collapses.
• Hard negatives. Tasks that share surface vocabulary with the trigger and fall outside the contract. These price over-triggering and keep the fix for recall from silently destroying precision.

A complete item is a task prompt, an environment fixture (repo state, files, tool stubs that make the task runnable), and an expected label, fire or hold. A held-out slice gets human verification before anything is trusted. The generation machinery here is a small instance of the synthetic eval factory covered in the companion essay: same authoring loop, same gates, same insistence on receipts. Dozens of items per skill are enough for signal. The tier profile carries more information than raw volume, because the tier where compliance breaks tells you what kind of edit to make.

05Run the harness and score the log

Runs happen in the harness you actually deploy: Claude Code, Codex CLI, or your own loop. Each item gets a fresh session, a fixed model, a fixed config, and k rollouts, since triggering is stochastic at nonzero temperature. The unit of evidence is the transcript.

Scoring is the cheap part, and that is the point of aiming the eval at this behavior. A skill invocation is a discrete event in the tool log, a call carrying the skill's name, so the primary metric needs no judge at all. Grade fired or held by exact match against the item's label. Report trigger rate on positives and false-fire rate on negatives, per skill and per phrasing tier. Across rollouts, report trigger@k, the probability at least one of k rollouts fires,[3] alongside the all-k consistency rate, which is the number a deployment actually feels. A judge enters only for the secondary question: once the skill loaded, did the model follow what it says. That adherence score matters, and it stays cleanly separated from the triggering score so the two failure modes never blur.

06The /goal prompt

The edit that moves the number most is a standing prompt, invoked at session start as a slash command that takes the session's objective as an argument, or injected by the harness as a preamble. It has three parts.

• The objective. One or two sentences on what this session is for.
• The skill map. Every active skill with its one-line trigger condition, pulled straight from the contracts.
• The compliance rule. A standing instruction: before starting any task, check the map, and when a trigger condition matches, invoke the skill before doing the work.

The mechanism claims are testable, and the eval set is what tests them. Three hypotheses. Salience: the map sits close to the decision point, and the harness's own skill listing sits at the top of a long window. Recognition over recall: checking a task against an enumerated list is an easier problem than spontaneously remembering that a relevant list exists. Goal conditioning: a stated objective gives the model a criterion for judging whether an ambiguous task falls inside a trigger condition.

Measurement is an A/B on the frozen set with four arms: baseline, description edits alone, /goal alone, and both. The per-tier deltas separate the hypotheses. Lift concentrated on latent positives points at salience. Lift on paraphrases points at the map's vocabulary doing the matching. Movement in the false-fire rate prices what the compliance rule costs in precision, which is the check that keeps the /goal prompt honest.

07Attribute failures and edit

Every miss gets read and assigned one cause. The taxonomy stays small.

• Vocabulary mismatch. Paraphrase positives fail, verbatim ones pass. Edit: fold the failed paraphrases' vocabulary into the description.
• Capability summary. Verbatim positives fail. Edit: rewrite the description as the trigger predicate itself.
• Buried trigger. Latent positives fail. Edit: add the condition to the /goal map, so it is present at decision time.
• Sibling shadowing. A neighboring skill fires in place of the right one. Edit: split or merge the contracts until each condition has one owner.
• Deliberate decline. The transcript shows the model noticed the skill and proceeded without it. Edit: strengthen the compliance rule, and read these transcripts closely, since a pattern of declines sometimes means the contract is wrong about what the skill should cover.

One edit class per iteration keeps attribution clean, and the frozen set keeps the curve honest. The loop ends when the curve flattens. Whatever misses remain at that point are model-level, meaning context engineering has done what it can, and the residual is a property of the policy. That residual, precisely characterized, is the handoff to training.

08The general form

Step back from skills and the loop has a general shape. Four ingredients make it run: a behavior that leaves a measurable trace, a frozen eval set drawn from your own distribution, a knob you can edit between runs, and an attribution step that links each failure to one edit. Given those, hill-climbing is mechanical. Hold the set fixed, turn one knob, read the delta, keep what moves the number. Skills are a clean worked example because the trace is a discrete tool call and the grade is exact match. The same recipe covers a much wider surface.

• Tool selection. The right tool among overlapping ones, graded from the log the same way a skill invocation is.
• Output contracts. Schema and format adherence, graded by validators, hill-climbed through system-prompt and tool-description edits.
• Escalation calibration. The ask-versus-act boundary, graded against decision points with known correct actions.
• Retrieval. Whether the right document entered context before the answer was written, graded by lookup against the fixture.
• Delegation. Whether a task that warranted a subagent got one, graded from the spawn events in the transcript.

The quantitative measure is the ingredient that carries the weight, and it deserves the most suspicion. A metric that the edits can game turns the loop into Goodhart's law with extra steps, so a held-out slice stays frozen and unseen by the editing process, and a periodic audit checks that the number still tracks the outcome it stands in for. When the grade needs a judge, calibrate the judge before trusting the curve, since a drifting judge turns hill-climbing into noise. With those guards in place, the loop is a general instrument: own an eval set for the behavior, and the behavior becomes improvable.

09What post-training gets out of it

The loop is a context-engineering practice on its face. Its artifacts are post-training assets.

• A verifiable reward. Compliance is graded by exact match on a discrete event, which makes it a clean RLVR target: cheap to compute, hard to game, private by construction, and drawn from a distribution no public benchmark covers.
• Labeled trajectories. Every scored transcript is a demonstration of the triggering decision with a verified label. Failed rollouts paired with their corrected behavior are preference data for exactly the behavior the loop was measuring.
• An instruction-sensitivity probe. The same frozen set runs against any model in any harness. The compliance profile across phrasing tiers becomes a comparable, portable measurement of how much context engineering a given model needs before it uses the tools it is given.
• A regression suite. Model releases and harness updates both move triggering behavior. A set that took an afternoon to build answers, per release, whether your skills still fire.

The contribution is the loop. A trigger contract you can grade against, a synthetic set you own, a scalar you can watch move, and a /goal prompt whose lift is an A/B on a frozen set. Run it until the curve flattens, then hand the residual to training.

Sources

1. Anthropic engineering. Equipping agents for the real world with Agent Skills. 2025. anthropic.com/engineering.
2. Wang et al. Self-Instruct: Aligning Language Models with Self-Generated Instructions. 2022. arXiv:2212.10560.
3. Chen et al. Evaluating Large Language Models Trained on Code. 2021 (origin of pass@k). arXiv:2107.03374.

Companion piece: Manufacturing Synthetic Evaluation Data, which covers the generation machinery this loop borrows.