Skills for AI Agents Aren't Universal: Weak Models Need Their Own Version of the Prompt

Skills for AI

Agents Aren't Universal: Weak Models Need Their Own Version of the Prompt

A research group from the University of Chinese Academy of Sciences, UC Santa Cruz, the National University of Singapore, and the Institute of Software at the Chinese Academy of Sciences has uncovered a non-obvious problem in self-improving AI agents: the instructions an agent writes for itself after completing tasks — known as "skills" — don't help every model equally.

The idea behind harness self-evolution is that an agent, without touching its own weights, accumulates experience as text-based instructions and consults them when facing similar tasks later. Across three benchmarks and seven models of varying strength, the team found 103 task-plus-skill pairs where the effect of a given skill diverged sharply: GPT-5.6-luna gained 31.1 points from using it, while the weaker Gemma-4-E4B lost 8.8 points compared to having no instruction at all. In other words, the instruction wasn't just useless for the weak model — it actively made things worse.

Why weak models fail even when the instruction is right there

The researchers read through the logs of failed runs and found that the issue wasn't missing information. In 86% of cases, the weak model violated a requirement that was explicitly written into the skill — it simply couldn't orient itself around it. The failures broke down into four recurring patterns:

• Tool confusion — the agent tries to call a nonexistent tool, mistakes a file for a tool, or stops after an error instead of trying to work around it.
• Loss of task context — the agent drifts toward the examples and reference material embedded in the skill, forgetting the actual task, or saves the result to the wrong location.
• Incomplete instruction-following — the agent completes several steps and produces an output, but only satisfies part of the requirements explicitly stated in the skill.
• No verification — errors remain in the final answer because the agent never rereads or recalculates the result through an independent method.

A meta-skill as a universal template for rewriting instructions

To fix this, the researchers used Claude-Opus-5 as an assistant to draft a "meta-skill" — a fixed, task-independent protocol for rewriting instructions. It requires: clearly separating actual tools from documents within the instruction, pre-describing environment checks and common pitfalls; opening the skill with a statement of the user's task, the boundaries of permissible changes, and where to save the result, keeping this distinct from any examples; phrasing every requirement with an explicit way to verify it's been met; and closing the skill with a dedicated self-check step — reread the result, recompute it a different way, compare against a threshold, and retry if needed.

The team tested the effect in two modes. The first ran without execution logs: the powerful rewriter Kimi-K3 was given only the task, the original skill, and a list of tools available to the executor, and rewrote the skill using the meta-template. The result — three weak executors (Qwen3.6-flash, Gemma-4-E4B, Qwen3.5-9B) improved by 5.1–14.1 points over the original instruction. For comparison, giving that same Kimi-K3 just a single generic line — "rewrite this so it works for all models" — without the meta-skill produced a much smaller and less stable gain; Qwen3.5-9B even dipped slightly. The meta-skill adds another 5.1–6.0 points on top of that — more than half of the total improvement.

Self-learning in a loop: the model rewrites its own instruction

The second mode was a more realistic five-round loop, where the model both performs the task and rewrites its own instruction based on the execution trace and verifier feedback. The results here are telling: without the meta-skill, weak models rewriting their own instructions barely help themselves and the process is unstable — Qwen3.5-9B peaks in the third round and then slides back close to its starting point, while the weakest model, Gemma-4-E4B, stays below baseline across all rounds. With the meta-skill, the picture changes: all seven tested models — from Gemma-4-E4B up to the top-tier Kimi-K2.5 — outperform the no-meta-skill variant by the fifth round, by 2.0–5.1 points.

The practical takeaway for teams building agents with self-evolving instructions: don't assume the same skill is equally useful across executors built on different models — a flagship-powered executor and one running on a compact open model. If a project mixes models of different capability levels, instructions should be adapted separately for each rather than copied verbatim — otherwise a weaker model may end up performing worse than if it had received no instruction at all.