Skills for AI agents hurt weak models: a meta-skill recovered up to 14 points

Skills that make a strong model better can make a weak one worse. That is the conclusion of a ten-author group led by Zhihao Yang, in a paper on self-evolution of the LLM agent "harness." Across seven models and three benchmarks, the researchers found 103 task-skill pairs where a skill lifts GPT-5.6-luna by 31.1 points but drops Gemma-4-E4B by 8.8. Their proposed fix, adapting skills to the executor, gave weak models gains of 5.1 to 14.1 points.

One skill, two opposite effects

Harness self-evolution addresses a practical problem. An agent accumulates experience from completing tasks and records it as skills: text instructions, procedures and checks. The model weights do not change. This approach is cheaper than fine-tuning, and developers of coding agents use it extensively.

The central premise is that the model can do what the skill says. The authors tested this and found the premise does not always hold. A skill's effect depends on the capabilities of the executor. A strong model follows a long multi-step instruction and benefits. A weak one loses the thread, calls tools incorrectly and breaks the rest of the plan.

A more surprising finding concerns the causes of failure. In 86% of failed runs by weak models, the violated requirement was already spelled out in the skill. The model did not lack knowledge of what to do. It lacked the ability to carry the instruction through to execution.

The meta-skill: diagnosing four types of breakdown

This observation led to the idea of skill adaptation: rewrite an existing skill so that a specific model can execute it more easily. To do this, the authors sorted failures into four categories:

• tool execution errors; • context management errors; • failure to follow instructions; • verification errors.

They condensed these diagnoses into a fixed meta-skill, an instruction on how to rewrite other instructions. It serves as a guide for adaptation inside the self-evolution loop.

The researchers first tested the meta-skill on an external rewriter. Using it, Kimi-K3 adapted skills from the text alone, without access to execution results. Three weak executors improved by 5.1–14.1 points relative to the original skills.

Five rounds of self-evolution

Next, the team built adaptation into a full five-round loop. In it, each model edits its own skill based on execution trajectories and verifier feedback. There were seven executors, ranging from Gemma-4-E4B to Kimi-K2.5. The meta-skill consistently improved the final result by 2.0–5.1 points compared with the same loop without it.

Key figures from the study:

• 7 executor models, 3 benchmarks; • 103 task-skill pairs with diverging effects; • +31.1 points for GPT-5.6-luna and −8.8 for Gemma-4-E4B on these pairs; • 86% of weak-model failures involve requirements already written in the skill; • +5.1–14.1 points from external skill adaptation by Kimi-K3 for three weak executors; • +2.0–5.1 points from the meta-skill in the five-round loop; • 24 pages, 12 figures and 4 tables in the paper.

What to change in agent pipelines

For teams building agents on a mix of models, the takeaway is straightforward. Do not copy a skill library between executors without testing it. A skill tuned on a flagship model may do harm when moved to a small model such as Gemma-4-E4B. Before deployment, run it on the target executor and compare it with a no-skill baseline.

The second practical point concerns cost. Adaptation works even without execution traces: a strong rewriter model and the meta-skill are enough. That makes it a convenient way to prepare skills for cheap local models, which are more economical to run. The third point concerns diagnostics. The four failure categories (tools, context, instruction following, verification) can serve as a checklist when analyzing agent failures and when writing skills from scratch. The exact wording of the meta-skill is published on the project page, and the preprint is available on arXiv under number 2610.33024.