Software Engineering Agent Skills Are Turning Into Mysticism

This post was originally written in Chinese and translated to English by Grok 4.6. The original version is here.

Quite a few skills have gone viral recently. Ponytail claims to fix over-engineering, Caveman claims to slash token usage, andrej-karpathy-skills tries to raise the engineering quality of coding agents, and Superpowers combines brainstorming, planning, TDD, and code review into a full development workflow.

On September 3, 2026, these four repos sat at 120k, 100k, 210k, and 280k GitHub stars. React is around 250k.

They all hit real pain points. Coding agents really do ramble, add unnecessary abstractions, and start editing code before the requirements are clear.

But do those few dozen or hundred lines of Markdown actually solve these problems?

Two Kinds of Skills

What currently lives under the name "Agent Skill" falls into two categories, I think.

The first is the task-specific capability skill. It's closer to an SOP or runbook you'd give a new hire: the domain knowledge for a particular job, project context, steps, file formats, even executable scripts.

A skill for processing a company's financial-form PDFs, for example, might know which tools to call, how to locate form fields, what format the output has to satisfy, and come with a stable parsing script. This kind of skill supplies knowledge and tools the model didn't already have, so of course it can improve results substantially. That also matches Anthropic's original definition of Agent Skills: a combination of instructions, scripts, and references.

The SkillsBench paper from earlier this year supports this: across 87 tasks, human-written, task-matched curated skills raised the average pass rate from 33.9% to 50.5%.

The second is the generic behavioral skill. These usually add no new domain knowledge and no tools the model couldn't already use. They just emphasize a particular behavior:

write less code;
don't over-engineer;
think before implementing;
tests first (TDD);
don't ramble;
prefer the standard library;
only do what the user explicitly asked for.

These are generic engineering values and strategy choices. And most of the skills going around lately are this kind.

Packaging a Trade-off as Best Practice

This second kind of viral skill tends to follow the same pattern:

Find a model behavior everyone complains about. Almost all of these sit on a trade-off: plan first or start coding, explain more or talk less. Then pick one side of the trade-off, rewrite it as ALWAYS, MUST, NO EXCEPTIONS, and attach a persona that spreads well: "the laziest senior engineer," "talk like a caveman," "write code like Andrej Karpathy."

Then run some so-called evaluation that happens to measure the metric the instruction most directly affects (some don't even have an evaluation):

tell it to write less code, measure LOC;
tell it to talk less, measure output tokens;
tell it to plan first, treat process completeness as quality.

This can prove that skills change the model's behavior. What it doesn't prove is that those behavior changes produce better general performance.

Ponytail and Caveman

Ponytail is a typical case. It's essentially YAGNI, standard-library-first, and minimal implementation written as a stricter set of instructions. Colin Eberhardt re-ran its benchmark and found that "Follow YAGNI principles" already came close to the full skill. Add "one-liner solutions," and those seven English words beat Ponytail on its own benchmark.

To be fair, Ponytail does get the agent to write less code. That's a long way from what the project's marketing implies: install this skill and your coding agent magically acquires a senior engineer's engineering taste.

Caveman is similar. It does shorten the agent's narration. But in real agentic workloads, most of the output is already code, diffs, tool calls, and error messages. A JetBrains blog that forced Caveman on measured only an 8.5% drop in output tokens, not the advertised 65%. Task quality didn't improve at all.

Andrej Karpathy Skills

andrej-karpathy-skills is similar. It borrows Karpathy's name (this skill was of course not written by Karpathy) and sounds like it will make the agent think the way Karpathy does. The core is just: think first, keep it simple, change less, and make the verification target explicit.

Kun Cheng measured it on 192 paired ProgramBench tasks. After adding these guidelines, average test pass rate dropped from 53.7% to 51.5%, and cost went up 5%. The main failure mode wasn't that the model ignored the instructions. Following "minimum code" and "only implement what's explicitly required" led it to actively shrink the feature surface.

The interesting part: on most of the regressions, the model actually wrote more code. It would talk itself into "a compact custom evaluator, don't try to become a database," then refuse to just call SQLite, which was the simplest and most correct approach.

Superpowers

Superpowers is more complete than the ones above. It isn't just a style prompt. It's a whole software-development methodology: brainstorm first, get design confirmation, create a worktree, write an implementation plan, then strictly execute red-green-refactor TDD.

These practices are valuable on some projects. But why does every feature have to be TDD? Why does a tiny internal tooling change also need a design doc first? Why can't exploratory implementation help us understand the problem first?

Kun Cheng (strongly recommend following him on X) measured this on ProgramBench too. Forced TDD workflow, same 192 tasks: pass rate dropped 4 percentage points, cost up 55%, turns up 69%. The agent wrote incomplete tests, implemented only enough to turn its own tests green, then stopped early with false confidence.

One benchmark result doesn't prove TDD is bad. But it should at least make people a bit more sober: even an engineering method that's valuable for human teams is not always better when you force it onto an agent with no exceptions.

Can a Skill Deliver Better General Performance?

Generic behavioral skills can change an agent's output style. They can make a coding agent shorter, more cautious, more eager to write tests. Ponytail, for example, pushes the model toward minimalism. Superpowers pushes it toward process rigor.

What they can't guarantee is that the new strategy is a better fit for the current task. They often just move the error from one side to the other:

less over-engineering, more under-implementation;
more tests, not necessarily more useful coverage;
more planning, also more context consumption, cost, and time spent on the task.

If a few simple instructions really did stably raise correctness across all tasks, I believe model and harness vendors have stronger levers than a community skill: post-training, tool design, a verification harness, or putting the instruction in the provider's system prompt. It's unlikely to sit around as a Markdown recipe in a GitHub repo, waiting to be discovered.

Claude Code's system prompt has long said not to over-engineer, not to make changes the user didn't ask for, not to add unnecessary abstractions. A lot of the lines in these skills are things the model already reads every time it starts. What the community skill does is say the same things again, longer, more absolute, more meme-able, plus a persona.

Thanks to Kun Cheng, Colin Eberhardt, and JetBrains' Denis Shiryaev for spending real money running these skills one by one. Someone has to do it.

References: