What a Skill Changed, and What It Didn't

A task went well with a reusable skill enabled.

It also had clear requirements, repository instructions, connected tools, a useful test, and an engineer who corrected an early assumption.

Which part made the result better?

I would not answer that from the final diff alone. The diff can establish what changed. It cannot, by itself, tell us which part of the workflow caused the change to be correct.

I participate in evaluating reusable agent skills and refining how they are applied to ordinary engineering work. The interesting question is not whether the process produced a polished artifact. It is what observable contribution the skill made, what it cost, and what remains unknown.

This article describes an evaluation approach. It does not report internal programme results, measurements, or a controlled comparison that has established a productivity gain.

A skill is one input among several

A reusable skill packages instructions or a workflow. It may help discover context, plan a change, validate it, or handle review. It is not a separately trained model or an autonomous employee.

Its contribution depends on what was already present.

If repository rules already require a focused check immediately after an edit, a skill that repeats that instruction may add consistency, but it is not evidence that the check originated with the skill. If a connected tool supplies current requirements, the tool can explain why the agent avoided a stale assumption.

The engineer's domain knowledge matters too. A person may notice that two services disagree, narrow the investigation, or reject an obsolete proposal. Leaving that correction out makes the skill look more independent than it was.

I want the record to include those existing controls. Otherwise we are comparing a named framework with an imaginary workflow that has no useful practices of its own.

Observe behavior before assigning credit

The earlier evaluation article discusses what to measure. Here the narrower question is which explanation the observations support.

Use a deliberately illustrative record, not a real reporting form or an internal task:

Task: update a client without changing its callers
Observed behavior: agent checked a contract before editing
Existing control: repository instructions also require that check
Correction: engineer identified one stale source
Verified outcome: retained change passes the relevant gates
Skill contribution: may have reinforced discovery; attribution unresolved
Residual risk: deployment compatibility needs a separate decision

The conclusion is modest because the evidence is modest. That is not a failed evaluation. It is an evaluation that did not replace uncertainty with enthusiasm.

If the skill visibly introduced a previously missing check, say so. If the task does not distinguish its effect from existing instructions, say that instead.

A useful outcome is not always delivered code

A cross-repository investigation can end with an obsolete PR being closed. An architecture task can produce a proposal with explicit assumptions for discussion. A session-handoff change can produce a clearer continuation scheme.

Those outcomes have value. They are not interchangeable with a merged feature, an approved architecture, or guaranteed reproducibility.

Describe the actual artifact and its decision state. A reviewable proposal is ready to evaluate; it is not production rollout. A documented handoff reduces reliance on conversation history; it does not establish identical behavior across models and people.

In the supplied generalized experiences behind this series, recovery and refinement are part of the work. I would not remove the initial incomplete diagnosis simply because the final action was sensible.

The sequence explains what the process learned. A cleaned-up success story explains much less.

Separate work from overhead without pretending overhead is worthless

Implementation, framework setup, planning, repeated approvals, and reporting take different kinds of effort.

A specification can prevent an expensive mistake. It can also repeat a small task's already clear requirements. Reporting can support honest evaluation. It can also consume time that should be counted rather than hidden.

Keep normal engineering effort, framework overhead, and reporting effort distinguishable. Do not turn them into a single impressive number before understanding what each category contains.

The classification will sometimes require judgement. A discussion that clarifies an ambiguous requirement is engineering work even if a framework prompted it. A template completed only to satisfy the framework is a different contribution.

I would make that distinction visible rather than use it to prejudge whether the framework is good or bad.

Tiny tasks do not need a retrospective specification

For a small, clear change, the ticket and nearby contract may already be an adequate written requirement. A long specification can create another artifact to review without resolving a real uncertainty.

For ambiguous, risky, or multi-session work, a plan and checkpoints before implementation can be useful. They capture assumptions while there is still time for them to guide the work.

Do not write a specification afterwards and present it as evidence that the task followed specification-driven development. A retrospective note can explain the result, but it did not shape decisions that happened before it existed.

This is a simple chronology rule. It prevents polished documentation from making the process look more formal than it actually was.

Not using a skill can be the proportionate choice

An evaluation can tempt people to apply every available skill to every task so the record has plenty to discuss.

That changes the thing being evaluated. We are no longer observing useful selection; we are imposing ceremony for evidence collection.

Use a skill when it helps the task's current stage and risk. Record when a tiny or non-coding task offers limited evidence. Do not manufacture a repository-wide plan or another planning file after the fact to fill a reporting category.

A good workflow should be able to say "not needed here" without that becoming an admission that the whole framework failed.

Repeated success still has confounders

Several tasks going well can suggest a useful pattern. It does not automatically establish a causal productivity improvement.

Tasks differ in difficulty, familiarity, requirements, code quality, tools, models, and human involvement. An experienced engineer may become better at the domain during the same period the skills are introduced.

Without an appropriate baseline and comparison, avoid percentages, cost savings, or defect-rate reductions. Task counts and green checks are observations, not impact metrics by themselves.

Qualitative findings remain worthwhile: a missing boundary became explicit, a handoff had a clearer source of current decisions, or the process detected an unrelated generated change before review.

Name them as observations. They do not need a fabricated multiplier to be interesting.

Keep the evaluation record separate from public storytelling

An internal assessment may need task-level evidence, correction history, and reporting detail. A public article does not need to reproduce those artifacts.

Use generalized, explicitly labelled composite examples. Remove internal identities, programme details, source snippets, links, counts and model policy. Do not attach an employer's name to an anonymized case merely because the employment record is already public.

Even a collection of individually harmless details can make an episode identifiable. Preserve the engineering mechanism while changing or omitting unnecessary specifics.

The public claim should be something the available evidence can support without inviting the reader to inspect a closed system. "I evaluated and refined workflows" is a different claim from "this programme improved company delivery."

Feed the result back into the right layer

When an error repeats, decide which control should change.

An outdated repository convention belongs in repository context. A general validation habit may belong in a reusable workflow. A missing permission boundary may require tool access to change, not another paragraph in the prompt.

If the failure came from stale task evidence, repair the handoff or current-state check. If the skill added unnecessary steps, narrow when it applies. If the result remains ambiguous, keep the ambiguity in the record rather than forcing a favorable or unfavorable rating.

This is where evaluation becomes engineering work. The record is not the final product; the corrected operating arrangement is.

Useful attribution is specific enough to change the next task and modest enough not to invent a cause.

A smaller conclusion can be more reusable

I do not need every evaluation to end with a winner. I need to know which practice helped under which conditions, what required correction, and what the evidence could not distinguish.

That lets the next task inherit a more useful check or a simpler process instead of a larger claim.

The five articles in this wave share that approach. Context needs freshness. Green checks need scope. Tool output needs inspection. Reviews need a revision. Skills need honest attribution. None of those mechanisms removes the need for engineering judgement; they give it better material to work with.

That is the agentic engineering I want to describe: an environment that can learn from ordinary failures without pretending the agent was correct all along.

If a skill helped your work, what changed in the actual sequence of decisions? I'd be interested in that observation, including what your existing tools or judgement contributed alongside it.