Introduction
In the first half of 2026, Agent research saw growing interest in Harness auto‑evolution techniques. This family of approaches leverages a dedicated Meta‑Agent to analyze execution traces and task feedback from target agent instances. Based on these observations, the Meta‑Agent iteratively adjusts the agent’s Harness, which covers prompts, tool definitions, middleware layers, memory mechanisms and control workflows.
A key research paper titled Rethinking the Evaluation of Harness Evolution for Agents raises a critical question for practitioners: are reported performance gains from Harness evolution truly driven by improved Harness logic itself, or do they simply stem from increased sampling budget, repeated retries and more execution attempts?
The paper sets up controlled experiments. It puts four different strategies under identical computational budgets: parallel sampling, sequential refinement, Harness Evolution and Harness Scaling. All experiments share the same base LLM and initial Harness configuration, with a fixed total allowance of five execution attempts per task. The core variable lies in how those five budget slots get allocated. Researchers separate performance metrics collected on training tasks versus unseen hold‑out evaluation tasks, to test whether evolved Harness assets can generalize to new scenarios.
Experimental results on the Terminal‑Bench 2.1 dataset deliver counter‑intuitive findings. Harness auto‑evolution fails to outperform straightforward test‑time repeated sampling. When evolved Harness configurations are deployed on completely unseen tasks, marginal performance improvements shrink almost to zero. For production agent stacks that route traffic across multiple LLM backends, developers may leverage an API gateway to manage inference traffic; Treerouter offers one option for unified request orchestration.
1. Blind Spots in Harness‑Evolution Evaluation
Current evaluation methodologies for Harness auto‑evolution contain two major blind spots that can create misleading performance illusions.
First is the sampling budget confounder. Harness auto‑evolution forms a feedback loop: run agent tasks, collect traces, modify Harness parameters, and run again. Compared to running a static initial Harness once, this iterative workflow consumes far more inference attempts. If evaluators only compare final task success rates without normalizing for total execution budget, performance gains brought by extra sampling attempts can easily be misattributed to improvements inside the Harness itself.
Second is over‑fitting to benchmark tasks. Many research teams optimize Harness evolution using public benchmark datasets, guided by task traces and unit‑test feedback. After dozens of revision cycles, the evolved Harness accumulates hard‑coded command patterns, file‑path conventions, error‑handling rules and verification logic tailored specifically to benchmark samples. While performance rises on familiar training tasks, these modified rules may not transfer to unseen real‑world assignments. Independent validation on disjoint test sets is mandatory.
The paper’s Terminal‑Bench 2.1 aggregate pass@1 results (without unit‑test feedback) are summarized below:
| Method | Average pass@1 |
|---|---|
| Initial static Harness (baseline) | 68.2 |
| Parallel Sampling | 72.3 |
| Sequential Refinement | 69.3 |
| Harness Evolution | 67.4 |
| Harness Scaling | 71.8 |
Parallel Sampling achieves the highest score at 72.3. Harness Scaling reaches 71.8. Harness Evolution underperforms the original baseline, scoring only 67.4. This dataset demonstrates that modifying and refining Harness structures does not automatically translate to higher task completion rates.
2. Four Contrasted Strategies Under Equal Budget Constraints
To isolate benefits originating from “better Harness logic” versus “more execution attempts”, all four approaches run under a unified K=5 execution budget for each task. They differ in whether they mutate the Harness, and how computational budget gets allocated.
- Parallel Sampling: Harness remains fixed. Generate K independent execution trajectories and select the best result. No modification is applied to Harness definitions. Optimization focuses purely on exploring multiple candidate trajectories for each individual task.
- Sequential Refinement: Harness remains fixed. Each new attempt builds upon execution traces from prior runs. The agent deepens exploration iteratively without altering underlying Harness templates.
- Harness Evolution: Harness gets updated iteratively. A Meta‑Agent ingests execution traces, summarizes recurring failure modes, and revises a shared cross‑task Harness to be reused across future jobs.
- Harness Scaling: Per‑task Harness modification. The Meta‑Agent adjusts Harness copies for every single task based on its own execution history. Changes are not propagated globally across different tasks.
Conceptually, Parallel Sampling expands search breadth via independent trajectories. Sequential Refinement increases search depth by learning from prior‑run outputs. Harness Evolution accumulates shared improvements intended for cross‑task reuse. Harness Scaling produces one‑off Harness adjustments scoped to individual problems.
Figure 2 in the source paper visualizes workflow differences:
- Parallel Sampling explores wide trajectories with frozen Harness.
- Sequential Refinement explores deeper trajectories while keeping Harness unchanged.
- Harness Evolution maintains a centrally stored, shared Harness that evolves across batches of tasks.
- Harness Scaling generates independently‑evolved Harness instances for each separate task.
3. Experimental Setup
Terminal‑Bench 2.1 serves as the primary evaluation suite, containing 89 command‑line agent tasks. This release fixes external dependency bugs present in version 2.0. The maximum reasoning budget is capped at 128 k tokens. Three large‑model back‑ends are used for statistical robustness: Claude Opus 4.6, GPT‑5.4 and GPT‑5.4‑mini, all configured with high‑reasoning mode.
All four experimental branches start from exactly the same minimal initial Harness. The agent only has access to a Bash tool; no pre‑built skills, middleware or persistent memory modules are enabled. Every task receives exactly K=5 total execution attempts. The experimental environment disables built‑in external retrieval functions native to Agent Harness Engineering, so Meta‑Agents can only learn from self‑generated execution traces, without unit‑test or external ground‑truth feedback. This setup isolates the pure performance of Harness‑evolution logic.
Three distinct experimental scenarios are tested:
- No unit‑test feedback: the agent judges result quality purely by interpreting its own output traces.
- With unit‑test feedback: external unit‑test outcomes are available as ground‑truth signals.
- Disjoint training‑test split: tasks are partitioned into training sets for Harness evolution and held‑out test sets for generalization measurement.
4. Experimental Outcomes
4.1 Scenario One: No Unit‑Test Feedback
Without unit‑test signals, agents must evaluate outcome quality relying solely on self‑reflection over generated traces. The aggregated pass@1 results are shown below.
| Method | Claude Opus 4.6 | GPT‑5.4 | GPT‑5.4‑mini | Average |
|---|---|---|---|---|
| Baseline (initial Harness, single run) | 69.9 | 75.3 | 59.4 | 68.2 |
| Parallel Sampling | 74.7 | 79.2 | 62.9 | 72.3 |
| Sequential Refinement | 73.0 | 73.0 | 61.8 | 69.3 |
| Harness Evolution | 71.4 | 69.7 | 61.2 | 67.4 |
| Harness Scaling | 76.0 | 78.1 | 61.3 | 71.8 |
Parallel Sampling delivers the highest average pass@1 improvement, lifting average performance from 68.2 up to 72.3. Harness Scaling achieves competitive results on Claude Opus 4.6, but gains diminish on smaller‑capacity GPT‑5.4‑mini. Notably, Harness Evolution underperforms the original baseline on average. When external objective feedback is absent, Meta‑Agents struggle to reliably tell true failure from misleading trace artifacts. Erroneous self‑diagnosis leads to harmful Harness edits that degrade subsequent task performance.
4.2 Scenario Two: With Unit‑Test Feedback
When unit‑test pass/fail signals become available, all methods outperform the static single‑attempt baseline. Metrics include pass@1 (success rate on first attempt) and pass@5 (success rate within five total attempts).
Even with reliable ground‑truth feedback, Harness‑evolution variants still cannot outperform simple test‑time sampling baselines. Parallel Sampling reaches an average pass@1 of 86.0 and pass@5 of 86.0. Sequential Refinement attains average pass@1 of 84.3 and pass@5 of 91.8. Harness Evolution only reaches average pass@1 of 75.8.
This observation yields an important insight: multi‑attempt sampling improves the probability of finding at least one valid solution within five tries (pass@5). However, better Harness revisions do not necessarily raise the success probability of a brand‑new independent first‑shot attempt (pass@1). Improvements found through repeated trial‑and‑error do not automatically encode reusable generalized logic.
4.3 Scenario Three: Disjoint Training and Evaluation Tasks
To measure generalization capability, researchers split the 89 tasks: 45 training tasks for Harness evolution, 10 validation tasks for picking optimal Harness snapshots, and 34 completely unseen test tasks. Harness evolution runs on training data only; final pass@1 scores are reported on isolated test tasks.
After optimization on training samples, evolved Harness yields very limited gains on unseen test assignments. Average pass@1 only rises by 0.6 points overall. Performance gains observed on training datasets largely disappear when transferred to new problems. This strongly indicates that Harness‑evolution mechanisms tend to memorize patterns specific to training benchmarks rather than learning broadly‑applicable agent logic.
4.4 What Exactly Does Meta‑Agent Modify Inside Harness?
Examination of real‑world Harness‑Scaling revision artifacts reveals what changes the Meta‑Agent actually produces. Most edits target prompt text blocks and exception‑handling workflows. Examples include:
- Adding fine‑grained time‑out constraints and early‑exit conditions for hanging shell commands.
- Inserting explicit sanity‑check steps after file‑system operations.
- Tuning cancellation logic for asynchronous background tasks.
- Adding specialized error‑recovery snippets for niche failure modes observed in training traces.
The majority of these patches are highly task‑specific. They fix concrete failure cases seen during training traces, instead of discovering universal algorithmic improvements applicable across diverse agent scenarios. General‑purpose reusable agent mechanisms remain rare among automatically generated Harness revisions.
5. Discussion and Practical Implications for Agent Engineering
5.1 Scope of Experimental Limitations
These conclusions should not be over‑generalized to every possible agent workload. Terminal‑Bench 2.1 focuses heavily on shell‑command‑based system operations. Real‑world production agents may incorporate complex multi‑skill composition, long‑running multi‑stage workflows and memory layers not represented in this benchmark.
The experiments also fix a strict K=5 attempt budget. Results could shift given substantially larger computational budgets. The research does not negate the potential long‑term value of Harness evolution; it highlights that current implementations must be carefully benchmarked against simple test‑time sampling baselines under equal‑budget constraints.
5.2 Practical Evaluation Guidelines for Agent Researchers and Engineers
When assessing Harness‑evolution or agent self‑improvement systems, practitioners should adopt three critical evaluation checks:
- Enforce equal‑budget comparison: Always contrast against test‑time sampling baselines with identical total inference attempts. Higher pass@5 alone does not prove improved Harness quality. Pay close attention to pass@1 metrics on independent fresh tasks.
- Test generalization on disjoint datasets: Performance on training‑distribution tasks means little. Validation must run on fully unseen hold‑out tasks to rule out benchmark over‑fitting.
- Distinguish two separate sources of gain: Gains can come either from broader sampling search space, or from genuinely improved reusable Harness logic. Only gains that lift pass@1 on unseen tasks can be attributed to better generalized Harness design.
Many teams building production agent systems combine both approaches. Test‑time sampling handles high‑volume ordinary tasks, while Harness evolution can be applied cautiously for domain‑specific tuning, provided strict generalization validation is enforced.
Conclusion
Harness auto‑evolution is an active and promising research direction for Agent systems. Yet existing experimental evidence warns us against over‑hyping its real‑world returns. Under controlled fixed‑budget conditions on Terminal‑Bench 2.1, Harness‑evolution approaches cannot surpass the performance of straightforward multi‑attempt test‑time sampling. Most observed gains come from increased numbers of execution retries, rather than generalized reusable Harness improvements. Auto‑tuned Harness logic frequently over‑fits training‑set idiosyncrasies and shows limited transfer to new tasks.
For engineering teams building self‑improving agent pipelines, standardized evaluation workflows are essential. Equal‑budget baseline comparison, disjoint‑set generalization testing and close inspection of both pass@1 and pass@5 metrics are required to tell apart genuine algorithmic advancement from artifacts of larger sampling budgets.
Learn more：https://treerouter.com
