The agents can attack. They cannot yet vouch for what they found.

Autonomous penetration-testing agents are becoming more capable. That does not make their reports self-verifying. Five recent preprints ask a question that a headline success rate cannot answer: does the evidence in an agent's trace support the claim it makes?

Our position is simple: an agent's finding should earn its place in a report through reproducible evidence. More activity, a recovered flag, or agreement between reviewers is not a substitute.

Capability is rising. That is not the disputed part.

Lovelace and colleagues compared two PentestGPT-based systems on three public targets. The newer autonomous system completed all three. The legacy system still completed roughly half the subtasks on targets it did not finish. The authors explicitly warn that model, harness, autonomy and memory changed together, so the comparison does not isolate the cause of the improvement. Read the capability study.

This is a reason to measure offensive capability, not a reason to accept every generated conclusion. A system can execute a useful attack step and still make an unsupported inference about the result.

A flag is an outcome. The trace explains how it happened.

CTF-ABACUS examined 1,435 attempts by six models across 240 challenges. Of the recovered flags in its provenance analysis, 72.9% were supported by execution, 10.9% by human-verified derivations without observable execution, and 16.2% by neither. Trace-verified exploitation accounted for 62–87% of recovered flags across benchmarks. Read the trace-provenance study.

Those categories matter. A mathematical derivation can be valid without an exploit execution. A flag exposed directly by the environment is a different event. A memorised answer is different again. Combining them into one success count hides what the system demonstrated.

More probes do not repair a broken evidence chain.

ATOBench evaluated five model routes over 450 episodes, paired between ordinary and deliberately altered target responses. In its SQL-injection contract, grounded verification fell from 44.0% to 0% under the registered response transformations. Basket-ownership and JWT-artifact tasks behaved differently; this was not a uniform collapse across every task. Read ATOBench.

The result is specific to the benchmark, transformations and verification criteria. It does not establish that agents can never verify SQL injection. It does establish that successful reporting depends on how a system handles conflicting observations, finds another evidence path and decides when to stop claiming success.

Ten agreeing reviewers can still be wrong.

Refute-or-Promote describes a campaign in which ten reviewers endorsed a supposed OpenSSL padding oracle. An empirical test disproved it. The author reports externally accepted outcomes from the broader campaign, but explicitly says no vulnerability was discovered autonomously. Read the methodology and its failure case.

Agreement is useful only when it adds independent evidence. Repeating an assumption through several agents can make a report sound more confident without making it more correct. The larger Hackers or Hallucinators study, which examined 13 open-source frameworks and two baselines, is another reason to evaluate the complete system rather than its model name.

What an evidence gate should require

• A reproducible claim. State the affected component, preconditions, observed behaviour and expected behaviour.
• A preserved evidence path. Keep the relevant requests, responses, execution output and baseline or differential comparison.
• A test that could disprove the finding. A successful check must rule out the plausible alternative explanation, not merely repeat the original prompt.
• An honest disposition. Label an unverified lead as a lead. Keep failed reproduction attempts and material limitations with the record.
• Human ownership. A named reviewer remains responsible for scope, interpretation and the decision to report.

These are engineering requirements we advocate, not a claim that SecurePeak independently reproduced the cited experiments. The sources are preprints with different tasks and methods; their percentages should not be combined into a universal agent failure rate.

The deliverable is evidence a recipient can evaluate. The finding is the conclusion that evidence supports.