Local SWE-bench agent — experiment results

Scope: exploratory local-model evaluations, October 20–22, 2026. Five SWE-bench Verified issues. The denominator below counts nominal model–issue combinations, not independent benchmark issues. For the October 20 pilot, the two-model × five-issue count comes from the user-maintained batch index; individual attempt-status records for that pilot are not bundled. The other 70 attempts have per-instance outcome summaries. Agent instructions, model sets, and reliability conditions changed between batches; do not pool these runs as a standardized leaderboard result.

Batch inventory

Definition: “Applicable” means a nonempty generated patch passed git apply --check against the intended base checkout; “resolved” means the separate SWE-bench evaluator marked the submitted prediction resolved. An applicable patch is not a successful issue repair.

Confirmed successful evaluation

• Batch: 20261022-015250
• Prediction: matplotlib__matplotlib-24026 / Qwen3-14B-Q4_K_M
• Evaluator split: 20261022-015250-a
• results.json: submitted_instances=2, completed_instances=2, resolved_instances=1, unresolved_instances=1, infra_failure_instances=0, error_instances=0
• resolved_ids: ["matplotlib__matplotlib-24026"]
• Source file changed in the prediction: lib/matplotlib/stackplot.py
• The other candidate in split -a, sympy__sympy-19346 from Qwen3-14B-Q5_K_M, was unresolved. An additional Qwen2.5 candidate for the same Matplotlib issue was evaluated separately (split -b) and was unresolved.

Original-batch input collision: The unsplit 20261022-015250.predictions.jsonl contained three prediction rows but only two unique instance_id values (two different models both predicted matplotlib__matplotlib-24026). Its initial evaluator log reported only two submitted instances and zero resolved. Splitting the same patches into -a and -b yielded the confirmed Qwen3 resolution. The exported patch bytes for Qwen3 were identical between the unsplit and split inputs (SHA-256 prefix ea27cf0aac3954c6). This strongly suggests duplicate-ID handling in the combined evaluator input obscured the winning candidate; the root log alone does not specify which duplicate was evaluated.

Final-batch evaluation

Batch 20261022-193725 produced three applicable diffs. They were split because two models predicted the same instance:

Both final split evaluators reported zero infrastructure failures and zero evaluation errors.

Five task identifiers

• matplotlib__matplotlib-24026
• scikit-learn__scikit-learn-14983
• scikit-learn__scikit-learn-26323
• sympy__sympy-11618
• sympy__sympy-19346

Model artifacts and families

These represent 5 artifact variants, correlated to 6 filenames, not independent model architectures. See the original Org source for complete filenames and model download URLs.

Audit and methodological limitations

1. Version provenance: The supplied analysis extracts show zeros for navigation/token counts in the early 20261022-015250 and 20261022-051312 batches because those early audit records did not expose later structured metrics. Those zero columns are not evidence that no navigation occurred.
2. Mixed interventions: The v2.3 and v2.4 variants differ in step budget and system instructions, preventing single-variable attribution.
3. v2.4 final-step defect: The archived run_agent uses if step >= 13: ... elif step == MAX_STEPS: ... when MAX_STEPS=16. The elif is unreachable at step 16, so the intended final patch-only instruction was absent. The controller still rejected navigation on the last step. So, effectively, the model was being sent the warnings that it was almost out of attempts from step 13, but no final warning at step 16 (the last step) that this was the last attempt, and to submit a patch if there was one at that turn.
4. Manual evaluation: Candidate predictions were assembled into evaluator inputs manually; when multiple models targeted the same issue, evaluation was split into distinct runs. The initial unsplit 20261022-015250 input included a duplicate instance_id, showing why per-model grouping matters. The same candidate may appear in more than one exported evaluation input. Do not sum evaluator submission counts as unique patches.
5. Small sample: Five tasks, reused across configurations. The observed single resolution is a factual outcome, not a statistically reliable estimate of general coding-agent capability.
6. Partial evaluation artifacts: The archive contains machine-readable results.json for the confirmed win and final two split runs; other runs are corroborated by the supplied evaluator root logs and Org summary, not by bundled results.json files.

Source evidence in the private research archive

• SWEBench-batch-run-summaries-models-and-manual-commands.org — experiment index, model provenance, manual evaluator and llama.cpp commands.
• swebench-batch-analysis/verified-5-*__20261022-*/1-final-summary-per-model-instance.log — attempt status breakdown for four October 22 batches.
• swebench-batch-evaluations-single-known-resolved-patch/20261022-015250-a/logs/evaluation/20261022-015250-a/results.json — machine-readable success confirmation.
• swebench-batch-evaluations-last-run/20261022-193725-{a,b}/logs/evaluation/*/results.json — final evaluation outcomes.
• latest-scripts/swebench_minimal_agent_v2.4.py and latest-scripts/05-batch-minimal-agent-runner-v2.4.bash — archived final implementation.

Related Resources