Agentless: Demystifying LLM-based Software Engineering Agents

BENCHMARK CRITICISM - directly relevant to benchmark validity concerns. The paper: (1) Shows that a simple 'Agentless' approach outperforms complex autonomous agents on SWE-bench Lite (32.00% vs. lower for agents). (2) 'Manually classified the problems in SWE-bench Lite and found problems with exact ground truth patch or insufficient/misleading issue descriptions' - this is direct evidence of benchmark contamination. (3) Constructs SWE-bench Lite-S by excluding problematic issues 'to perform more rigorous evaluation and comparison.' This paper is important because it demonstrates that: (a) Benchmark results can be inflated by problematic test cases. (b) Simple baselines can outperform complex methods when the benchmark is cleaned. (c) Benchmark contamination is a real issue that affects vendor claims.