SWE-Bench+: Enhanced LLM Coding Benchmark
SWE-Bench is widely used to evaluate whether large language models can resolve real-world GitHub issues, yet the quality of its benchmark instances remains underexplored. We present SWE-Bench+, an enhanced benchmark framework that improves evaluation reliability by addressing two risks: solution leakage in issue descriptions and weak tests that allow plausible but incorrect patches to pass. We analyze 217 commonly resolved issues from SWE-Bench Lite and SWE-Bench Verified across three top-performing agents, yielding 651 model-generated patches. Our analysis identifies five recurring quality-problem patterns under solution leakage and weak tests. We further develop SoluLeakDetector, which detects solution-leaking content, and TestEnhancer, which strengthens validation tests. We find that 60.83% of commonly resolved issues contain solution leakage and 77.88% are problematic overall. SoluLeakDetector achieves 80.45% accuracy, while TestEnhancer identifies plausible patches for 97.11% of weak-test issues and reduces average resolution rates by 27.00 percentage points on Lite and 36.27 percentage points on Verified.
Tue 7 JulDisplayed time zone: Eastern Time (US & Canada) change
12:00 - 12:30 | Benchmarks, Datasets, and Evaluation of AIware Benchmark & Dataset Track / ArXiv Track / Main Track at MB 1.210 Chair(s): Mohammad Hamdaqa Polytechnique Montreal | ||
12:00 5mTalk | ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation Benchmark & Dataset Track Yeheng Chen Shanghai Jiao Tong University, Chaoxiang Xie Hohai University, Yuling Shi Shanghai Jiao Tong University, Wenhao Zeng Shanghai Jiao Tong University, Yongpan Wang Shanghai Jiaotong University, CN, Hongyu Zhang Chongqing University, Xiaodong Gu Shanghai Jiao Tong University DOI | ||
12:05 5mTalk | SWE-Bench+: Enhanced LLM Coding Benchmark Benchmark & Dataset Track Haoran Xue York University, CA, Reem Aleithan York University, Canada, Nafid Enan York University, CA, Gias Uddin York University, Canada, Song Wang York University DOI | ||
12:10 5mTalk | Do Agents Dream of Root Shells? Partial-Credit Evaluation of LLM Agents in Capture the Flag Challenges Benchmark & Dataset Track Ali Al-Kaswan Delft University of Technology, Netherlands, Maksim Plotnikov Delft University of Technology, NL, Maxim Hájek Delft University of Technology, NL, Roland Vízner Delft University of Technology, NL, Arie van Deursen TU Delft, Mali Izadi Google & TU Delft DOI | ||
12:15 5mTalk | A Dataset of Agentic AI Coding Tool Configurations Benchmark & Dataset Track Matthias Galster University of Canterbury, Seyedmoein Mohsenimofidi Heidelberg University, Levi Böhme Universität Bayreuth, DE, Jai Lal Lulla Singapore Management University, Muhammad Auwal Abubakar Otto-Friedrich Universität Bamberg, DE, Christoph Treude Singapore Management University, Sebastian Baltes Heidelberg University DOI | ||
12:20 5mTalk | AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub Benchmark & Dataset Track DOI Pre-print | ||
12:25 5mTalk | TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution ArXiv Track Jiale Amber Wang University of Waterloo, Kaiyuan Wang Google, Inc., Pengyu Nie University of Waterloo Pre-print | ||