September 26, 2026
AI coding assistants have become indispensable tools for developers, yet a groundbreaking study exposes a critical limitation in their performance. While these advanced systems excel at identifying the correct files within a codebase, they frequently fall short when it comes to isolating the precise lines of code that require modification. This gap between file-level accuracy and line-level precision could explain why many AI-generated fixes still miss the mark.
A team of researchers from Shanghai Jiao Tong University and other institutions developed SWE-Explore, a new benchmark designed to dissect the initial phase of AI-driven code repair. Unlike traditional evaluations that focus solely on whether a bug was fixed, SWE-Explore isolates the search process, assessing how effectively AI agents identify relevant code sections before attempting repairs. The findings reveal a stark contrast between an agent’s ability to navigate a project’s structure and its capacity to zero in on the exact lines that matter.
The SWE-Explore benchmark introduces a novel approach to evaluating AI coding assistants by separating the search phase from the repair phase. Traditional benchmarks often measure success based on whether a bug is resolved, without considering whether the AI even examined the correct code. This oversight can mask underlying weaknesses in how these systems locate and interpret relevant information.
To establish a reliable reference for what constitutes “relevant” code, the research team analyzed over 800 bug-fix scenarios from successful runs by leading AI models, including GPT-5.4, Gemini 3 Pro, Claude Sonnet 4.6, and Kimi K2.6. By tracking which files and lines these models reviewed before implementing fixes, the team identified patterns in the code sections that consistently proved useful. These insights formed the basis of SWE-Explore’s evaluation criteria, ensuring a more nuanced assessment of AI performance.
The dataset underpinning SWE-Explore spans 203 open-source projects across ten programming languages, with Python accounting for 64% of the tasks. This diversity ensures the benchmark reflects real-world coding challenges, from simple scripts to complex software systems. The inclusion of languages like Go, JavaScript, and Rust further broadens the scope, making SWE-Explore a comprehensive tool for evaluating AI coding agents.
The study pitted traditional keyword-based search methods against nine AI-driven systems, including general-purpose coding agents like Claude Code, Codex, and OpenHands, as well as specialized research tools designed for code localization. The results were revealing: while keyword searches barely outperformed random chance, AI agents demonstrated a clear advantage by systematically exploring the codebase rather than relying on static keyword matches.
However, this advantage was largely confined to the file level. AI agents consistently identified the correct source files and ranked them highly, but their performance plummeted when evaluated at the line level. On average, general-purpose agents covered only 14% to 19% of the lines that were critical to resolving the bugs. This discrepancy highlights a fundamental limitation in how these systems process and prioritize code.
To determine whether stronger language models could bridge this gap, the researchers tested the same agent architecture with six different models from OpenAI, Anthropic, Google, Moonshot, and Zhipu. While the GPT family led the pack, the overall pattern remained unchanged: file-level accuracy consistently outpaced line-level precision. This suggests that the issue isn’t merely a matter of model strength but rather a deeper challenge in how AI agents interpret and contextualize code.
Among the specialized systems, CoSIL stood out for its unique approach. By treating code as a network of interconnected building blocks, it achieved significantly higher line-level coverage than its counterparts. Other tools, like AutoCodeRover, demonstrated precision but were overly conservative, while OrcaLoca minimized noise at the cost of missing many relevant lines. These trade-offs underscore the complexity of designing AI systems that can both locate and understand code effectively.
One of the most compelling findings from the study is the existence of a minimum context threshold for successful bug fixes. In a controlled experiment, the researchers manipulated the amount of relevant code provided to the repair model, ranging from 0% to 100% of the core regions, sometimes supplemented with irrelevant code. The results were striking: for simpler tasks, repairs consistently failed unless at least 50% of the critical code lines were included in the context.
The success rate only began to improve once coverage exceeded 75%, indicating that AI-driven repairs don’t benefit from incremental context. Instead, they require a critical mass of relevant information before they can generate accurate fixes. This threshold effect suggests that even the most advanced AI coding assistants may struggle with tasks where the necessary context is fragmented or poorly defined.
The study’s findings underscore a fundamental challenge in AI-driven code repair: the ability to identify the right file is only half the battle. Without precise line-level accuracy, AI agents risk generating fixes that are either incomplete or entirely off-target. This limitation could explain why many developers still find themselves manually reviewing and adjusting AI-generated patches, despite the impressive capabilities of modern coding assistants.
For AI coding tools to become truly reliable, they must evolve beyond file-level navigation and develop a deeper understanding of code semantics. This could involve integrating more sophisticated static analysis techniques, leveraging contextual embeddings, or even incorporating human feedback to refine their search strategies. Until then, the gap between finding the right file and fixing the right line will remain a critical hurdle in AI-assisted development.
SWE-Explore is a benchmark developed to evaluate how effectively AI coding agents locate relevant code sections before attempting repairs. It isolates the search phase to provide a more detailed assessment of AI performance.
AI agents excel at identifying the correct files but often fail to pinpoint the exact lines that need modification. This is due to limitations in how they process and contextualize code, rather than a lack of model strength.
The dataset was built by analyzing successful bug-fix attempts from leading AI models. Researchers tracked which files and lines these models reviewed before implementing fixes, using this data to define relevant code sections.
The study found that AI-driven repairs typically fail unless at least 50% to 75% of the critical code lines are included in the context. Below this threshold, fixes are unlikely to succeed.
While general-purpose agents like Claude Code and Codex performed similarly, the specialized system CoSIL achieved the highest line-level accuracy by treating code as a network of interconnected blocks.
The SWE-Explore benchmark sheds light on a critical weakness in AI coding assistants: their ability to locate the right files often doesn’t translate to identifying the exact lines that need fixing. This gap in line-level precision could explain why many AI-generated patches still require manual intervention, despite advancements in model capabilities.
The study’s findings highlight the need for more sophisticated approaches to code search and repair. While AI agents have made significant strides in navigating codebases, their performance at the line level remains inconsistent. Future developments may focus on integrating static analysis, contextual embeddings, or human feedback to bridge this gap and make AI-driven coding tools more reliable for developers.
