Security teams often know the type of bug they are hunting. Someone still has to open the company's code and find the files that contain it. That search takes hours on a large project.
Most published artificial intelligence tests skip the search. They paste the bad snippet into the model and ask it to label the bug or write a fix. Cisco Foundation AI published a different test on September 14. The paper is the Vulnerability Localization Benchmark, VLoc Bench, by Aman Priyanshu, Supriti Vijay, Kimia Majd, and colleagues (Priyanshu et al.). I wrote about Cisco's Antares models in July when the first weights went up. This paper is the scoreboard, plus a second job: what the model says after developers have shipped the fix.
The model is told the kind of bug, then it searches the company code
A repository is the folder that holds a product's source code. Each test starts from a public GitHub security notice. The authors kept one copy of the folder from before the developers fixed the bug, and one copy from after the fix. They check whether the model named the same code files the developers later changed. Test files and help pages do not count.
The model does not get the full report of the bug. It gets a short label for the type of bug, such as "the program does not check what a user typed" or "the program lets someone open a file outside the allowed folder." Those labels come from the Common Weakness Enumeration list. The model may search and read files up to fifteen times. Then it hands back a list of file names, or it says it found nothing.
There are 500 cases from 290 real projects, in six software ecosystems (npm, pip, Maven, Go, Rust, Composer), covering 147 types of bugs. Seventy-eight percent of the cases have a public vulnerability number. Most sit in the high or medium severity bands.
That setup is closer to real work than most AI security tests. A notice at a company usually names the kind of bug, not the file. Someone still has to open the project and find it. The two saved copies let Cisco score both parts of that job: did the model point at the files the developers later changed, and did it stop pointing after those files were repaired. Fifteen searches is the limit. A person can keep looking. The model has to choose. That is why a 0.223 score here is a result on work that looks like yours, not on a puzzle with the answer already circled.
Cisco's small model almost matches GPT-5.5 at finding the files
Parameter count is a rough size for a model. Three billion is small next to the giant general models people use for chat.
These are the scores for how well each system pointed to the files that had the bug. A perfect score is 1.0. That means it named every right file and no extra files.
- GPT-5.5, strongest setting: 0.229
- Cisco Antares-3B: 0.223
- Cisco Antares-1B: 0.209
- GLM-5.2, 753 billion parameters: 0.186
- Gemini 3 Pro: 0.152
- Semgrep: 0.086
- Semgrep pack aimed at known bug types: 0.052
- CodeQL: 0.023
- Horusec: 0.020
- IBM Granite 4.0, before Cisco trained it for this search: 0.000
Source: Priyanshu et al., Table 2.
Antares is Cisco Foundation AI's family of models for this file search. The 350 million and 1 billion parameter versions are public downloads on Hugging Face. I covered the cost claim in July. Cisco said running all 500 cases on one high-end graphics card costs under a dollar, against a much larger bill to run GPT-5.5 on the same set (Bellamkonda, 22 July 2026).
Training for the search closed most of the gap with the frontier model. Llama 3.3, a 70 billion parameter general model, scores 0.012 on the same hunt.
One hundred ninety of the 500 cases, 38.4 percent, beat every system in the paper. The authors say the size of the project predicts difficulty more than the type of bug. Scores drop about thirteen times from the smallest projects to the largest.
After the fix, Cisco's model says it found nothing on 3 or 4 of every 100 cleaned projects
The second half of the test uses the same project after the developers have shipped the fix. The right answer is that this recorded bug is gone. The score is how often the system reports nothing.
Picture 100 of those cleaned projects. Antares-3B says it found nothing on about three or four of them. On the other 96 or 97 it still names files, as if the bug were still there. That 3.4 percent is how often it does the second job correctly.
That number matters more than the 0.006 gap with GPT-5.5. A 0.223 score can shorten the list a person opens when a new notice arrives. A 3.4 percent score on the second job means the same tool will keep opening tickets on code that is already fixed. If you let it halt work, it will halt work on code that is already fixed.
GPT-5.5 at the high setting reports nothing on 27.9 percent of those cleaned projects, which means it still names files on about 72 percent of them. Antares-1B reports nothing on 0.8 percent. GPT-5 Nano, which scores 0.024 finding files, reports nothing on 86.8 percent. The untrained Granite model reports nothing every time, because it never names files in the first place. Semgrep reports nothing on 91.2 percent of cleaned projects and scores 0.086 when the bug is still there (Priyanshu et al., Table 3).
The authors plotted the find-score against the second-job score for 31 systems, 27 language models and four scanners. No system sits high on both.
Use the file names to decide where a person should look first
A 0.223 score can cut the number of files a person has to open. That is the useful job. Do not let the model decide whether new code can ship.
Cisco's own model cards describe the product as a tool that ranks files for a known type of bug. They leave explaining the bug, writing an attack, and writing the fix outside the product. Outside testers who ran the 1 billion parameter model saw the same habit on cleaned projects. The model still answers the question of where this type of bug would live.
Ask before this tool can halt work
If we attach this search to incoming security notices, who reads the file list before a ticket is opened? On last quarter's already-fixed projects, how often would this search have opened a ticket again?
The test, the leaderboard, and the paper sit at cisco-foundation-ai.github.io/vulnerability-localization-benchmark. The paper is Priyanshu et al., arXiv 2609.15939, 14 September 2026.
Sources
Priyanshu, Aman, et al. "Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale." arXiv, 14 Sept. 2026, arxiv.org/abs/2609.15939.
Cisco Foundation AI. "VLoc Bench." 2026, cisco-foundation-ai.github.io/vulnerability-localization-benchmark.
Vijay, Supriti, et al. "Antares: Foundation Models for Agentic Vulnerability Localization." arXiv, 3 Aug. 2026, arxiv.org/abs/2608.02407.
Cisco. "Introducing Antares: Highly Efficient Open Weight AI Models for Vulnerability Localization." Cisco Blogs, 21 July 2026, blogs.cisco.com/ai/introducing-antares-the-most-efficient-open-weight-ai-models-for-vulnerability-localization.
Bellamkonda, Shashi. "Cisco's Antares Finds Vulnerabilities Cheaply." shashi.co, 22 July 2026, www.shashi.co/2026/07/ciscos-antares-finds-vulnerabilities.html.
