BigSleep LLM Agent Finds 45 V8 Vulnerabilities, Fuzzilli 37, With 25% Overlap

Over six months, from April to September 2026, the BigSleep LLM agent found 45 working vulnerabilities in V8, while the generative fuzzer Fuzzilli found 37. Google runs both tools in production to check fresh code in Chrome's JavaScript engine. The Project Zero team compared their results and drew two conclusions. The overlap between the findings is small: about 25% in the bug tracker (20 of 81 working reports). In a reconstructed experiment on historical bugs, it reached roughly 60%. Either way, a significant share of vulnerabilities is found by only one of the two approaches.

How 441 Reports Became 45 Vulnerabilities

A raw count from the Chromium bug tracker hotlists gives BigSleep a slight edge: 441 reports against 316 for Fuzzilli. But early on, BigSleep found many configuration and integration problems, such as a blocklist for unsafe flags that did not work (crbug.com/503869331). Bugs like these have almost no real security impact.

If you keep only type:vulnerability, the picture flips: 178 for Fuzzilli and 95 for BigSleep. This number is inflated too. Some reports are duplicates or were closed as WAI ("works as intended"). A simple filter on the fixed status does not help either. For example, Clusterfuzz filed and closed the Fuzzilli bug crbug.com/511935486 on the same day because the test case stopped reproducing. It is almost certainly a duplicate of crbug.com/511210507, but the stack-trace crash signature differed slightly, so the system did not merge them. No engineer ever touched that bug.

So the team counted only "working" reports: not duplicates, not WAI, and with at least one fix commit. For the Type-Vuln category, that gave 45 for BigSleep against 37 for Fuzzilli.

Overlap: A

Quarter in the Tracker, About 60% in the Experiment

To count shared findings, the team deduplicated bugs in four steps:

• explicit duplicates, where one system's report is marked as a duplicate of the other's, such as crbug.com/522410841 and crbug.com/522860268; • implicit duplicates, where the team looks for the fix commit of "non-working" reports and merges the bugs tied to it. The link can be multi-way: a BigSleep bug and a Fuzzilli bug can both duplicate a third; • duplicates from the Clusterfuzz database that never reached the tracker; • different bugs closed by a single CL (for example, four bugs in crrev.com/c/7965120), which also count as duplicates.

The result: 20 shared bugs out of 81, with slightly more than half found by Fuzzilli first.

This count understates the overlap, so the team ran an experiment. They took about 1150 historical Fuzzilli bugs from 2023 to mid-2026. From these they kept about 600 vulnerabilities, then 132 bugs reproducible in a public debug+asan build of d8 on x64 without special flags. After deduplication, 112 remained. For each one, they found a likely introducing commit, mostly by bisection, and ran the agent 25 times with the task of finding bugs in that commit and confirming them with a test case that crashes the build. That adds up to 2800 runs, on the production BigSleep configuration of the time with the Gemini 3.6 flash model.

The result: about 62% of the original bugs were rediscovered, another 12% or so of commits yielded a different, already fixed bug, and for 26% nothing was found. The setup is deliberately stacked against the agent: the commit may have been identified incorrectly, and in reality a bug can also be caught through a later commit in the same code.

Memorization, Difficulty, and What Remains for the Fuzzer

The team separately checked whether the model recalls bugs from its training data. Gemini 3.6 flash has a knowledge cutoff of March 2026. Chrome bugs were then made public roughly three months after the fix, so the effective cutoff for Fuzzilli bugs falls in January 2026. The team found no statistically significant memorization on bugs from before 2026. For other models, the check should be repeated.

For bug "difficulty", meaning the time it took Fuzzilli to find the bug, the dependence is statistically significant, though small in magnitude. Its direction is opposite to the ideal scenario, in which an LLM easily picks up what the fuzzer finds hard and the fuzzing budget can be reallocated.

The reverse experiment (using the fuzzer to find BigSleep's past bugs) was not run. A fuzzer cannot be pointed at a specific commit as easily. Moreover, most of BigSleep's older bugs were found after months or years of fuzzing vulnerable builds. In production the fuzzer already has a head start: BigSleep runs once a day and reports crashes with a delay of 1–2 days.

Practical Takeaways for Security Teams

The cost per finding cannot be compared yet. Clusterfuzz has been optimized for years, while BigSleep has had no comparable work, and neither system has been tuned for cost per bug.

Takeaways from the methodology:

• run both fuzzing and agentic analysis: their blind spots differ noticeably; • give the agent the diff of a specific commit and require a PoC. Stopping at the first crash limits token spend; • check the PoC on builds from before and after the fix to tell the target bug from a different one; • use sanitizer builds and a "safe flags only" mode; • count metrics only from working reports with fix commits, and deduplicate them together with Clusterfuzz; • when evaluating on historical bugs, compare the success rate before and after the model's cutoff date to rule out memorization.