On LLMs and vulnerability research
I actually wanted to write about this for months now, but it has been difficult, because the space was moving too fast, that it was impossible to catch up on the progress happening, almost unreal. Every few weeks the topic changed because the models changed.
But, anyway, now I think the direction is pretty clear. LLMs have become genuinely exceptional for vulnerability research, and when I say research, I mean cutting-edge, elite level vuln research, and not just explain this CVE to me or write a semgrep rule., now you can throw a fairly nasty codebase at a good reasoning model and it will actually get shit done, spit out vuln some really complex vulns, with almost 0 false positives (atleast at the code level they won't be hallucinations).
which is… kinda wild.
I remember, not long ago, people were adamant on how intelligence will eventually plateau (because LLMs are next-token prediction engines), well, we are way past that argument, and it now feels pointless to me. sure, technically, yes. but once a system can spend thousands of tokens reasoning through some very complex problem, implementation detail stops being a useful description of the capability.
A few things basically arrived at the same time, which made the new (almost astronomical) intelligence possible, it started with, huge context windows, then we got much better reasoning, and eventually tools like codex / claude code that let the model actually move around a repo and test things.
put those together and you suddenly have something that looks suspiciously like a staff-level security researcher who reads insanely fast and never gets tired.
The part i find most interesting is that vulnerability research is already pretty compositional, and a lot of bugs that feel new are really familiar ideas interacting in a weird place. some assumption breaks, two components disagree, some data crosses a boundary it probably shouldn’t, whatever.
humans are good at noticing those combinations because we build intuition over years.
models are getting pretty good at it because… well, they’ve seen an absurd amount of code and they can now spend actual compute connecting the pieces.
obviously the model still needs direction, this is probably the part people underestimate the most.
find vulnerabilities in this repo is a terrible instruction. the search space is basically infinite. give it a threat model, tell it what assumptions matter, point it toward the scary code, give it some historical context, totally different result.
so yeah, prompting matters, but i almost don’t want to call it prompting anymore. it’s more like giving a researcher the right mental model before they start looking.
and then there’s the non-determinism thing. AI SASTs sell this as their "moat", they claim to bring order to chaos. People treat this like a fatal flaw, but for vuln research i’m not sure it is. if you run the same model five times and it explores five slightly different hypotheses… that’s actually useful.
the bug is still there, you’re just searching for it from different directions. And honestly, this also makes me less excited about some of the giant AI-security architectures people have been building.
a couple years ago everyone was making elaborate RAG systems, multi-agent setups, custom orchestration, seventeen tools talking to each other, etc etc. some of that is still useful, but the base models are eating a LOT of that complexity.
When the model can read most of the repo, reason for a long time, use a terminal, inspect files, run tests and revise its own theory… you start wondering how much machinery you really need around it.
which is why i think vulnerability research is going to get much more accessible (maybe even it's a solved problem now), much faster than people expect.
not every kind of security research. obviously, but a surprisingly large chunk of the work is basically, understand a system, form hypotheses, trace them carefully, verify what happens, and models are getting VERY good at exactly that loop. So the role of the researcher changes a bit, i.e. less time manually digging through every boring path, and more time deciding which paths are worth digging into.
and tbh, that already feels like it’s happening.