Skip to main content

Can LLMs Actually Investigate Cyberattacks?

· 2 min read
Khalid Alasiri
Khalid Alasiri
Cybersecurity Researcher, Entrepreneur & AI Solutions and Security Architect

A paper that crossed my feed last week is doing something I think more security-and-LLM work should do: instead of asking "is this LLM smart," it asks a narrower, more useful question — can an LLM actually investigate a security incident from raw system logs, the way an analyst would.

The paper is AuditBench (Anand et al., 2026). What stood out to me wasn't the leaderboard of which model scored highest — it's how they built the benchmark.

Grounded in what IR teams actually do​

They didn't invent arbitrary tasks. They grounded everything in four things incident-response teams actually do:

  1. Triage an alert
  2. Find persistence
  3. Trace lateral movement
  4. Spot exfiltration

Then, for three of those, they tied the scenarios directly to MITRE ATT&CK techniques, executed them for real using Atomic Red Team on live VMs, and captured the resulting telemetry through auditd and ProcMon.

So the ground truth isn't synthetic — it's "we ran this technique, here's what it actually looked like in the logs."

The framework → telemetry pipeline​

That framework-to-telemetry pipeline is the part I keep coming back to. It's a specific, hard problem: taking a behavior described at the ATT&CK level and turning it into something observable and gradable in raw system data.

That's a step I think about a lot in my own work right now, which has been moving toward using ATT&CK-style frameworks to reason about behavior — including in settings where LLMs are involved on either side of that reasoning, not just as the thing being evaluated. I'm not sharing specifics on that yet, but this paper is a useful reference point for anyone thinking about the same gap: getting from "described attacker behavior" to "something a model can actually be evaluated against."

Are raw logs even the right input?​

The other thing worth flagging: AuditBench also tests whether raw logs are even the right input format for an LLM, comparing them against a more compact, provenance-based representation. That's a systems question as much as a model question — and it's one that gets skipped in a lot of "let's just throw logs at an LLM" work.

Paper​

Anand et al., Benchmarking and Exploring the Capabilities of LLMs for Attack Investigations, arXiv:2606.10281, 2026.

→ Read on arXiv · PDF