AI Threat Hunting Starts With Data: Logs, EDR Telemetry and Quality

AI changes threat hunting by finding patterns, clustering anomalies and helping analysts write queries faster, but it can only find what your data shows. The quality of hunting depends on log coverage, consistent timestamps, normalised fields and enough retention.

Feb 28, 2025 - 10:47
Updated: 
101.8k

AI in Cybersecurity in Cybersecurity & Ethical Hacking · Explainer

SOC & Blue Team

AI Threat Hunting Starts With Data: Logs, EDR Telemetry and Quality

Quick answer: AI changes threat hunting by finding patterns, clustering anomalies and helping analysts write queries faster, but it can only find what your data shows. The quality of hunting depends on log coverage, consistent timestamps, normalised fields and enough retention. The most useful sources are endpoint telemetry from EDR or Sysmon, identity and authentication logs, DNS and proxy logs, cloud control-plane logs and network flow data. CISA's 2024 event logging guidance and NIST SP 800-92 describe what to collect. In India, CERT-In requires logs to be kept for 180 days. Fix the data first, then add AI.

Key takeaways

  • AI hunting models are limited by the telemetry they receive.
  • Endpoint, identity, DNS and cloud logs give the best hunting value.
  • Normalised fields and synchronised clocks matter more than model choice.
  • Map your data to MITRE ATT&CK data sources to find blind spots.
  • CERT-In requires 180 days of log retention for covered entities.

Table of Contents

Why Does Data Quality Decide AI Hunting Results?

Threat hunting is a hypothesis-led search for attackers that existing alerts missed. Our sibling post on how AI detects and prevents threats explains the models. This post covers what feeds them.

An anomaly model trained on endpoint logs from half your laptops will call the other half normal by silence. A language model asked to summarise authentication events cannot spot lateral movement if workstation logons were never collected. Missing data does not raise an error. It produces quiet, false confidence.

Which Data Sources Matter Most?

SourceWhat it showsHunting use
EDR or Sysmon process telemetryProcess creation, command lines, parent-child linksUnusual execution chains, living-off-the-land tools
Identity and authentication logsSign-ins, MFA events, privilege changesCredential misuse, impossible travel
DNS and web proxy logsDomains contacted, request patternsBeaconing, newly registered domains
Cloud control-plane logsAPI calls, role changes, key creationPersistence in cloud tenants
Network flow dataWho talked to whom, volume, durationExfiltration, internal scanning
Email security logsDelivery, clicks, attachmentsPhishing follow-up

The CISA Best Practices for Event Logging and Threat Detection prioritise many of the same sources. If you are new to endpoint data, read our EDR beginner guide. Microsoft documents Sysmon as a free way to add process and network events on Windows.

What Makes Log Data Good Enough for AI?

Quality factorProblem if missing
Time synchronisation (NTP)Events appear out of order; timelines break
Normalised field namesThe same user appears as three different entities
Full command linesMalicious arguments are invisible
Asset and identity contextAI cannot tell a server admin from a sales laptop
RetentionSlow intrusions fall outside the search window

NIST SP 800-92 covers log generation, storage and protection. Shared schemas such as OCSF help normalise fields across vendors. The CERT-In Directions require covered entities to keep ICT logs for 180 days within India and to report incidents within 6 hours.

A quick check that Windows hosts are syncing time:

w32tm /query /status

How Do You Find Blind Spots?

List the MITRE ATT&CK data sources you collect, such as Process Creation, Logon Session and Network Traffic Flow. Then compare them with the techniques most relevant to your threat model. Techniques with no supporting data source are hunts you cannot run, with or without AI. Also measure how many assets actually send logs, not just how many should.

Where Does AI Actually Help?

  • Baselining: learning normal behaviour per host or user.
  • Clustering: grouping rare command lines so analysts review dozens, not thousands.
  • Query drafting: turning a hypothesis into a SIEM query for human review.
  • Summarising: condensing a timeline for the incident report.

Every AI lead still needs an analyst to confirm it and follow the response process in NIST SP 800-61 Rev. 3.

How Should a Team Get Started?

  1. Inventory log sources and the share of assets reporting.
  2. Fix time sync, field naming and command-line capture.
  3. Meet the 180-day retention requirement.
  4. Map data to ATT&CK and write three hypotheses you can test.
  5. Add AI for baselining and triage, then measure analyst time saved.

For hands-on practice, ask about our SOC analyst and threat hunting training.

References

Frequently Asked Questions

It needs endpoint telemetry, identity and authentication logs, DNS and proxy logs, cloud control-plane logs and network flow data, ideally normalised and time synchronised.

No. A model cannot see activity that was never recorded, so gaps in telemetry become blind spots that look like normal behaviour.

Many attacks use legitimate tools, and only the full command line shows the suspicious arguments that separate abuse from normal administration.

The CERT-In Directions of April 2022 require covered entities to keep ICT system logs for a rolling 180 days within Indian jurisdiction.

Sysmon is a free Microsoft Sysinternals tool that records detailed process, network and file events on Windows for monitoring and hunting.

They show which telemetry is needed to observe each technique, so teams can compare what they collect against the threats they care about.

EDR is a strong base, but identity, DNS, cloud and network data are needed to follow attackers who move beyond a single endpoint.

Inventory your log sources, check how many assets actually report, and fix time synchronisation and field naming so the tool has reliable input.

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Wow Wow 0
Sad Sad 0
Angry Angry 0
Vaishnavi

Vaishnavi is a skilled tech professional at the Ethical Hacking Training Institute in Pune, responsible for managing and optimizing the technical infrastructure that supports advanced cybersecurity education. With deep expertise in network security, backend operations, and system performance, she ensures that practical labs, online modules, and assessments run smoothly and securely. Her behind-the-scenes contributions play a vital role in delivering a seamless and secure learning experience for aspiring ethical hackers.