AI in Malware Detection: Endpoint Models, False Positives and Evasion

AI helps detect malware on endpoints by scoring files and behaviour with machine learning models instead of relying only on signatures. A lightweight model on the device judges files quickly, cloud models give a second opinion with wider data, and behaviour models watch process activity for patterns such as mass file encryption.

Mar 06, 2025 - 16:02
Updated: 
101.7k

AI in Cybersecurity in Cybersecurity & Ethical Hacking · Explainer

SOC & Blue Team, Cyber Threats, Scams & Online Safety

AI in Malware Detection: Endpoint Models, False Positives and Evasion

Quick answer: AI helps detect malware on endpoints by scoring files and behaviour with machine learning models instead of relying only on signatures. A lightweight model on the device judges files quickly, cloud models give a second opinion with wider data, and behaviour models watch process activity for patterns such as mass file encryption. This catches many new variants, but it also blocks some legitimate software, and attackers try to evade or disable it. Good results come from tuning: handling false positives through proper submissions and narrow exclusions, enabling tamper protection and keeping humans in the loop for response.

Key takeaways

  • Endpoint protection now mixes signatures, local ML, cloud ML and behaviour monitoring.
  • Behaviour detection can catch threats that change their files.
  • False positives are normal; handle them without opening wide gaps.
  • Attackers target the security agent itself, so tamper protection matters.
  • Judge products on your own testing, not marketing claims.

Table of Contents

How Does AI Detect Malware on Endpoints?

LayerWhat it doesTrade-off
Signatures and reputationMatches known bad files and hashesMisses new variants
Local ML modelScores files on the device, works offlineSmaller model, less context
Cloud MLChecks metadata against larger models and global telemetryNeeds connectivity, shares data with vendor
Behaviour monitoringWatches process, file, registry and network actionsActs after code starts running

Microsoft documents this layered design for Defender Antivirus cloud protection, where suspicious files can be held briefly while cloud models decide. Our article on automated malware classification explains how such models are trained.

Why Does Behaviour Detection Matter?

Attackers can change a file’s bytes endlessly, but what malware does is harder to change. Ransomware must read and rewrite many files. Credential theft must touch credential stores. Behaviour models link these actions into a chain and can stop the process or roll back changes. This is also how defenders catch threats with no file at all, as explained in our post on fileless malware.

Why Do AI Detectors Produce False Positives?

Models judge likelihood, and some legitimate software looks suspicious. Common triggers include in-house tools without code signing, packed installers, admin scripts that change many settings, and new builds with no reputation yet. Benign software far outnumbers malware, so even a small error rate produces visible blocks.

How Should You Handle a False Positive?

  1. Confirm it is really benign: check the source, signer and hash with the owning team.
  2. Submit the file to the vendor for review, as Microsoft describes in its false positive guidance.
  3. If you must allow it, use the narrowest option: a specific hash or signed publisher, not a whole folder or drive.
  4. Record the exception with an owner and review date.
  5. Code-sign internal tools so they build reputation.

Broad folder exclusions are a common gap that attackers look for.

How Do Attackers Try to Evade AI Detection?

Evasion approachDefensive response
Disabling or tampering with the agent (T1562.001)Tamper protection, alerts on agent health
Using built-in admin toolsBehaviour rules, application control, logging
Packing and obfuscationMemory scanning, behaviour monitoring
Adversarial changes to filesMultiple model types, retraining
Abusing broad exclusionsRegular exclusion audits

NIST AI 100-2 classifies evasion attacks against ML models in more depth.

How Do You Harden AI-Based Endpoint Protection?

  • Turn on cloud protection, behaviour monitoring and tamper protection.
  • Limit local admin rights so malware cannot easily stop the agent.
  • Send endpoint alerts to a monitored SIEM and keep logs 180 days, as CERT-In requires.
  • Back up offline and test restores, following the CISA ransomware guide.
  • Follow the handling steps in NIST SP 800-83.

How Should You Evaluate an AI Detection Product?

Pilot it on your own systems and software. Measure false positives on your line-of-business apps, alert clarity and response options. Public results such as MITRE ATT&CK Evaluations show detection behaviour but do not rank vendors. Run safe simulations only with written authorisation, as required under the IT Act 2000, Sections 43 and 66. To learn endpoint defence hands-on, ask about our SOC and endpoint security training.

References

Frequently Asked Questions

Machine learning models score files and process behaviour for signs of malice, working alongside signatures, reputation checks and cloud lookups in modern endpoint protection.

They do different jobs. Signatures reliably catch known threats, while ML and behaviour models catch more new variants, so modern products combine both approaches.

Unsigned, packed or brand-new software can look suspicious to models. Confirm the file is safe, submit it to the vendor and use a narrow allow rule if needed.

Broad folder exclusions are risky because attackers can drop malware there. Prefer exclusions by hash or signed publisher, with an owner and review date.

Attackers often try, especially with admin rights. Tamper protection, limited local admin rights and alerts on agent health reduce this risk.

It watches what programs do, such as mass file changes or credential access, and stops activity that matches malicious patterns even if the file is new.

Many do send file metadata or samples for cloud analysis. Review the vendor’s settings and your DPDP Act obligations before enabling sample submission.

Pilot them on your own environment, measure false positives on business apps and review alert quality, rather than relying on vendor claims or rankings.

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Wow Wow 0
Sad Sad 0
Angry Angry 0
Vaishnavi

Vaishnavi is a skilled tech professional at the Ethical Hacking Training Institute in Pune, responsible for managing and optimizing the technical infrastructure that supports advanced cybersecurity education. With deep expertise in network security, backend operations, and system performance, she ensures that practical labs, online modules, and assessments run smoothly and securely. Her behind-the-scenes contributions play a vital role in delivering a seamless and secure learning experience for aspiring ethical hackers.