# 10 malware detection techniques and where each one works best

*Threat Detection*

No single detection method can identify every malicious file, process, script, or behavior. Learn how ten complementary techniques uncover threats across different stages of execution.

![Author Karthik Pandian](https://www.manageengine.com/ems/images/tools/employee/karthik-p-maketer.png)

**Karthik Pandian** · Product Marketer, ManageEngine  
Published July 15, 2026 · 10 min read  
Last updated July 15, 2026

Malware detection is no longer limited to comparing files against a database of known signatures. Modern attacks may execute through scripts, trusted system utilities, injected processes, encrypted payloads, or memory-resident code. Detecting these threats requires multiple techniques, each analyzing different telemetry and operating at a different stage of execution. Modern [malware detection](https://www.manageengine.com/malware-protection/malware-detection.html) software combines file analysis, process monitoring, memory inspection, network activity, and behavioral correlation instead of relying on a single indicator.

This guide compares ten malware detection techniques based on:

- The telemetry they analyze
- The threats they are best suited to detect
- The stage at which they operate
- Their principal limitations

## Why malware detection shifted from files to behavior

Traditional malware detection focused largely on identifying known malicious files through signatures, hashes, and file attributes. These methods remain effective because they can detect previously analyzed threats quickly and with relatively low processing overhead.

However, attackers increasingly use obfuscation, [polymorphic code](https://www.manageengine.com/malware-protection/articles/polymorphic-malware.html), scripts, trusted system utilities, process injection, and memory-based execution to reduce the number of obvious malicious files available for inspection.

MITRE ATT&CK documents techniques such as encrypted payloads, polymorphic code, reflective DLL injection, and process hollowing that can make conventional file analysis less effective.

Modern detection therefore looks beyond what a file is and examines what processes, scripts, users, and applications do. Process ancestry, command lines, memory activity, file changes, and network connections can expose malicious activity even when no known signature is available.

**Real-world example**

**[Volt Typhoon](https://www.cisa.gov/resources-tools/resources/identifying-and-mitigating-living-land-techniques):** CISA documented how Volt Typhoon used legitimate administrative tools, valid credentials, and living-off-the-land techniques to blend with normal network activity. Because much of the activity involved native tools and processes, defenders needed behavioral context rather than relying only on malicious-file indicators.

## 1. Signature and hash-based detection

[Signature-based detection](https://www.manageengine.com/malware-protection/articles/signature-based-detection.html) compares files against known malicious indicators, including cryptographic hashes, byte sequences, malware family signatures, and pattern-based rules.

When a file matches a known indicator, it can be identified before or during execution. This makes signature detection fast and effective against established malware families and previously observed payloads.

**Works best for:**  
Known malware families, previously identified malicious files, and high confidence early detection.

**Main limitation:**  
Signatures can only identify threats that have already been analyzed and added to the detection database. Repacking, encrypting, recompiling, or slightly modifying a file can produce a different hash without significantly changing its behavior. MITRE ATT&CK identifies polymorphic code, encryption, compression, and junk-code insertion as methods attackers may use to make malicious files more difficult to detect or analyze. Signature matching should therefore remain one detection layer, but not the only one.

## 2. File reputation analysis

File reputation analysis evaluates whether a file is known, trusted, uncommon, suspicious, or malicious based on local and cloud intelligence.

Unlike a direct signature match, reputation analysis considers broader context. A newly observed unsigned executable downloaded from an unusual source may deserve closer inspection even when it does not match a known malware hash. Reputation data can help security tools make faster decisions about unfamiliar files and prioritize them for additional analysis.

**Works best for:**  
Newly encountered files, rare or low prevalence executables, and suspicious downloads and attachments.

**Main limitation:**  
A file being new or uncommon does not make it malicious. Legitimate internal applications and newly released software may initially have little reputation data. Attackers may also abuse trusted certificates, compromise legitimate software, or use widely available tools with established reputations. Reputation must therefore be evaluated alongside file origin, signer information, execution context, and runtime behavior.

## 3. Static code analysis

Static analysis examines a file before it runs. It can inspect executable headers, imported functions, embedded strings, digital signatures, section characteristics, and compiler artifacts.

**MITRE ATT&CK reference**

Combinations of Windows APIs such as VirtualAllocEx, WriteProcessMemory, and CreateRemoteThread may indicate that an executable has the capability to inject code into another process. [MITRE ATT&CK](https://attack.mitre.org/techniques/T1055/001/) documents these APIs as one method used to perform DLL injection.

[Static analysis](https://www.manageengine.com/malware-protection/articles/malware-analysis.html) may also identify packed or encrypted sections, suspicious imports, embedded commands or network indicators, and anti-analysis functionality.

**Works best for:**  
Detecting suspicious files before execution, identifying known code patterns, and finding embedded configuration data.

**Main limitation:**  
Static indicators are not conclusive on their own. Legitimate applications may use APIs associated with debugging, instrumentation, or software management. Malware can also hide its functionality through packing, encryption, dynamic API resolution, or code assembled only during execution. Static analysis is therefore most useful when paired with reputation and runtime monitoring.

## Additional malware detection techniques used in modern security

No single detection engine can identify every threat. Modern endpoint security combines multiple complementary techniques that strengthen detection before, during, and after malware execution. These methods improve detection accuracy, reduce false positives, and help uncover threats that evade traditional scanning.

### Heuristic analysis

Heuristic detection identifies suspicious files by examining their structure, instructions, API usage, and execution logic instead of relying solely on known signatures. It helps detect previously unseen malware variants, polymorphic malware, and zero-day threats by identifying characteristics commonly associated with malicious software.

**Works best for:**  
Unknown malware, zero-day threats, packed or obfuscated executables.

**Main limitation:**  
May generate false positives if legitimate software exhibits unusual behavior.

### Dynamic analysis and sandboxing

Static inspection cannot always reveal malicious intent. Dynamic analysis executes suspicious files inside an isolated sandbox to observe their behavior safely. During execution, the sandbox monitors process creation, registry changes, file modifications, memory activity, and network communication to determine whether the sample behaves maliciously.

Modern malware increasingly incorporates sandbox-evasion techniques, making dynamic analysis most effective when combined with behavioral monitoring and runtime detection.

**Real-world example (2026):** Picus Labs documented [LummaC2](https://www.manageengine.com/malware-protection/adversaries/lumma-stealer.html) using mouse-movement analysis to detect automated sandboxes and avoid executing its malicious payload.

### Allowlisting and blocklisting

Allowlisting permits only trusted applications to execute, while blocklisting prevents known malicious files, hashes, or applications from running. Together, they reduce attack surface by restricting unauthorized software execution.

Although blocklisting remains effective against known malware, allowlisting offers stronger protection against unknown threats by preventing unapproved applications from executing in the first place. However, both approaches require continuous maintenance to remain effective.

### Entropy-based detection

Packed and encrypted malware often exhibits unusually high entropy because its contents appear random. Entropy analysis helps identify executables that may have been compressed, encrypted, or obfuscated to evade signature detection.

Although high entropy alone does not confirm malicious intent, it serves as a valuable early indicator when combined with static analysis, behavioral detection, and threat intelligence.

## 4. Behavioral detection through process telemetry

Behavioral detection focuses on what processes do after they begin running. It monitors activity such as:

- Process creation and ancestry
- Command-line arguments
- Registry and file changes
- Scheduled-task creation
- Credential-access attempts
- Network connections
- Security-control tampering

Individual events may appear harmless when viewed separately. A document application launching PowerShell, followed by a command shell and a trusted Windows utility, becomes more suspicious when the complete execution chain is examined.

Process ancestry also helps distinguish routine administration from unusual execution. PowerShell launched by an approved management tool may be expected. PowerShell launched by a document reader and followed by an external download deserves greater scrutiny.

**Works best for:**  
Previously unknown malware, fileless and multi-stage attacks, and trusted-tool abuse.

**Main limitation:**  
Legitimate IT tools often perform actions that resemble malicious behavior. Scripts, software installers, remote-management products, and administrative utilities can all create processes, modify files, or execute commands. Behavioral detection must therefore consider process ancestry, user activity, endpoint role, and related events to reduce false positives.

**Real-world example**

**[Akira ransomware](https://www.cisa.gov/news-events/cybersecurity-advisories/aa24-109a):** CISA reported that Akira operators abused legitimate remote-access products. The advisory also describes the use of command-line techniques for network and domain discovery. These tools are not inherently malicious, but their execution context and surrounding activity can reveal unauthorized use.

## 5. Memory and runtime analysis

Some malware executes primarily or entirely in memory, leaving little conventional malicious content on disk. Memory and runtime analysis examines active processes and memory regions for suspicious conditions such as:

- Shellcode execution
- Reflective DLL loading
- Process hollowing
- Code injection
- Unbacked executable memory
- Unusual memory permissions

For example, a memory region with read, write, and execute permissions may deserve inspection because it allows code to be written and executed from the same location. This does not prove malicious activity, but it becomes significant when combined with injection behavior or unusual process ancestry.

MITRE ATT&CK documents reflective DLL injection and other process-injection techniques that allow code to execute inside the address space of another process.

**Works best for:**  
Fileless malware, in-memory payloads, and process injection and hollowing.

**Main limitation:**  
Legitimate applications, browsers, development tools, and just-in-time compilers may also allocate executable memory. Memory detections must therefore be supported by process context, execution flow, API activity, and other endpoint telemetry.

## 6. Script and living-off-the-land detection

Many attacks use legitimate interpreters and native utilities instead of deploying a conventional executable. Commonly abused tools include:

- PowerShell
- Windows Management Instrumentation
- MSHTA
- Rundll32
- Regsvr32
- CertUtil
- Bitsadmin

These tools are widely used for legitimate administration, so blocking them outright may disrupt normal operations. Detection instead focuses on how a tool is invoked and what happens next. Suspicious indicators may include encoded or obfuscated commands, downloads from unusual destinations, unexpected parent processes, attempts to disable security controls, and activity outside approved workflows.

CISA describes [living-off-the-land](https://www.manageengine.com/malware-protection/articles/living-off-the-land-attack.html) activity as the abuse of legitimate native tools and processes to conduct malicious operations while blending into normal environments.

**Works best for:**  
Fileless execution, malicious script activity, and native utility abuse.

**Main limitation:**  
The same utilities are regularly used by administrators, deployment tools, and automation systems. Detecting malicious use requires command-line analysis, process relationships, user context, and environmental baselines.

**Real-world example · 2025**

**[Medusa ransomware](https://www.cisa.gov/news-events/cybersecurity-advisories/aa25-071a):** CISA reported that Medusa ransomware operators used PowerShell and a signed vulnerable driver to disable security tools before launching encryption. The activity appeared legitimate until command-line behavior and execution context revealed the attack.

## 7. Network and command-and-control detection

Malware often communicates with external infrastructure to download additional payloads, receive instructions, transmit information, or maintain persistence. Network-aware detection analyzes connections in relation to the process that created them. Relevant signals may include:

- Connections to known malicious destinations
- Unusual outbound traffic
- Repeated beacon-like communication
- Suspicious DNS activity
- Secondary payload downloads

MITRE ATT&CK documents how attackers may use common application layer protocols to blend command-and-control traffic into ordinary web, DNS, email, or file-transfer activity.

**Works best for:**  
Command-and-control communication, secondary payload downloads, and suspicious outbound connections.

**Main limitation:**  
Network activity may be encrypted or routed through legitimate cloud services and compromised infrastructure. A destination may also be unknown at the time of an attack. Network indicators become more useful when correlated with the originating process, command line, user, file, and preceding endpoint activity.

## 8. Ransomware behavior detection

Ransomware detection looks for actions associated with preparing for or performing large-scale encryption rather than relying only on the identity of an executable. Relevant behavioral signals may include:

- Rapid modification of many files
- Mass file renaming
- Unusual encryption activity
- Attempts to delete shadow copies
- Termination of backup or database services
- Security-tool tampering

MITRE ATT&CK describes ransomware operations that encrypt files on local and remote drives to disrupt availability.

**Works best for:**  
Previously unknown ransomware, rapid encryption attempts, and destructive file operations.

**Main limitation:**  
Legitimate applications may also modify many files. Backup software, synchronization tools, compression utilities, and database operations can produce activity that resembles ransomware. Detection should therefore evaluate modification rate, process reputation, file types, process ancestry, encryption characteristics, and security-control tampering.

**Real-world example**

**Medusa ransomware:** CISA reported that [Medusa actors](https://www.manageengine.com/malware-protection/adversaries/medusa-ransomware.html) attempted to use vulnerable or signed drivers to terminate or remove endpoint security tools. Monitoring driver loading, related command-line activity, process ancestry, and subsequent security-tool termination can help reveal the attack before encryption begins.

## 9. Threat intelligence and reputation-based enrichment

Threat intelligence enriches detections with external context by correlating [Indicators of Compromise (IoCs)](https://www.manageengine.com/malware-protection/malware-scanning-and-forensic.html), malicious IP addresses, domains, file hashes, certificates, and vulnerability intelligence from trusted feeds. Rather than identifying malware directly, it helps analysts quickly determine whether suspicious activity is associated with known adversaries or active attack campaigns.

Threat intelligence is most effective when combined with behavioral analysis and endpoint telemetry to prioritize investigations and accelerate incident response.

Endpoint environments generate more events than analysts can reasonably evaluate one by one. Machine learning can assist by identifying unusual patterns, comparing activity against learned baselines, scoring combinations of signals, and prioritizing events for investigation.

Instead of treating each activity independently, a model may consider relationships between process execution, memory behavior, command-line activity, file modifications, network communication, and user and endpoint history. This can help surface patterns that do not match a single fixed signature or rule.

**Works best for:**  
Large volumes of endpoint telemetry, previously unseen behavioral patterns, and multi-signal anomaly detection.

**Main limitation:**  
Machine learning does not automatically understand malicious intent. Its effectiveness depends on telemetry quality, model design, representative baselines, and continuous validation. Poorly tuned models may generate false positives, while incomplete telemetry may cause malicious behavior to be missed. Machine learning should support detection logic and analyst judgment rather than replace them.

## 10. Automated response and deception-based detection

Modern endpoint security does more than detect malicious activity—it actively lures attackers into revealing themselves. Deception-based detection uses decoy files, fake credentials, and other high-value assets that legitimate users should never access. Any interaction with these assets immediately signals suspicious behavior, enabling security teams to trigger automated containment before ransomware or malware can spread.

### Honeypot and deception-based detection

Unlike traditional detection methods that wait for malicious behavior to occur, deception-based detection creates controlled traps designed to attract attackers. These may include decoy files, fake credentials, mapped network shares, or honeytokens that appear valuable but have no legitimate purpose. Because authorized users never interact with these assets, any access attempt becomes a high-confidence indicator of malicious activity.

Deception techniques are particularly effective against ransomware, credential theft, and lateral movement, allowing security teams to detect attacks during reconnaissance or the earliest stages of execution.

### Attack-chain correlation

A single alert rarely explains an entire compromise. An attack may begin with a malicious document, continue through a script interpreter, establish persistence, access credentials, communicate with external infrastructure, and eventually encrypt files. Each stage may generate a separate signal.

Attack-chain correlation connects related activity into a timeline by examining process ancestry, shared files and indicators, user and endpoint context, command-line sequences, network destinations, and time relationships. Threat intelligence can enrich this timeline with known indicators of compromise, malicious infrastructure, active campaigns, and exploited vulnerabilities.

CISA recommends using its Known Exploited Vulnerabilities catalog to prioritize vulnerability remediation. However, the catalog tracks exploited vulnerabilities, not malware signatures. It can help identify relevant exposure and attack paths, but it is not a substitute for endpoint detection.

**Works best for:**  
Multi-stage attacks, root cause analysis, and reconstructing the initial compromise.

**Main limitation:**  
Correlation is only as complete as the underlying telemetry. Missing process, network, identity, or endpoint data can create gaps in the attack timeline. Threat intelligence can also become outdated as infrastructure, malware variants, and attacker behavior change. It should enrich behavioral evidence rather than replace it.

## Comparing modern malware detection techniques

The coverage ratings below indicate how useful each technique can be for identifying a category of activity. They do not guarantee that every threat in that category will be detected.

| Detection technique | Primary telemetry | Unknown activity | Fileless threats | LOLBin abuse | Ransomware | Main limitation |
|---|---|---|---|---|---|---|
| Signature and hash detection | File hashes and known patterns | Limited | Limited | Limited | Limited | Requires a known indicator |
| File reputation analysis | Reputation and file context | Moderate | Limited | Limited | Limited | New does not always mean malicious |
| Static code analysis | File structure and code properties | Moderate | Limited | Limited | Moderate | Obfuscation can hide functionality |
| Behavioral process detection | Processes and command lines | Strong | Strong | Strong | Strong | Legitimate activity may look suspicious |
| Memory and runtime analysis | Process memory and runtime events | Strong | Strong | Moderate | Strong | Legitimate applications may behave similarly |
| Script and LOLBin detection | Scripts and native utilities | Strong | Strong | Strong | Moderate | Requires strong environmental context |
| Network and C2 detection | Process-linked network activity | Moderate | Moderate | Moderate | Moderate | Encryption can conceal traffic |
| Ransomware behavior detection | File activity and impact behavior | Strong | Moderate | Moderate | Strong | Bulk file operations may look similar |
| ML-assisted correlation | Multi-source behavioral telemetry | Strong | Strong | Strong | Strong | Depends on data quality and tuning |
| Attack-chain correlation | Related endpoint and threat signals | Strong | Strong | Strong | Strong | Missing telemetry creates gaps |

## What happens after malware is detected?

Detection identifies suspicious or malicious activity. Response limits what that activity can do next. Once sufficient evidence confirms a threat, an endpoint security platform may:

- Terminate malicious processes
- Quarantine affected files
- Isolate compromised endpoints
- Block malicious communication
- Recover affected files where supported
- Investigate the root cause and attack chain

Response is not itself a malware detection technique. It is the operational action taken after detection reaches an appropriate level of confidence.

## Why layered malware detection matters

Every technique covers a different blind spot. Signatures identify known threats quickly, but may miss modified files. Static analysis can find suspicious capabilities before execution, but obfuscation can hide them. Behavioral monitoring exposes malicious actions, while memory analysis identifies threats that do not exist as conventional files. Network monitoring can reveal external communication that only becomes meaningful when tied to a suspicious process.

Modern campaigns reinforce this need. Volt Typhoon relied heavily on legitimate tools and credentials. Akira operators abused remote-access products and command-line utilities. Medusa actors attempted to disable endpoint security before launching encryption.

The strongest detection strategy uses each technique where it provides the most reliable signal and correlates the results across the attack lifecycle.

## How Malware Protection Plus combines multiple detection techniques

Modern malware cannot be detected by a single engine. [ManageEngine Malware Protection Plus](https://www.manageengine.com/malware-protection/) combines multiple detection techniques into a unified security workflow, including signature matching, heuristic analysis, behavioral AI, memory inspection, script monitoring, attack chain correlation, threat intelligence, and automated response. By correlating these detection layers in real time, it helps security teams identify, investigate, contain, and remediate advanced threats before they lead to ransomware, data exfiltration, or business disruption.

## The takeaway

Modern malware detection is not about choosing between signatures, behavior, memory, scripts, network activity, or machine learning. It is about combining these techniques so that the limitations of one layer are covered by the strengths of another. Together, they give security teams the context needed to detect, investigate, and contain modern endpoint threats.

![Author Karthik Pandian](https://www.manageengine.com/ems/images/tools/employee/karthik-p-maketer.png)

**Karthik Pandian**  
Product Marketer, ManageEngine  

Product Marketing Expert with ManageEngine's Unified Endpoint Management and Security portfolio. He specializes in translating complex UEM concepts into clear, actionable insights that help IT teams manage and secure their endpoints with confidence.

## Frequently asked questions

### How do you detect malware before it causes damage?

Modern malware is detected by continuously monitoring process behavior, memory activity, scripts, and network connections. Instead of relying only on known signatures, advanced detection identifies suspicious behavior that indicates an attack in progress.

### Why do traditional antivirus solutions miss modern malware?

Traditional antivirus relies primarily on known malware signatures. Modern threats use fileless execution, code obfuscation, and legitimate system tools to bypass signature-based detection, making behavioral analysis essential.

### What should malware detection software look for?

Effective malware detection software should monitor process execution, memory activity, command-line behavior, registry changes, script execution, and network communication to identify malicious activity across the entire attack lifecycle.

### Can malware detection software identify unknown or zero-day threats?

Yes. Advanced malware detection software uses behavioral analysis, AI, heuristic detection, and memory scanning to identify suspicious activity, allowing it to detect previously unseen malware without waiting for signature updates.

### How does AI improve malware detection?

AI analyzes execution patterns, process behavior, and anomalies to identify malicious activity that traditional signature-based tools may miss. This helps detect emerging and zero-day threats in real time.

### Why is attack chain analysis important for malware detection?

Attack chain analysis connects related events from initial access to execution, persistence, and lateral movement. It helps security teams understand how an attack unfolded, identify the root cause, and respond more effectively.