10 malware detection techniques and where each one works best

Why malware detection shifted from files to behavior

Traditional malware detection focused largely on identifying known malicious files through signatures, hashes, and file attributes. These methods remain effective because they can detect previously analyzed threats quickly and with relatively low processing overhead.

However, attackers increasingly use obfuscation, polymorphic code, scripts, trusted system utilities, process injection, and memory-based execution to reduce the number of obvious malicious files available for inspection.

MITRE ATT&CK documents techniques such as encrypted payloads, polymorphic code, reflective DLL injection, and process hollowing that can make conventional file analysis less effective.

Modern detection therefore looks beyond what a file is and examines what processes, scripts, users, and applications do. Process ancestry, command lines, memory activity, file changes, and network connections can expose malicious activity even when no known signature is available.

1. Signature and hash-based detection

Signature-based detection compares files against known malicious indicators, including cryptographic hashes, byte sequences, malware family signatures, and pattern-based rules.

When a file matches a known indicator, it can be identified before or during execution. This makes signature detection fast and effective against established malware families and previously observed payloads.

Known malware families, previously identified malicious files, and high confidence early detection.

Signatures can only identify threats that have already been analyzed and added to the detection database. Repacking, encrypting, recompiling, or slightly modifying a file can produce a different hash without significantly changing its behavior. MITRE ATT&CK identifies polymorphic code, encryption, compression, and junk-code insertion as methods attackers may use to make malicious files more difficult to detect or analyze. Signature matching should therefore remain one detection layer, but not the only one.

2. File reputation analysis

File reputation analysis evaluates whether a file is known, trusted, uncommon, suspicious, or malicious based on local and cloud intelligence.

Unlike a direct signature match, reputation analysis considers broader context. A newly observed unsigned executable downloaded from an unusual source may deserve closer inspection even when it does not match a known malware hash. Reputation data can help security tools make faster decisions about unfamiliar files and prioritize them for additional analysis.

Newly encountered files, rare or low prevalence executables, and suspicious downloads and attachments.

A file being new or uncommon does not make it malicious. Legitimate internal applications and newly released software may initially have little reputation data. Attackers may also abuse trusted certificates, compromise legitimate software, or use widely available tools with established reputations. Reputation must therefore be evaluated alongside file origin, signer information, execution context, and runtime behavior.

3. Static code analysis

Static analysis examines a file before it runs. It can inspect executable headers, imported functions, embedded strings, digital signatures, section characteristics, and compiler artifacts.

Static analysis may also identify packed or encrypted sections, suspicious imports, embedded commands or network indicators, and anti-analysis functionality.

Detecting suspicious files before execution, identifying known code patterns, and finding embedded configuration data.

Static indicators are not conclusive on their own. Legitimate applications may use APIs associated with debugging, instrumentation, or software management. Malware can also hide its functionality through packing, encryption, dynamic API resolution, or code assembled only during execution. Static analysis is therefore most useful when paired with reputation and runtime monitoring.

Additional malware detection techniques used in modern security

No single detection engine can identify every threat. Modern endpoint security combines multiple complementary techniques that strengthen detection before, during, and after malware execution. These methods improve detection accuracy, reduce false positives, and help uncover threats that evade traditional scanning.

• Heuristic analysis

Heuristic detection identifies suspicious files by examining their structure, instructions, API usage, and execution logic instead of relying solely on known signatures. It helps detect previously unseen malware variants, polymorphic malware, and zero-day threats by identifying characteristics commonly associated with malicious software.

Works best for:

Unknown malware, Zero-day threats, Packed or obfuscated executables

Main limitation:

May generate false positives if legitimate software exhibits unusual behavior.

• Dynamic analysis and sandboxing

Static inspection cannot always reveal malicious intent. Dynamic analysis executes suspicious files inside an isolated sandbox to observe their behavior safely. During execution, the sandbox monitors process creation, registry changes, file modifications, memory activity, and network communication to determine whether the sample behaves maliciously.

Modern malware increasingly incorporates sandbox-evasion techniques, making dynamic analysis most effective when combined with behavioral monitoring and runtime detection.

Real-world example (2026): Picus Labs documented LummaC2 using mouse-movement analysis to detect automated sandboxes and avoid executing its malicious payload.

• Allowlisting and blocklisting

Allowlisting permits only trusted applications to execute, while blocklisting prevents known malicious files, hashes, or applications from running. Together, they reduce attack surface by restricting unauthorized software execution.

Although blocklisting remains effective against known malware, allowlisting offers stronger protection against unknown threats by preventing unapproved applications from executing in the first place. However, both approaches require continuous maintenance to remain effective.

• Entropy-based detection

Packed and encrypted malware often exhibits unusually high entropy because its contents appear random. Entropy analysis helps identify executables that may have been compressed, encrypted, or obfuscated to evade signature detection.

Although high entropy alone does not confirm malicious intent, it serves as a valuable early indicator when combined with static analysis, behavioral detection, and threat intelligence.

4. Behavioral detection through process telemetry

Behavioral detection focuses on what processes do after they begin running. It monitors activity such as:

• Process creation and ancestry
• Command-line arguments
• Registry and file changes
• Scheduled-task creation
• Credential-access attempts
• Network connections
• Security-control tampering

Individual events may appear harmless when viewed separately. A document application launching PowerShell, followed by a command shell and a trusted Windows utility, becomes more suspicious when the complete execution chain is examined.

Process ancestry also helps distinguish routine administration from unusual execution. PowerShell launched by an approved management tool may be expected. PowerShell launched by a document reader and followed by an external download deserves greater scrutiny.

Previously unknown malware, fileless and multi-stage attacks, and trusted-tool abuse.

Legitimate IT tools often perform actions that resemble malicious behavior. Scripts, software installers, remote-management products, and administrative utilities can all create processes, modify files, or execute commands. Behavioral detection must therefore consider process ancestry, user activity, endpoint role, and related events to reduce false positives.

5. Memory and runtime analysis

Some malware executes primarily or entirely in memory, leaving little conventional malicious content on disk. Memory and runtime analysis examines active processes and memory regions for suspicious conditions such as:

• Shellcode execution
• Reflective DLL loading
• Process hollowing
• Code injection
• Unbacked executable memory
• Unusual memory permissions

For example, a memory region with read, write, and execute permissions may deserve inspection because it allows code to be written and executed from the same location. This does not prove malicious activity, but it becomes significant when combined with injection behavior or unusual process ancestry.

MITRE ATT&CK documents reflective DLL injection and other process-injection techniques that allow code to execute inside the address space of another process.

Fileless malware, in-memory payloads, and process injection and hollowing.

Legitimate applications, browsers, development tools, and just-in-time compilers may also allocate executable memory. Memory detections must therefore be supported by process context, execution flow, API activity, and other endpoint telemetry.

6. Script and living-off-the-land detection

Many attacks use legitimate interpreters and native utilities instead of deploying a conventional executable. Commonly abused tools include:

• PowerShell
• Windows Management Instrumentation
• MSHTA
• Rundll32
• Regsvr32
• CertUtil
• Bitsadmin

These tools are widely used for legitimate administration, so blocking them outright may disrupt normal operations. Detection instead focuses on how a tool is invoked and what happens next. Suspicious indicators may include encoded or obfuscated commands, downloads from unusual destinations, unexpected parent processes, attempts to disable security controls, and activity outside approved workflows.

CISA describes living-off-the-land activity as the abuse of legitimate native tools and processes to conduct malicious operations while blending into normal environments.

Fileless execution, malicious script activity, and native utility abuse.

The same utilities are regularly used by administrators, deployment tools, and automation systems. Detecting malicious use requires command-line analysis, process relationships, user context, and environmental baselines.

7. Network and command-and-control detection

Malware often communicates with external infrastructure to download additional payloads, receive instructions, transmit information, or maintain persistence. Network aware detection analyzes connections in relation to the process that created them. Relevant signals may include:

• Connections to known malicious destinations
• Unusual outbound traffic
• Repeated beacon-like communication
• Suspicious DNS activity
• Secondary payload downloads

MITRE ATT&CK documents how attackers may use common application layer protocols to blend command-and-control traffic into ordinary web, DNS, email, or file-transfer activity.

Command-and-control communication, secondary payload downloads, and suspicious outbound connections.

Network activity may be encrypted or routed through legitimate cloud services and compromised infrastructure. A destination may also be unknown at the time of an attack. Network indicators become more useful when correlated with the originating process, command line, user, file, and preceding endpoint activity.

8. Ransomware behavior detection

Ransomware detection looks for actions associated with preparing for or performing large scale encryption rather than relying only on the identity of an executable. Relevant behavioral signals may include:

• Rapid modification of many files
• Mass file renaming
• Unusual encryption activity
• Attempts to delete shadow copies
• Termination of backup or database services
• Security-tool tampering

MITRE ATT&CK describes ransomware operations that encrypt files on local and remote drives to disrupt availability.

Previously unknown ransomware, rapid encryption attempts, and destructive file operations.

Legitimate applications may also modify many files. Backup software, synchronization tools, compression utilities, and database operations can produce activity that resembles ransomware. Detection should therefore evaluate modification rate, process reputation, file types, process ancestry, encryption characteristics, and security-control tampering.

9. Threat intelligence and reputation-based enrichment

Threat intelligence enriches detections with external context by correlating Indicators of Compromise (IoCs), malicious IP addresses, domains, file hashes, certificates, and vulnerability intelligence from trusted feeds. Rather than identifying malware directly, it helps analysts quickly determine whether suspicious activity is associated with known adversaries or active attack campaigns.

Threat intelligence is most effective when combined with behavioral analysis and endpoint telemetry to prioritize investigations and accelerate incident response.

Endpoint environments generate more events than analysts can reasonably evaluate one by one. Machine learning can assist by identifying unusual patterns, comparing activity against learned baselines, scoring combinations of signals, and prioritizing events for investigation.

Instead of treating each activity independently, a model may consider relationships between process execution, memory behavior, command-line activity, file modifications, network communication, and user and endpoint history. This can help surface patterns that do not match a single fixed signature or rule.

Large volumes of endpoint telemetry, previously unseen behavioral patterns, and multi-signal anomaly detection.

Machine learning does not automatically understand malicious intent. Its effectiveness depends on telemetry quality, model design, representative baselines, and continuous validation. Poorly tuned models may generate false positives, while incomplete telemetry may cause malicious behavior to be missed. Machine learning should support detection logic and analyst judgment rather than replace them.

10. Automated response and deception-based detection

Modern endpoint security does more than detect malicious activity—it actively lures attackers into revealing themselves. Deception-based detection uses decoy files, fake credentials, and other high-value assets that legitimate users should never access. Any interaction with these assets immediately signals suspicious behaviour, enabling security teams to trigger automated containment before ransomware or malware can spread.

• Honeypot and deception-based detection

Unlike traditional detection methods that wait for malicious behaviour to occur, deception-based detection creates controlled traps designed to attract attackers. These may include decoy files, fake credentials, mapped network shares, or honeytokens that appear valuable but have no legitimate purpose. Because authorised users never interact with these assets, any access attempt becomes a high-confidence indicator of malicious activity.

Deception techniques are particularly effective against ransomware, credential theft, and lateral movement, allowing security teams to detect attacks during reconnaissance or the earliest stages of execution.

• Attack-chain correlation

A single alert rarely explains an entire compromise. An attack may begin with a malicious document, continue through a script interpreter, establish persistence, access credentials, communicate with external infrastructure, and eventually encrypt files. Each stage may generate a separate signal.

Attack-chain correlation connects related activity into a timeline by examining process ancestry, shared files and indicators, user and endpoint context, command-line sequences, network destinations, and time relationships. Threat intelligence can enrich this timeline with known indicators of compromise, malicious infrastructure, active campaigns, and exploited vulnerabilities.

CISA recommends using its Known Exploited Vulnerabilities catalog to prioritize vulnerability remediation. However, the catalog tracks exploited vulnerabilities, not malware signatures. It can help identify relevant exposure and attack paths, but it is not a substitute for endpoint detection.

Works best for:

Multi-stage attacks, root cause analysis, and reconstructing the initial compromise.

Main limitation:

Correlation is only as complete as the underlying telemetry. Missing process, network, identity, or endpoint data can create gaps in the attack timeline. Threat intelligence can also become outdated as infrastructure, malware variants, and attacker behavior change. It should enrich behavioral evidence rather than replace it.

Comparing modern malware detection techniques

The coverage ratings below indicate how useful each technique can be for identifying a category of activity. They do not guarantee that every threat in that category will be detected.

What happens after malware is detected?

Detection identifies suspicious or malicious activity. Response limits what that activity can do next. Once sufficient evidence confirms a threat, an endpoint security platform may:

• Terminate malicious processes
• Quarantine affected files
• Isolate compromised endpoints
• Block malicious communication
• Recover affected files where supported
• Investigate the root cause and attack chain

Response is not itself a malware detection technique. It is the operational action taken after detection reaches an appropriate level of confidence.

Why layered malware detection matters

Every technique covers a different blind spot. Signatures identify known threats quickly, but may miss modified files. Static analysis can find suspicious capabilities before execution, but obfuscation can hide them. Behavioral monitoring exposes malicious actions, while memory analysis identifies threats that do not exist as conventional files. Network monitoring can reveal external communication that only becomes meaningful when tied to a suspicious process.

Modern campaigns reinforce this need. Volt Typhoon relied heavily on legitimate tools and credentials. Akira operators abused remote-access products and command-line utilities. Medusa actors attempted to disable endpoint security before launching encryption.

The strongest detection strategy uses each technique where it provides the most reliable signal and correlates the results across the attack lifecycle.

How Malware Protection Plus combines multiple detection techniques

Modern malware cannot be detected by a single engine.ManageEngine Malware Protection Plus combines multiple detection techniques into a unified security workflow, including signature matching, heuristic analysis, behavioral AI, memory inspection, script monitoring, attack chain correlation, threat intelligence, and automated response. By correlating these detection layers in real time, it helps security teams identify, investigate, contain, and remediate advanced threats before they lead to ransomware, data exfiltration, or business disruption.