Behavioral Detection
In-Memory Malware Detection: how to detect malware that never touches disk
Modern attacks increasingly execute within process memory instead of relying on a malicious executable stored on disk, making runtime behavior the primary surface for detection.
Modern attacks increasingly execute within process memory instead of relying on a malicious executable stored on disk. Not every attack is entirely fileless. Many still use a loader, script, or persistence mechanism that briefly interacts with the file system before shifting the core payload into memory. In-memory malware detection focuses on identifying this runtime activity by monitoring process behavior, memory execution, and system interactions, rather than relying solely on file-based scanning.
In this article, you'll learn how in-memory malware, fileless malware, and memory-resident malware differ, the techniques attackers use to evade detection, and how security teams correlate seemingly isolated behavioral signals into high-confidence detections that expose attacks before they escalate.
What is in-memory malware detection?
In-memory malware vs. fileless malware
Although often used interchangeably, in-memory malware, fileless malware, and memory-resident malware describe different aspects of an attack.
| Term | What it means | Can touch disk? |
|---|---|---|
| In-memory malware | Malicious code that executes primarily within process memory | Yes |
| Fileless malware | Attacks that minimize reliance on traditional executable files by using scripts or trusted system tools | Sometimes |
| Memory-resident malware | Malware that remains active primarily or partly in memory during execution | Yes |
| Memory forensics | The post-incident analysis of memory to recover evidence and reconstruct attacker activity | Not applicable |
An in-memory attack is not necessarily fileless. Many attacks still use a script, loader, or persistence mechanism that briefly interacts with the file system before moving the payload into memory. What defines an in-memory attack is where the malicious code executes, not whether every stage of the attack avoids the disk.
Why do attackers execute malware in memory?
Running malware in memory helps attackers reduce their visibility while blending into legitimate system activity. By avoiding a conventional executable, they make detection, investigation, and forensic analysis significantly more challenging. Common advantages include:
- Fewer file artifacts
Fewer file artifacts are left behind for traditional antivirus to scan.
- Execution within trusted processes
Malicious activity runs inside trusted processes, making it appear legitimate.
- Runtime-only payload decryption
Payloads are decrypted only at runtime, leaving little evidence on disk.
- Abuse of trusted tools
Trusted scripting engines and Windows utilities are abused to evade signature-based detection.
- Reduced static analysis effectiveness
Shifting execution into memory reduces the effectiveness of static analysis.
- Limited forensic evidence
Malicious code may disappear once the system shuts down, limiting forensic evidence.
How does an in-memory attack unfold?
Although every intrusion is different, most in-memory attacks follow a similar execution pattern. An attacker first gains access through a phishing email, malicious download, or exploited vulnerability. A loader or script then launches the payload, which is decrypted or unpacked in memory before being injected into a running process. Once active, the malware performs actions such as credential theft, system discovery, lateral movement, or command-and-control communication, all while leaving little evidence on the file system.
What are the common in-memory execution techniques?
Attackers use a range of techniques to execute malicious code directly in memory while blending into legitimate system activity.
- Process injection (MITRE ATT&CK T1055)
Malicious code is written into the memory of a trusted process using APIs such as VirtualAllocEx, WriteProcessMemory, and CreateRemoteThread, allowing the payload to execute under the identity of a legitimate application.
- Process hollowing
A legitimate process is launched in a suspended state, its original memory is replaced with a malicious payload, and execution is resumed. While the executable on disk remains unchanged, the running process is entirely attacker-controlled.
- Reflective DLL loading
Instead of using the standard Windows loader, a DLL loads itself directly into memory, bypassing normal module registration and reducing its visibility to traditional security tools.
- Shellcode execution and runtime unpacking
Attackers stage encrypted shellcode in executable memory and decrypt it only at runtime, making the payload difficult to detect through static analysis or file-based scanning.
- Script-based execution
PowerShell, JavaScript, and VBScript are commonly used to execute malicious logic. While these techniques are often associated with fileless attacks, scripts may still be written to disk before the payload moves into memory.
- Living-off-the-Land (LOLBin) activity
Trusted Windows utilities such as rundll32.exe, mshta.exe, and certutil.exe are abused to execute or deliver malicious code, allowing attackers to blend into normal administrative activity.
How does in-memory malware detection work?
Detecting in-memory malware requires more than a single indicator. Modern security platforms correlate multiple behavioral signals to distinguish malicious activity from legitimate system operations.
- Memory inspection
Identifies executable private memory, unexpected permission changes, and memory regions that are not backed by a legitimate file.
- Process behavior monitoring
Tracks parent-child relationships, cross-process memory access, and applications behaving outside their normal operational profile.
- Script and command-line analysis
Examines encoded PowerShell commands, obfuscated scripts, and suspicious interpreter chains using technologies such as AMSI.
- API monitoring
Observes runtime calls including VirtualAllocEx, WriteProcessMemory, CreateRemoteThread, and NtProtectVirtualMemory. Individually these APIs are legitimate, but when they occur in a suspicious sequence, they strongly indicate process injection or shellcode execution. Modern EDR platforms enrich memory detection with telemetry from ETW, AMSI, kernel callbacks, and process creation events, allowing multiple behavioral signals to be correlated during execution.
- Behavioral correlation
Brings individual signals together. A single API call or memory allocation rarely confirms an attack, but when combined with abnormal process behavior, script execution, and network activity, they provide the high-confidence detection needed to identify and stop in-memory malware before it can progress.
Runtime detection continuously monitors how processes, scripts, and applications behave while they execute, identifying suspicious API calls, process injection attempts, abnormal parent-child relationships, and other behavioral indicators in real time. Memory scanning complements this by inspecting the contents of process memory for injected code, shellcode, executable memory regions, and other malicious artifacts that may never appear on disk. Together, they provide the visibility needed to detect in-memory attacks throughout their execution lifecycle.
From individual signals to high-confidence detection
No single memory event confirms an attack. Legitimate applications, debuggers, JIT compilers, and security tools routinely allocate executable memory, create threads, and invoke low-level Windows APIs. What separates malicious activity from normal system behavior is the correlation of multiple runtime signals.
Consider a common attack sequence: a document launches PowerShell, which allocates memory in another process, writes code into it, starts a remote thread, and then initiates an outbound network connection. Individually, each action may appear legitimate. Together, they form a behavioral pattern consistent with process injection.
Modern detection engines correlate multiple indicators before generating a high-confidence alert, including:
- Process lineage and command-line activity
Such as encoded PowerShell commands or unusual parent-child relationships.
- Cross-process memory operations
Including memory allocation, remote writes, and thread creation.
- Memory execution context
Such as executable private memory or code running from unbacked memory regions.
- Network and post-exploitation behavior
Including suspicious outbound connections, credential access attempts, and follow-on attacker activity.
Behavioral correlation, rather than any single indicator, is what enables reliable in-memory malware detection while reducing false positives.
Common indicators of in-memory malware
| Signal | May indicate | Context caveat |
|---|---|---|
| Execution from private memory | Shellcode or unpacked payload | JIT compilers behave similarly |
| Cross-process memory writes | Process injection | Debuggers perform comparable operations |
| Unexpected executable (RWX) memory | Payload staging | Browsers and development tools may allocate executable memory |
| Unusual process lineage | Script-based or LOLBin execution | Administrative tools can produce uncommon process chains |
| Executable memory without file mapping | Reflective DLL loading | Must be correlated with runtime behavior |
| Runtime memory permission changes | Payload preparation | Some legitimate applications modify memory permissions |
Why is detecting in-memory malware challenging?
In-memory malware is difficult to detect because many of the techniques it relies on are also used by legitimate software. JIT compilers, debuggers, browsers, and security products routinely allocate executable memory or create remote threads during normal operation. Packed applications, encrypted payloads, and the transient nature of volatile memory further complicate analysis, while continuous memory monitoring must balance visibility with endpoint performance. Effective detection therefore depends on behavioral context and signal correlation rather than isolated events.
Unlike runtime detection, which identifies suspicious activity as it happens, memory forensics focuses on analyzing captured memory during or after an incident. Investigators examine memory images to recover injected code, decrypted payloads, process artifacts, credentials, and other evidence that may never exist on disk. While it is primarily an incident response capability, memory forensics also helps validate detections and reconstruct the full attack timeline.
Real-world examples
How to investigate an in-memory malware alert
Once suspicious memory activity is detected, the goal is to determine how the payload reached memory, what it executed, and whether the attack extended beyond the affected endpoint. Review the complete process tree, command-line arguments, scripts, and parent-child relationships before identifying the origin of the executable memory. Correlate cross-process activity with file, registry, persistence, and network artifacts to reconstruct the attack chain. If malicious activity is confirmed, isolate the endpoint, preserve forensic evidence, remove the threat, and remediate the original entry point to prevent reinfection.
Improving in-memory malware detection
Reliable detection depends on visibility across the entire execution chain rather than a single detection method. Combine file inspection with runtime memory monitoring, behavioral analysis, and process telemetry to detect attacks at different stages of execution. Baseline normal process relationships, monitor memory permission changes and cross-process operations, and use threat intelligence to enrich detections rather than relying on it alone. Regular testing against real-world attack techniques also helps validate detection coverage and reduce blind spots.
How Malware Protection Plus detects in-memory threats
Malware Protection Plus continuously monitors runtime process behavior, script execution, memory operations, and process relationships to detect attacks that execute beyond the file system. Rather than relying on a single indicator, it correlates memory activity, API sequences, process lineage, and behavioral signals to distinguish genuine in-memory attacks from legitimate system activity.
Security teams can investigate incidents through process trees, root cause analysis, and MITRE ATT&CK mapping before responding with process termination, endpoint isolation, quarantine, and recovery where applicable.
