Malware: technical analysis for SOC & pentest

Malware — short for malicious software — covers every category of code deliberately designed to compromise, disrupt, or exfiltrate data from a target system. The technical analysis of malware spans static examination of PE headers, import tables, and embedded strings, through dynamic execution in controlled sandboxes such as Cuckoo or Any.run. Modern malware families rely on modular architectures: a lightweight dropper fetches a stage-two payload over encrypted C2 channels, often using domain generation algorithms (DGA) to evade blocklists. Ransomware encrypts files using hybrid schemes combining AES session keys wrapped with RSA. Rootkits hook kernel structures like SSDT or DKOM to hide processes. Fileless variants live entirely in memory, injecting shellcode into legitimate processes via techniques such as Process Hollowing or Reflective DLL Injection. Understanding these mechanisms is essential for SOC analysts performing triage, reverse engineers building detection signatures, and Red Team operators validating endpoint defenses against realistic implants.

Detecting malware at the SOC level requires correlating multiple data sources. On Windows endpoints, Event ID 4688 (process creation with command-line logging enabled) and Sysmon Event IDs 1, 3, 7, and 10 are primary sources: they capture process spawning, network connections, image loads, and process access respectively. Suspicious parent-child relationships — such as winword.exe spawning powershell.exe — are high-fidelity indicators. EDR telemetry exposes memory injection patterns: unexpected remote thread creation (Sysmon Event ID 8) or unbacked executable memory regions flagged by tools like Volatility or PE-sieve. Network-level detection focuses on beaconing patterns: regular outbound intervals, low-entropy domain names, JA3/JA3S TLS fingerprints matching known C2 frameworks such as Cobalt Strike or Sliver. YARA rules applied to memory dumps or file scans allow hunting for specific byte sequences, packed headers, or obfuscated string patterns. Threat intelligence platforms correlate file hashes, IP addresses, and domain IOCs against known malware campaigns, enabling rapid triage and contextualization of alerts before escalation.

From a Red Team perspective, malware development focuses on bypassing modern defensive layers. Signature evasion starts with custom payload encoding, string encryption, and import table obfuscation to defeat static antivirus scanning. AMSI (Antimalware Scan Interface) bypass techniques — such as patching the AmsiScanBuffer function in memory — neutralize PowerShell-based detection. ETW (Event Tracing for Windows) tampering blinds EDR telemetry by patching NtTraceEvent. Sandbox evasion uses environmental checks: CPU core count, screen resolution, user interaction delays, and the presence of analysis tools like Wireshark or Process Monitor. For persistence, malware abuses scheduled tasks (T1053.005), registry Run keys (T1547.001), or WMI subscriptions (T1546.003) per the MITRE ATT&CK framework. Hardening countermeasures include enabling Credential Guard, applying Attack Surface Reduction (ASR) rules in Microsoft Defender, enforcing application allowlisting via AppLocker or WDAC, and deploying canary files in sensitive directories to trigger early ransomware detection before mass encryption begins.