Instiq
Chapter 3 · Host-Based Analysis·v1.0.0·Updated 7/20/2026·~16 min

What's changed: Initial version

3.1Endpoint technologies

Key points

Covers the roles of host-resident HIDS (host-based IDS), antimalware/antivirus (AV), and the host firewall, and the difference between signature/rule-based and behavioral/predictive (predictive AI) based detection foundations, framed as the SOC judgment of "which technology caught this alert" and "which method works against known vs. unknown threats."

Sensors at the network boundary alone cannot see inside encrypted traffic or malicious actions carried in with valid credentials. That is exactly why you need visibility on the endpoint (host) itself. This section organizes the division of labor among three host-resident defenses—HIDS (monitoring host behavior, files, and logs), AV/antimalware (detecting/quarantining malicious code), and the host firewall (controlling traffic to/from that host)—and on what basis they decide something is "bad" (signature vs. behavior), from the perspective of an analyst reading alerts.

3.1.1Roles of host-resident defenses

  • A HIDS (host-based intrusion detection system) monitors that host's system calls, process creation, file/registry changes, log tampering, and the like to detect suspicious behavior. Whereas a network IDS (NIDS) inspects packets on the wire, a HIDS watches the host's internal state, so it more readily catches execution occurring beyond encryption or post-compromise activity from within. Modern EDR integrates response/containment capabilities on top of HIDS.
  • Antimalware/antivirus (AV) matches code in files or memory against known malware and, on a hit, quarantines/removes it. Traditional (legacy) AV centers on signature matching—strong against known samples but weak against unknown/variant ones. Next-generation AV (NGAV) additionally uses behavior/machine learning.
  • A host firewall is the host's own packet filter, permitting/denying traffic to/from the host by port, protocol, or application (e.g., Windows Defender Firewall, Linux iptables/nftables). Whereas a network firewall guards the boundary, the host firewall protects each individual endpoint as a last line of defense and constrains internal lateral movement.

3.1.2Detection foundations: signature vs. behavioral/predictive

  • Signature/rule-based detection fires when input matches predefined known characteristics—a hash, byte sequence, or known-malicious pattern. It yields few false positives with clear rationale, but misses unknown/zero-day threats or slightly altered variants (because no signature exists yet).
  • Behavioral/predictive (predictive AI) based detection flags deviation from a normal baseline or a chain of actions typical of malice (e.g., a document app launching PowerShell and connecting outbound). It can react to unknown threats, but tends to produce more false positives when legitimate work looks similar. Predictive AI/machine learning is an advanced form of this behavioral approach.
Exam point

Most-tested: HIDS monitors host-internal behavior (strong beyond encryption and on post-compromise activity); AV matches known malware, with legacy AV centered on signatures; the host firewall controls traffic per endpoint to constrain lateral movement; signatures are strong on known/weak on unknown, while behavioral/predictive AI reacts to unknown but raises false positives. Practice stating, per technology, "on what basis did this alert fire."

In the SOC you are triaging two notifications that fired simultaneously from one business PC. One is an AV alert—"detected and quarantined the known trojan Trojan.GenericKD"; the other is a HIDS/EDR behavioral alert—"winword.exe spawned a child process powershell.exe -enc <Base64> and attempted a connection to an external IP." Jumping to "the AV quarantined a known item, so it is handled, and the HIDS alert is probably a false positive" is a dangerous call. The AV signature hit only stopped that one sample, whereas winword.exe (a document app) launching obfuscated PowerShell to reach out is a sign of an unknown/variant payload or fileless execution not yet in any signature—precisely what behavioral/predictive detection exists to catch. The sound judgment treats the HIDS alert as leaning true-positive: isolate the endpoint and preserve the process tree, destinations, and dropped files. Conversely, a setup with only AV and no behavioral monitoring misses exactly this "document macro -> obfuscated script -> external C2" chain because no signature exists—that is how "legacy AV is strong on known, weak on unknown" shows up in practice. From the host-firewall angle, blocking this endpoint's outbound traffic to unfamiliar destination ports means that even if execution slips through, C2 establishment and lateral movement are constrained at the last line of defense. Telling which technology raised an alert and on what basis is the key to avoiding both underreaction (missing the unknown) and overreaction (over-reacting to a false positive).

TechnologyWhat it watchesDetection basisStrength/limit
HIDS/EDRProcesses, files/registry, system callsBehavior (rules too)Strong on post-compromise/beyond encryption; some FPs
AV/antimalwareCode in files/memorySignature (NGAV adds behavior)Strong on known; weak on unknown/variant
Host firewallTraffic to/from the hostRules by port/protocol/appConstrains lateral movement per host; no payload inspection
Warning

Trap: "Legacy AV running with the latest definitions reliably blocks even unknown malware" is wrong—signature matching only fires when input matches a known characteristic, so unknown/zero-day threats and variants with no signature slip through, and must be complemented by behavioral/predictive monitoring. Also wrong: "a HIDS inspects every packet on the network"—that is the role of a NIDS; a HIDS watches the internal state of that host.

HIDS, antimalware, host firewall, and signature vs. behavioral detection.
Which technology caught this alert

3.1.3Section summary

  • HIDS/EDR watches host-internal behavior, AV watches malicious code, and the host firewall watches traffic to/from the endpoint—supplying visibility that boundary sensors lack
  • Signatures/rules are strong on known and weak on unknown; behavioral/predictive AI reacts to the unknown but raises false positives—the two are complementary
  • Read alerts by "which technology fired and on what basis"—do not rest on an AV known-quarantine alone, and do not miss the post-compromise activity in a behavioral alert

Sign in to track progress — Log in.

Quick check

(just a quick review)

Q1. On a business PC, right after AV quarantines a known trojan, HIDS/EDR raises a behavioral alert: "winword.exe spawned powershell.exe -enc <Base64> and attempted a connection to an external IP." What is the most appropriate first action for the SOC analyst?

Q2. A novel malware (zero-day) that matches no known signature is brought onto a host with valid credentials and executed. Which technology-and-basis combination is most likely to detect this execution on the host?

Q3. Lateral movement is suspected: a compromised host is repeatedly attempting connections to the management ports of other hosts on the same segment. Which technology is best suited to constrain this internal traffic per host as a last line of defense?

Check your understandingPractice questions for Chapter 3: Host-Based Analysis