4.5Availability Monitoring, Resource Monitoring, and Operations Tools

Checking the current state with individual commands cannot catch failures that happen overnight or on weekends. Building continuous watchers that alert when thresholds are crossed is the foundation that keeps a 24/7 service running.

4.5.1What to watch for, and the OOM killer

• Anomalies to catch early: resource exhaustion (memory, disk running out), overload (sustained high CPU/traffic), and abnormal termination (process/service crashes).
• The OOM killer (Out-Of-Memory Killer) is the kernel mechanism that, when physical memory and swap are nearly exhausted, forcibly selects and terminates a process. It is the first suspect when investigating an unexpected process termination.

4.5.2SNMP, thresholds, and representative tools

• SNMP (Simple Network Management Protocol) is a protocol for remotely retrieving device and server status in a standardized way; monitoring tools use it as the basis for collecting target status.
• Basic monitoring design: set thresholds for standard management items (CPU/memory/storage usage, traffic, availability, logs, response time) and fire an alert when crossed. Availability monitoring (is it alive), log monitoring, response monitoring, and usage monitoring are distinct categories with different purposes.
• Representative tools: Icinga2 and Nagios (classic availability/service monitoring), collectd (a metrics-collection daemon), MRTG and Cacti (graphing traffic and other metrics), and Zabbix (an integrated platform for collection, visualization, and alerting).

Monitoring design is easiest to reason about in three layers: what to measure, how to measure it, and when to alert. First classify targets into availability (is it alive), logs (error pattern occurrence), response (latency), and usage (CPU/memory/storage/traffic), then define standard management items and thresholds for each (e.g., memory usage over 90% sustained for 5 minutes). Crossing a threshold triggers an alert to the administrator. On the implementation side, SNMP is the foundation for retrieving network device and server information remotely in a standardized form; collection daemons like collectd and integrated platforms like Zabbix use it to gather metrics. Nagios and its successor Icinga2, specialized in availability and service monitoring, extend checks via plugins and can send alerts to various channels on threshold breach. MRTG and Cacti excel at visualizing SNMP-collected traffic and similar data as time-series graphs, suited to spotting trends like "when did traffic start climbing". The OOM killer, meanwhile, is often the cause of a symptom rather than something you monitor directly—when a specific process dies for no apparent reason, the standard move is to check dmesg or journalctl logs for OOM killer activity. Setting up memory threshold monitoring beforehand means an alert can catch the problem before the OOM killer ever has to act.

4.5.3Section summary

• Monitoring classifies targets into availability, logs, response, usage; threshold breach triggers an alert; SNMP underlies remote retrieval
• Nagios/Icinga2 = availability/service monitoring / collectd = collection / MRTG, Cacti = graphing / Zabbix = integrated. OOM killer force-terminates on memory exhaustion (unrelated to disk)