5.1Secure protocols

For a network security architect, "just encrypt it" is not enough. What matters is the ability to work backward from requirements and the threat model to decide which protocol, at which layer, with which options (mode, key-exchange method), fits a given communication requirement (site-to-site VPN, email, web access, remote administration, wireless LAN). This section organizes the representative secure protocols along three axes: the layer they apply to, the protection they provide (confidentiality/integrity/non-repudiation), and their configuration choices.

5.1.1IPsec (transport/tunnel mode, ESP/AH, IKE)

• IPsec is a protocol suite that provides packet-level confidentiality, integrity, and authentication at the IP layer. Transport mode protects only the IP payload and suits end-to-end communication (host-to-host), while tunnel mode encapsulates the entire packet, including the IP header, inside a new IP header, suiting site-to-site VPNs (gateway-to-gateway).
• ESP (Encapsulating Security Payload) provides confidentiality (encryption), integrity, and source authentication. AH (Authentication Header) provides integrity and source authentication only and does not encrypt (AH is also weak against NAT traversal, since NAT rewriting the IP header tends to break authentication). Whether the payload must stay confidential is the deciding factor between ESP and AH.
• IKE (Internet Key Exchange) securely negotiates and agrees on the keys used by IPsec. Built on Diffie-Hellman key exchange, it mutually authenticates both parties and establishes a Security Association (SA). It is also used for periodic key rotation (rekeying), reducing both operational burden and compromise risk compared to manual key management.

5.1.2SSL/TLS and HTTPS

• SSL/TLS encrypts application-level communication at the transport layer. The handshake verifies the server certificate, agrees on the key-exchange algorithm, and generates the session key (a symmetric key); the bulk of subsequent communication is then encrypted with that symmetric key (a hybrid scheme where public-key cryptography handles key exchange/authentication and symmetric-key cryptography handles bulk encryption).
• TLS 1.3 is the latest version: it reduces the number of handshake round-trips (1-RTT, with 0-RTT resumption possible under certain conditions) compared to older versions, and removes weakened legacy key-exchange/cipher options (such as RSA key exchange or old hashes) while mandating PFS (Perfect Forward Secrecy). With PFS, even if a long-term private key is later compromised, past session keys cannot be decrypted.
• HTTPS wraps HTTP communication in SSL/TLS. It simultaneously prevents web-server impersonation (server authentication via the server certificate) and prevents eavesdropping/tampering with the communication content, making it the standard solution for protecting web access.

5.1.3STARTTLS, SSH, S/MIME, and wireless LAN

• STARTTLS lets a protocol that originally communicates in plaintext (SMTP/IMAP/POP3, etc.) upgrade to TLS on the same port partway through the session. This reuses existing infrastructure without opening a dedicated encrypted port, but carries the risk of a STRIPTLS attack, where an attacker tampers with the plaintext phase before the upgrade to block the transition to encryption.
• SSH (Secure Shell) encrypts and protects remote login, file transfer, and command execution. Using public-key authentication makes it stronger than password authentication, and its port forwarding (tunneling) feature can also carry other TCP traffic over the SSH connection.
• S/MIME encrypts the email body with public-key cryptography and provides sender authentication, tamper detection, and non-repudiation via a digital signature. Both sender and recipient need certificates, making it suited to highly confidential inter-organizational email.
• WPA2 encrypts wireless LAN traffic using AES-based CCMP, but vulnerabilities (e.g., KRACK) have been identified in the 4-way handshake used with pre-shared keys. WPA3 is the successor, adopting a stronger key exchange (SAE: Simultaneous Authentication of Equals) that strengthens resistance to offline dictionary attacks and adds PFS.

Suppose a network architect at a company must satisfy three requirements at once. First, a site-to-site VPN linking headquarters and a branch office, where all traffic between the two sites (including source/destination IPs) must stay confidential and tampering must be detected—here the architect chooses IPsec tunnel mode with ESP. Tunnel mode encapsulates the entire original IP header, hiding the inter-site network layout from outsiders, and ESP's encryption satisfies the confidentiality requirement (AH would not encrypt, so it is unsuitable). Second, the existing internal SMTP mail server must be encrypted without changing its port number or requiring a rebuild—STARTTLS fits. However, since an attacker could exploit the pre-upgrade plaintext phase with a STRIPTLS attack to force the session to stay unencrypted, pairing STARTTLS with a policy that mandates the upgrade on the sending domain's side (such as MTA-STS) is an important practical design decision. Third, developers need to log in to production servers securely and also want to protect traffic to an internal management console—SSH public-key authentication plus port forwarding avoids the risk of leaked passwords while consolidating access to other TCP services through the SSH tunnel. Choosing the right protocol and options for secure-protocol selection in practice comes down to three lenses: what must stay confidential (payload only, or the IP header too), compatibility with existing infrastructure, and the operational burden of key management.

5.1.4Section summary

• ESP encrypts (confidentiality + integrity + authentication); AH does not encrypt (integrity + authentication only). Tunnel mode suits site-to-site VPNs, transport mode suits end-to-end
• TLS 1.3 shortens the handshake and mandates PFS. STARTTLS upgrades to encryption on the existing port, but watch for STRIPTLS attacks
• SSH protects remote operations via public-key authentication and port forwarding; S/MIME protects email via encryption plus signature; WPA3 is stronger than WPA2 via SAE