4.3Replication failures and recovery

Handling failures in a replication setup adds a dimension beyond single-server failure handling: how do you reorganize the primary/standby relationship? Promotion (failover) when the primary is completely lost, re-synchronizing an old primary back into the new topology, guarding against interruptions in WAL delivery, and the apply-time conflicts unique to logical replication—these concentrate much of the hands-on judgment within Gold's failure-response domain.

4.3.1Promotion and re-sync: pg_ctl promote and pg_rewind

• pg_ctl promote—the command that promotes a streaming-replication standby to become the new primary. It is the core operation of failover on primary failure: after running it, the standby exits read-only mode and begins operating as an ordinary read/write primary.
• pg_rewind—a tool that re-configures the old primary (the original primary, once recovered from its failure) as a standby following the new primary's timeline. It rewinds and re-synchronizes only the divergent portion of the WAL history between the old and new primaries, so it restores the old primary faster than rebuilding it entirely from a base backup. It does, however, require the old primary's WAL, so it must be used soon after the divergence occurs.
• pg_receivewal—a standalone client tool (formerly pg_receivexlog) that continuously receives the WAL stream from the primary and keeps saving it as files. Besides serving as a dedicated WAL archive destination, it is also used as a supplementary way to preserve WAL while normal archiving (archive_command) is not functioning for some reason.

4.3.2Logical replication conflicts and error handling

• Logical replication conflicts—a state where the subscriber fails to apply a change. This can occur, for example, when a subscriber-side table has a UNIQUE constraint that does not exist on the publisher side, and a replicated INSERT violates it. Once a conflict occurs, the apply worker stops at that point, and every subsequent change stalls too, so an ignored conflict brings replication to a complete halt.
• Resolving a conflict requires manual intervention on the subscriber side (deleting or fixing the conflicting data, or skipping the offending transaction) to let replication resume. It is important that this is a failure mode unique to logical replication, with no equivalent in physical copying like streaming replication.
• Streaming replication error handling—when the connection between the standby's walreceiver and the primary's walsender drops, the basic behavior is to automatically attempt reconnection, depending on configuration. If the cause is transient (a network blip, a temporarily overloaded primary), reconnecting resolves it—but if WAL beyond the needed range has already been discarded (the standby fell behind past a retention window such as wal_keep_size), reconnection alone cannot recover, and you must take a fresh base backup.

Let's trace a failover response end to end. Suppose the primary becomes unresponsive due to a hardware failure. Monitoring detects the anomaly, and you select the standby that has received the most recent WAL for promotion (with multiple standbys, the replication lag you had been watching via pg_stat_replication is the deciding factor). Running pg_ctl promote on the chosen standby lets it start accepting writes as the new primary. The complication arises when the original primary turns out not to be completely destroyed and can later be recovered. Simply restarting the old primary alongside the new one creates a split-brain situation where both accept writes and consistency collapses—this must be avoided at all costs. The correct procedure is to re-fold the old primary in as a standby subordinate to the new primary. This is where pg_rewind comes in. The old and new primaries diverge into separate timelines the instant promotion happens, but pg_rewind fetches and overwrites only the data blocks that differ since the divergence, from the new primary, letting you reshape the old primary into "a healthy standby following the new primary" faster than rebuilding it entirely from a base backup. This does assume the old primary still retains the post-divergence WAL, so you need to act soon after detecting the failure. If the post-divergence WAL is already gone, or the divergence is too large for pg_rewind to handle, it is more reliable to simply retake a base backup with pg_basebackup. One more often-overlooked point in replication operations is the role of pg_receivewal. Even if normal WAL archiving via archive_command stops working for a while due to some misconfiguration, running pg_receivewal in parallel preserves that period's WAL as files, preventing a gap that would otherwise bite you the next time PITR is needed.

4.3.3Section summary

• pg_ctl promote = promoting a standby to primary (the core of failover). pg_rewind = differentially re-syncing the old primary as a standby of the new one (faster than re-basing, but assumes the old primary's WAL is available)
• pg_receivewal continuously receives WAL and can also preserve it when archive_command is malfunctioning. A logical replication conflict stops the apply worker and requires manual intervention. Streaming replication reconnects automatically by default, but WAL loss requires retaking a base backup