2.5Event-driven architecture and messaging

If all integration between microservices is implemented as synchronous API calls, the moment a callee's response is delayed, the caller is also kept waiting, raising the risk of a cascading failure that propagates a fault outward. A representative countermeasure to this problem is asynchronous messaging. But asynchronous integration trades the benefit of "not keeping the caller waiting" for new concerns: messages can arrive out of order, be delivered as duplicates, or have their processing result not reflected immediately (eventual consistency). This section covers how to choose among messaging infrastructure options and the pitfalls unique to asynchronous integration.

2.5.1The roles of MQ, ESB, and API Gateway

• A message queue (MQ) is asynchronous-integration infrastructure in which a sender places a message on a queue and a receiver pulls it off and processes it whenever convenient. It absorbs the processing-speed gap between sender and receiver, and even if the receiver is temporarily down, the message stays queued and can be processed later, making it well suited to loose coupling.
• An ESB (enterprise service bus) centrally handles message brokering, protocol conversion, data transformation, and routing among multiple systems. It can simplify integration between heterogeneous systems, but concentrating too much business logic into the ESB risks the ESB itself becoming a single point of failure or a bottleneck for change—the trap of a bus that grows "smart" and bloated rather than staying dumb plumbing between smart endpoints.
• An API Gateway is the front door that accepts client requests, centrally handles cross-cutting concerns such as authentication, rate limiting, and routing, and then forwards each request to the appropriate microservice. It saves individual services from having to reimplement authentication and rate limiting, but the gateway itself can become a single point of failure or a performance bottleneck, so redundancy is assumed as a prerequisite.

2.5.2Async integration concerns: ordering, duplication, eventual consistency

• Message ordering cannot be assumed to be preserved end to end: in a distributed environment with multiple senders or multiple queue partitions, messages are not guaranteed to arrive at the receiver in the order they were sent. When order matters to the business (for example, the sequence of a deposit and a withdrawal on an account), a design is needed that pins messages sharing the same key (such as an account ID) to the same partition.
• Duplicate delivery (at-least-once delivery): many messaging platforms guarantee only that a message is delivered "at least once," accepting the possibility that the same message arrives more than once due to network retries and similar causes. This makes it essential to design the receiver's processing to be idempotent—producing the same result even if the same message is processed more than once (for example, recording processed message IDs to prevent double processing).
• Eventual consistency: in asynchronous integration, a time gap arises before an update in one service is reflected in another. The design must assume that data may temporarily disagree right after an update, and account for it in how it is presented to users (an "in progress" indicator, etc.) and in compensating logic; for a business that demands instant consistency at all times (for example, a payment where balances must match instantly or the outcome is critical), asynchronous integration itself can be a poor fit.

A system architect is considering switching the integration from the order service to the inventory, payment, and shipping services—currently synchronous API calls—to asynchronous messaging. Under the current synchronous approach, a delayed response from the shipping service used to make the order service wait proportionally, and a failure on the shipping service's side could cause the order-confirmation process itself to fail—a cascading failure that had occurred in the past. The architect therefore migrates to an event-driven architecture in which an order-confirmed event is published to a message queue, and the inventory, payment, and shipping services each pull and process the event whenever convenient. This means even if the shipping service is temporarily down, the order-confirmed event stays queued and processing can resume once the shipping service recovers, substantially reducing the cascading-failure risk. But this migration introduces new concerns. First, if the same order-confirmed event reaches the payment service twice due to a network retry or similar cause, payment processing could execute twice, leading to the serious incident of double-charging, so the payment service must be changed to an implementation with idempotency: recording received event IDs and skipping processing for an ID that has already been handled. Second, if an "order confirmed" event and its cancelling "order cancelled" event are published in extremely close succession for the same order, depending on the messaging infrastructure's configuration, the later cancellation event could arrive first—an ordering flip, so a design is needed that pins events for the same order ID to the same partition to guarantee send order. Furthermore, because a delay of a few seconds to a few dozen seconds can occur between order confirmation and shipping arrangement being reflected, the customer-facing order-status screen was also designed to explicitly show an intermediate state such as "arranging shipment," avoiding the false impression of immediate completion. Migrating to asynchronous messaging thus trades the benefit of suppressing cascading failures for three new design obligations: idempotency, order guarantees, and how eventual consistency is presented.

2.5.3Section summary

• MQ absorbs sender/receiver speed gaps; ESB centralizes heterogeneous integration (watch for bloat); API Gateway centrally handles cross-cutting concerns
• Asynchronous integration requires handling message ordering flips, duplicate delivery, and eventual consistency
• Guard against duplicate delivery by making receiver processing idempotent, and when order matters, pin messages sharing a key to the same partition