Publisher-Subscriber Pattern

1. Overview

The Publisher-Subscriber (or Pub/Sub) pattern is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers. Instead, publishers categorize published messages into classes, without knowledge of which subscribers, if any, there may be. Similarly, subscribers express interest in one or more classes and only receive messages that are of interest, without knowledge of which publishers, if any, there are. This decoupling of publishers and subscribers can allow for greater scalability and a more dynamic network topology. The pattern’s origins can be traced back to early distributed systems and has become a cornerstone of modern cloud-native and microservices architectures.

2. Core Principles

The Publisher-Subscriber pattern is defined by a few core principles that together create a flexible and powerful communication model:

• Decoupling: The most fundamental principle is the decoupling of publishers and subscribers. Publishers are not aware of the subscribers, and subscribers are not aware of the publishers. They only interact through a central message broker.
• Message Broker: A central component, often called a message broker or event bus, is responsible for receiving messages from publishers and delivering them to the appropriate subscribers. This broker filters messages based on topics or channels.
• Topics/Channels: Publishers send messages to specific topics or channels. Subscribers subscribe to these topics to receive messages. This topic-based filtering is what allows for the selective dissemination of information.
• Asynchronous Communication: The communication between publishers and subscribers is inherently asynchronous. Publishers can send messages without waiting for subscribers to receive them, and subscribers can process messages at their own pace.

3. Key Practices

In complex, distributed systems, components often need to communicate with each other. A naive approach is for components to communicate directly. This leads to tight coupling, where each component needs to know the location and identity of the other components it communicates with. This tight coupling creates several problems:

• Scalability: Adding new components or scaling existing ones becomes difficult, as it requires updating the communication logic in all connected components.
• Resilience: If a component is unavailable, any component that communicates with it directly will also be affected, potentially leading to cascading failures.
• Flexibility: It is difficult to change the communication patterns or add new types of communication without modifying the existing components.

4. Implementation

The Publisher-Subscriber pattern addresses these problems by introducing an intermediary, the message broker, between the communicating components. Publishers send messages to the message broker, and the message broker delivers them to the interested subscribers. This approach provides a number of benefits:

• Loose Coupling: Publishers and subscribers are completely decoupled. They don’t need to know about each other’s existence, location, or implementation details.
• Improved Scalability: New publishers and subscribers can be added to the system without affecting the existing components. The message broker can also be scaled independently to handle large volumes of messages.
• Enhanced Resilience: If a subscriber is temporarily unavailable, the message broker can store the messages and deliver them when the subscriber comes back online. This improves the overall resilience of the system.
• Increased Flexibility: The pattern allows for a variety of communication patterns, including one-to-many, many-to-one, and many-to-many. It is also easy to add new types of messages and subscribers without modifying the existing publishers.

5. 7 Pillars Assessment

While the Publisher-Subscriber pattern offers significant advantages, it also introduces some trade-offs and considerations:

Message Delivery Guarantees: Different message brokers offer different guarantees for message delivery (e.g., at-most-once, at-least-once, exactly-once). It is important to choose a message broker that meets the reliability requirements of the application.

Complexity: The introduction of a message broker adds another component to the system that needs to be managed, monitored, and maintained.

6. When to Use

The Publisher-Subscriber pattern is widely used in a variety of applications and systems:

• Social Media: Social media platforms use the pub/sub pattern to deliver updates to users’ feeds. When a user posts an update, it is published to a topic, and all the user’s followers are subscribed to that topic.
• Internet of Things (IoT): In IoT applications, sensors publish data to topics, and various services subscribe to these topics to process the data, trigger alerts, or store it for analysis.
• Microservices Architectures: The pub/sub pattern is a common way for microservices to communicate with each other in an event-driven architecture. This allows for loose coupling and independent deployment of services.
• Financial Systems: Trading systems use the pub/sub pattern to disseminate real-time market data to traders and automated trading systems.

Technologies:

• Apache Kafka: A distributed streaming platform that is often used as a high-throughput message broker in pub/sub systems.
• RabbitMQ: A popular open-source message broker that supports multiple messaging protocols.
• Cloud Services: Cloud providers offer managed pub/sub services, such as AWS Simple Notification Service (SNS), Google Cloud Pub/Sub, and Azure Event Grid.

7. Anti-Patterns & Gotchas

In the cognitive era, where AI and machine learning are becoming increasingly prevalent, the Publisher-Subscriber pattern plays a crucial role in building scalable and responsive AI-powered applications. For example, in a real-time fraud detection system, a stream of financial transactions can be published to a topic. A machine learning model can subscribe to this topic, analyze each transaction in real-time, and publish an alert if it detects a fraudulent transaction. This allows for immediate action to be taken, preventing financial losses.

8. References

The Publisher-Subscriber pattern aligns well with the principles of the Commons, particularly in the context of building open and collaborative platforms:

• Shared Resource: The message broker can be seen as a shared resource that is used by all the components in the system. This promotes the efficient use of resources and avoids duplication of effort.
• Democratic Governance: The pattern allows for a decentralized and democratic form of communication, where any component can publish or subscribe to messages without requiring central approval.
• Equitable Access: All components have equitable access to the message broker and can participate in the communication process on an equal footing.
• Sustainability: The loose coupling and scalability of the pattern contribute to the long-term sustainability of the system, as it can evolve and adapt to changing requirements over time.
• Community Benefit: By enabling the creation of flexible and scalable systems, the Publisher-Subscriber pattern can be used to build platforms that provide significant benefits to a wide community of users.

References

[1] Microsoft. (n.d.). Publisher-Subscriber pattern. Azure Architecture Center. Retrieved from https://learn.microsoft.com/en-us/azure/architecture/patterns/publisher-subscriber [2] Wikipedia. (n.d.). Publish–subscribe pattern. Retrieved from https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern [3] Microservices.io. (n.d.). Publish/Subscribe. Retrieved from https://microservices.io/patterns/communication-style/publish-subscribe.html