Modern applications are equally intelligent and complex. To address the growing complexity and scalability challenges, software engineers often adopt a distributed approach such as microservices architecture. In a nutshell, microservice architecture involves breaking down software systems into independent services that represent specific business functions. Each microservice operates independently, enabling seamless development, testing, packaging, and deployment processes.
However, splitting a software system into a set of microservice instances can be challenging due to the need for effective coordination and communication. One solution that solves this issue effectively is a message broker.
This article will delve into messaging in a distributed infrastructure, the concept of message brokers, how they work, and their role in communication between distributed systems.
What is messaging in a message broker?
Messaging involves the exchange of information between different systems via messages in a loosely coupled manner. Just like you would typically hand over a package to a postal service and trust that the carrier will deliver, so is with messaging. An application passes the messages to a messaging system with the assurance that the information will arrive at its destination.
Most of the communication over the internet happens thanks to the HTTP protocol. The HTTP standard is a stateless protocol built around the request-response communication model where a client sends a request and the server returns a response. For this reason, HTTP is widely used for communication that mainly involves fetching a resource on a service but fails to address concerns such as:
- Scale and efficiency: With HTTP, we have to create a new connection for every request to access the underlying resources. Its stateless nature makes it inefficient to handle persistent connections with high throughput and concurrent connections.
- Reliability: With HTTP, it is difficult to acknowledge every instance of message delivery when issues such as network failure or system crash arise.
- Low Latency: HTTP does not address low overhead messaging in scenarios in cases like IoT environments where demands for low latency and async communication are crucial.
- Inefficient Pub/Sub model: It does not have a direct way of addressing pub/sub pattern to support multiple consumers that can retrieve messages from a single producer.
To support the need for these functionalities, other messaging protocols have been built specifically for messaging systems. Most message brokers implement these protocols to address the needs for decoupled systems, high throughput, low latency, and asynchrony. Some of these protocols include:
- AMQP (Advanced Message Queuing Protocol): An open standard for message queuing. AMQP works at the application layer to ensure a standard messaging protocol for all messaging systems.
- MQTT (Message Queue Telemetry Transport): MQTT is a lightweight protocol for use in low-power or embedded devices with constraints such as network and bandwidth.
- STOMP (Simple Text Oriented Messaging Protocol): An easy-to-implement messaging protocol for sending messages on top of popular transport layers such as TCP, WebSockets, and more.
How does a message broker work?
A message broker facilitates communication flow by setting up a robust channel for sending, processing, and receiving messages efficiently with scale, resiliency, and fault tolerance in mind. The source application or producer delivers a message to the broker which then translates and routes them to its destination where other applications can consume them (consumers). To make interoperability as seamless as possible, brokers utilize an asynchronous messaging architecture to:
- Route Messages: Within its architecture, the message broker can implement queues or topic endpoints to reach its subscribers.
- Offer persistence, acknowledgments, and guaranteed delivery: When a consumer receives a message it should be acknowledged otherwise the message goes back to the queue for retrials. In scenarios where acknowledgment on the consumer side keeps failing, the message broker routes the message in a designated Channel or Dead Letter Queue for further retries and inspection.
- Provide message transformation: Message brokers facilitate communication between applications with diverse formats and protocols, such as JSON and XML. The broker may be designed to natively support either XML or JSON formats, but in some cases, users may be limited to using other formats, particularly in low-level systems that handle binary data. To address this, the broker architecture includes a component that enables message format transformations between systems. This feature makes the broker flexible and adaptable to a variety of use cases.
Message transformation and conversion might introduce some performance overhead but having several formats outweighs this in terms of versatility and flexibility.
The message broker model
Publisher
The publisher is the application that pushes messages to the broker. A publisher writes messages to queues or topics, depending on the communication pattern implemented.
Subscriber
Subscriber is the application that retrieves the message from the broker. They register interest in particular queues or topics/ stations of interest and receive a notification when a new message is available.
Queues, Topics, or Stations
A queue is a data structure that acts as the message buffer for temporary storage until the consumer is ready to consume them. Messages can be ordered in a FIFO manner so that the consumer processes them in a certain order. A queue is a common implementation in a broker like RabbitMQ. With tools like Kafka and Memphis, we have topics and stations respectively. The broadcast mechanism of a queue involves one-to-many communication between the publisher and subscribers. Having a queue or topic in a message broker has various benefits:
- A queue ensures reliable message retention and assured delivery even if the consumer service is unavailable at the time of message the producer is publishing messages.
- It guarantees flexible asynchronous communication between the sender and the receiver. A producer can push messages to the queue while the consumer can process them at its own pace.
- We can offload processes within the stations or topics ensuring less overhead on the producer and consumer side.
Router
Routing is the mechanism by which messages are directed from publishers to subscribers via topics or queue endpoints based on routing rules. The routing rules that determine deliveries may include Headers or dynamic filtering based on message content. The broker may give functionality to configure routing rules and create/ delete topics.
Protocol Adapter
The protocol adapter is the bridge that allows brokers to support different protocols such as MQTT, STOMP, or AMQP. In this way, the broker can support a wide range of applications even though they have different native protocols.
Transformers
Transformers manipulate the messages as they pass through the broker. Transformers can perform operations such as converting formats, enhancing data, and validating data on messages.
How does a broker distribute messages?
Message brokers can use various distribution patterns to reach subscribers. These patterns include:
Publisher/ Subscriber: In a Pub/Sub pattern, producers or publishers push messages on a particular topic or endpoint to which consumers or subscribers can selectively subscribe. The producer can broadcast messages without knowing the subscribed consumers while multiple producers can publish on the same topic and messages can be received by multiple subscribers. This feature is built into the broker to ease integration needs.
Point-to-Point: point-to-point communication is common in a one-to-one communication between applications (we have one subscriber and one consumer). The producer writes messages to the queue where they are stored until the consumer is ready to receive them. Once received, the consumer can send back an acknowledgment to notify that message is received.
Request/ Reply messaging: This pattern involves publishing a request and then either waiting for a response with a specified time limit or receiving the response asynchronously on a designated subject.
Message broker vs event streaming
Message brokers serve as intermediaries in the communication between the originator of a message (the publisher) and the intended recipient (the subscriber). The broker facilitates message delivery from publishers and manages their distribution to the relevant subscribers, to complete a business process.
On the other hand, events contain information that represents a point-in-time fact or sequence of occurrences. They are descriptions of things that have happened, such as a user action, a change in a system's state, or the completion of a task that might be of interest to other systems or services. Event Streaming systems capture the relevant details, store and propagate them to other services or applications. They mainly work with the pub/ sub pattern where services subscribe to the events of interest and respond to them in a timely defined manner. Event Streaming platforms will manage data propagation and storage; they are accountable for transmitting and preserving information to other services or systems.
Message broker vs service bus
A message broker is highly centralized (but can be scaled with clustering). It receives messages from different sources and propagates them to the appropriate channel. Therefore in a distributed infrastructure, brokered messaging happens through this central system. A Service Bus is a highly decentralized system where services and applications interact directly over a shared set of communication interfaces. Service bus practice is common in Service Oriented Architecture (SOA) where direct and immediate communication between services is the main focus.
Message brokers are often more scalable than message buses for handling increased loads. Their centralized routing logic makes it easier to add more instances, while service bus may need complex strategies like partitioning into multiple instances for scaling.
Examples of message brokers
- NATS: NATS is an open-source cloud native messaging system written in Golang that implements the “fire and forget” messaging pattern. NATS is widely used as a broker or service mesh in microservice-based platforms.
- RabbitMQ: A widely used message broker that runs on top of the Erlang BEAM virtual machine to provide a robust, concurrent, and fault-tolerant messaging layer. Messages route to the exchange first rather than queues. It implements multiple protocols including AMQP, MQTT, and streams.
- Memphis: Memphis broker is a platform written in Golang, a language known for scalability and concurrency. It transforms real-time data streaming with features that enhance and streamline data-oriented development processes, such as out-of-the-box optimization, effortless Zero Ops, unparalleled debugging tools, and seamless schema management. The broker comes with an intuitive interface to manage and monitor applications.
- Apache Kafka: Apache Kafka is an open-source distributed event streaming platform built on open standards to handle the scalability of real-time data pipelines and streaming workflows.
- Popular cloud vendors offer messaging services like Amazon MQ and SQS for AWS, Google PubSub+, and Azure Service Bus for Microsoft Azure, to enable easy scaling and high availability for cloud-native applications.
Message broker use cases
Selecting the appropriate tool for your business is critical in driving growth and giving you an advantage over your competitors. We can utilize a message broker for a variety of business purposes, including:
Cloud-native environment and microservices
Businesses move to the cloud to tap the ubiquitous and on-demand pool of remote computing resources with minimal management, scalability, and reliability. In cloud-based solutions, message brokers play a critical role in enabling the communication of loosely coupled microservices. In scenarios where services need a response for real-time events like CRUD on databases, brokers can implement an event-driven pattern to publish and consume events for multiple subscribers. All this happens without the need to hard-code hostnames for service discovery.
Real-time streaming and analytics
Handling loads of data streaming, collection, and processing such as:
- Log analysis and monitoring: Message brokers can gather and manage system and application logs, forwarding them to real-time monitoring tools.
- Processing financial transactions: Modern financial systems generate large amounts of real-time data that need to be aggregated, processed, and analyzed. A message broker can sit between applications as the hub for collecting this data from different sources and forwarding it for further analysis that might involve: risk analysis, fraud detection, or financial reporting.
IoT and embedded systems
IoT and embedded systems involve several devices and sensors with diverse capabilities and resource constraints. It is essential to prioritize efficient, low-latency communication using a message broker and various protocols, like MQTT.
Despite its low resource consumption, devices can easily integrate with remote cloud systems to collect, process, and analyze real-time data, with the message broker serving as the central coordinating hub.
Benefits of message brokers
- Flexibility: A message broker is the backbone of any loosely coupled communication between systems or services. More and more applications and services can connect to the broker and expose their functionality to other services. The architecture assures a frictionless approach when we need to add, update, or remove applications without impacting or breaking the entire system. Additionally, message brokers offer various messaging patterns like point-to-point or pub/sub to meet different needs.
- Fault tolerance and resiliency: A message broker can handle a high throughput of messages that survives crashes and downtimes. Most message brokers will not store messages in memory and instead use a queue or a persistent layer to guarantee that no messages are lost. Receivers have a reliable delivery channel with acknowledgment, automatic failovers, and retries.
- Scalability: Message passing using asynchronous communication ensures non-blocking performance across multiple receivers, compared to APIs. To address system scalability, we add more message brokers horizontally without changing the architecture or design of the individual brokers. The additional brokers distributed the workload effectively to meet the scaling demands.
Challenges of message brokers
- Complexity in overall system architecture: Saying that distributed system practice is complex is an understatement. As instances of services and application increase, we may need to add more message brokers. However, message brokers can be complex systems, requiring careful configuration and management to ensure proper system functioning thus increasing the time for debugging, maintenance, and troubleshooting. Modern brokers, such as Memphis, simplify this approach by offering features that automate management tasks, including schema management, Ops configurations, and interfaces for tracing and monitoring.
- Subject to security vulnerabilities: Message brokers will most likely be used to store and transmit sensitive data, which makes them a potential target for attackers. We often have to configure and comply with security best practices like encryption, authentication, and access controls, to prevent unauthorized access or data loss.
Conclusion
As businesses increasingly adopt distributed practices, message brokers have become a critical piece of technology for the communication layer. Broker systems enhance interoperability mechanisms across different platforms, programming languages, and protocols. With a wide range of use cases from cloud-native, serverless, edge computing, or IoT, a messaging system acts as the standard middleware that provides an interface for communication that scales on demand.
This article explains the approach of the message broker in meeting the requirements of distributed architecture practices and how to leverage its features. To get started with a next-generation message broker specifically for real-time data streaming, check out Memphis.dev.
