What is a message broker?

Modern applications are equally intelligent and complex. To address the growing complexity and scalability challenges, software engineers often adopt a distributed approach such as microservices architecture. In a nutshell, microservice architecture involves breaking down software systems into independent services that represent specific business functions. Each microservice operates independently, enabling seamless development, testing, packaging, and deployment processes.

However, splitting a software system into a set of microservice instances can be challenging due to the need for effective coordination and communication.

This article will delve into messaging in a distributed infrastructure, the concept of message brokers – software applications that enable applications to communicate serially between different microservices using messages, how they work, and their role in communication and coordination between distributed systems or microservices.

What is messaging in a message broker?

Messaging involves the exchange of information between different systems via messages in a loosely coupled manner. Just like you would typically hand over a package to a postal service and trust that the carrier will deliver, so is with messaging. An application passes the messages to a messaging system with the assurance that the information will arrive at its destination.

Most of the communication over the internet happens thanks to the HTTP protocol. The HTTP standard is a stateless protocol built around the request-response communication model where a client sends a request and the server returns a response. For this reason, HTTP is widely used for communication that mainly involves fetching a resource on a service but fails to address concerns such as:

• Scale and efficiency: With HTTP, we have to create a new connection for every request to access the underlying resources. Its stateless nature makes it inefficient to handle persistent connections with high throughput and concurrent connections. 
• Reliability: With HTTP, it is difficult to acknowledge every instance of message delivery when issues such as network failure or system crash arise.
• Low Latency: HTTP does not address low overhead messaging in scenarios in cases like IoT environments where demands for low latency and async communication are crucial.
• Inefficient Pub/Sub model: It does not have a direct way of addressing pub/sub pattern to support multiple consumers that can retrieve messages from a single producer.

To support the need for these functionalities, other protocols have been built specifically for messaging systems. Most message brokers provide these protocols to address the needs for decoupled systems, high throughput, low latency, and asynchrony. Some of these protocols include:

• AMQP (Advanced Message Queuing Protocol): An open standard for multiple message queuing. AMQP works at the application layer to ensure a standard protocol.
• MQTT (Message Queue Telemetry Transport): MQTT is a lightweight protocol for use in low-power or embedded devices with constraints such as network and bandwidth.
• STOMP (Simple Text Oriented Messaging Protocol): An easy-to-implement protocol for sending messages on top of popular transport layers such as TCP, WebSockets, and more.

How does a message broker work?

A message broker provides reliable communication flow by setting up a robust channel for sending, processing, and receiving messages efficiently with scale, resiliency, and fault tolerance in mind. The source application or producer delivers a message to the broker which then translates and sends it to its destination where other applications can consume it (consumers). To make interoperability as seamless as possible, brokers utilize a serial messaging methodology to:

• Route Messages: Within its architecture, the message broker can implement topic endpoints to reach its subscribers.
• Offer persistence, acknowledgments, and guaranteed delivery: When a consumer receives a message it should be acknowledged otherwise it goes back to the queue for retrials. In scenarios where acknowledgment on the consumer side keeps failing, the message broker sends the message in a designated Channel or Dead Letter Queue for further retries and inspection.
• Provide message transformation: Message brokers facilitate communication between applications with diverse formats and protocols, such as JSON and XML. The broker may be designed to natively support either XML or JSON formats, but in some cases, users may be limited to using other formats, particularly in low-level systems that handle binary data. To address this, the broker architecture includes a component that enables message format transformations between systems. This feature makes the broker flexible and adaptable to a variety of use cases.

Message transformation and conversion might introduce some performance overhead but having several formats outweighs this in terms of versatility and flexibility.

The message broker model

Publisher

The publisher is the application that pushes messages to the broker. A publisher writes messages to queues or topics, depending on the communication pattern implemented.

Subscriber

Subscriber is the application that retrieves the message from the broker. They register interest in particular topics/stations of interest and receive a notification when one message is available.

Queues, Topics, or Stations

Message brokers often rely on an element known as a message queue to provide reliable message storage and secure delivery. A queue is a data structure that acts as the message buffer for temporary storage until the consumer is ready to consume them. Messages can be ordered in a FIFO manner so that the consumer processes them in a certain order. A queue is a common implementation in a broker like RabbitMQ. With tools like Kafka and Memphis, we have topics and stations respectively. The broadcast mechanism of a queue involves one-to-many communication between the publisher and subscribers. The advantages of using message brokers include the following benefits:

• It ensures reliable message retention and assured delivery even if the consumer service is unavailable at the time of message the producer is publishing messages.
• It guarantees flexible async communication between the sender and the receiver. A producer can push messages to the queue while the consumer can process them at its own pace.
• We can offload processes within the stations or topics ensuring less overhead on the producer and consumer side.

Router

Routing is the mechanism by which messages are directed from publishers to subscribers via topics based on rules. The rules that determine deliveries may include Headers or dynamic filtering based on message content. The broker may give functionality to configure rules and create/ delete topics.

Protocol Adapter

The protocol adapter is the bridge that allows brokers to support different protocols such as MQTT, STOMP, or AMQP. In this way, the broker can support a wide range of applications even though they have different native protocols.

Transformers

Transformers manipulate the messages as they pass through the broker. Transformers can perform operations such as converting formats, enhancing data, and validating data on messages.

How does a broker distribute messages?

Messages are distributed by various patterns to reach subscribers. These patterns include:

Publisher/ Subscriber: In a Pub/Sub pattern, producers or publishers push messages on a particular topic or endpoint to which consumers or subscribers can selectively subscribe. The producer can broadcast messages without knowing the subscribed consumers while multiple producers can publish on the same topic and messages can be received by multiple subscribers. This is known as a one-to-many relationship between the sender and receiver. This feature is built into the broker to ease integration needs.

Point-to-Point: Point-to-point messaging model is common in one-to-one communication between applications (we have one subscriber and one consumer). The producer writes messages where they are stored until the consumer is ready to receive them. Once received, the consumer can send back an acknowledgment to notify that the message is received. This type of communication is known as point-to-point messaging. Each message in the queue is sent to only one recipient and is consumed only once.

Request/ Reply messaging: This pattern involves publishing a request and then either waiting for a response with a specified time limit or receiving the response asynchronously on a designated subject.

Message broker vs event streaming

Brokers serve as intermediaries in the communication between the originator of a message (the publisher) and the intended recipient (the subscriber). The broker guarantees message delivery from publishers and manages their distribution to the relevant subscribers, to complete a business process.

On the other hand, events contain information that represents a point-in-time fact or sequence of occurrences. They are descriptions of things that have happened, such as a user action, a change in a system’s state, or the completion of a task that might be of interest to other systems or services. Event propagation systems capture the relevant details, store and propagate them to other services or applications. They mainly work with the pub/sub pattern where services subscribe to the events of interest and respond to them in a timely defined manner. Event propagation platforms will manage data propagation and storage; they are accountable for transmitting and preserving information to other services or systems.

Message broker vs service bus

A message broker is highly centralized (but can be scaled with clustering). It receives messages from different sources and propagates them to the appropriate channel, employing asynchronous processing to manage the flow effectively. Therefore, in a distributed infrastructure, brokered messaging happens through this central system. On the other hand, a Service Bus is a highly decentralized system where services and applications interact directly over a shared set of communication interfaces. Service bus practice is common in Service Oriented Architecture (SOA) where direct and immediate communication between services is the main focus.

Message brokers are often more scalable than message buses for handling increased loads. Their centralized messaging logic makes it easier to add more instances, while the service bus may need complex strategies like partitioning into multiple instances for scaling.

Examples of message brokers

1. NATS is an open-source cloud-native broker written in Golang that implements the “fire and forget” pattern. NATS is widely used as a broker or service mesh in microservice-based platforms.
2. RabbitMQ is widely used message broker that runs on top of the Erlang BEAM virtual machine to provide a robust, concurrent, and fault-tolerant messaging layer. Messages are sent to the exchange first rather than queues. It implements many messaging protocols including AMQP, MQTT, and streams.
3. Memphis.dev is a next-generation alternative to traditional message brokers. It enables building modern list-based applications that require large volumes of streamed and enriched data, modern protocols, zero ops, up to x9 faster development, up to x46 fewer costs, and significantly lower dev time for data-oriented developers and data engineers. Low footprint, highly resilient, cloud-native, and runs on any Kubernetes, on any cloud.
4. Apache Kafka is an open-source distributed event streaming platform built on open standards to handle the scalability of real-time data pipelines and streaming workflows.
5. Popular cloud vendors offer messaging services like ActiveMQ, SQS for AWS, Google PubSub+, Azure Service Bus for Microsoft Azure, Amazon MQ – an alternative cloud version of RabbitMQ and Apache ActiveMQ provided by Amazon Web Services, to enable easy scaling and high availability for cloud-native applications.

Message broker use cases

Selecting the appropriate tool for your business is critical in driving growth and giving you an advantage over your competitors. We can utilize a message broker for a variety of business purposes, including:

Cloud-native environment and microservices

Businesses move to the cloud to tap the ubiquitous and on-demand pool of remote computing resources with minimal management, scalability, and reliability. In cloud-based solutions, message brokers play a critical role in enabling the communication of loosely coupled microservices. In scenarios where services need a response for real-time events like CRUD on databases, brokers can implement an event-driven pattern to publish and consume events for multiple subscribers. All this happens without the need to hard-code hostnames for service discovery.

Real-time streaming and analytics

Handling loads of data streaming, collection, and processing such as:

• Log analysis and monitoring: Message brokers can gather and manage system and application logs, forwarding them to real-time monitoring tools.
• Processing financial transactions: Modern financial systems generate large amounts of real-time data that need to be aggregated, processed, and analyzed. A message broker can sit between applications as the hub for collecting this data from different sources and forwarding it for further analysis that might involve: risk analysis, fraud detection, or financial reporting.

IoT and embedded systems

IoT and embedded systems involve several devices and sensors with diverse capabilities and resource constraints. It is essential to prioritize efficient, low-latency communication using a message broker and various protocols, like MQTT.

Despite its low resource consumption, devices can easily integrate with remote cloud systems to collect, process, and analyze real-time data, with the message broker serving as the central coordinating hub.

Benefits of message brokers

• Flexibility: A message broker is the backbone of any loosely coupled communication between systems or services. More and more applications and services can connect to the broker and expose their functionality to other services. This approach ensures a frictionless approach when we need to add, update, or remove applications without impacting or breaking the entire system. Additionally, message brokers offer various messaging patterns like point-to-point or pub/sub to meet different needs.
• Fault tolerance and resiliency: A message broker can handle a high throughput of messages that survives crashes and downtimes. Most message brokers store messages until the consuming applications can process them. They do not store messages in memory and instead use a persistent layer to guarantee that no messages are lost. Receivers have a reliable delivery channel with acknowledgment, automatic failovers, and retries.
• Scalability: Message broker enables asynchronous messaging and ensures non-blocking performance across multiple receivers, compared to APIs. To address system scalability, we add more message brokers horizontally without changing the design of the individual brokers. The additional brokers distributed the workload effectively to meet the scaling demands.

Challenges of message brokers

• Complexity in overall system architecture: Saying that distributed system practice is complex is an understatement. As instances of services and applications increase, we may need to add more message brokers. However, this will result in complex systems, requiring careful configuration and management to ensure proper system functioning thus increasing the time for debugging, maintenance, and troubleshooting. Modern brokers, such as Memphis, simplify this approach by offering features that automate management tasks, including schema management, Ops configurations, and interfaces for tracing and monitoring.
• Subject to security vulnerabilities: Message brokers will most likely be used to store and transmit sensitive data, which makes them a potential target for attackers. We often have to configure and comply with security best practices like encryption, authentication, and access controls, to prevent unauthorized access or data loss.