Watch Memphis in ActionBook a demo
Modern applications are equally intelligent and complex. To address the growing complexity and scalability challenges, software engineers often adopt a distributed approach such as microservices architecture. In a nutshell, microservice architecture involves breaking down software systems into independent services that represent specific business functions. Each microservice operates independently, enabling seamless development, testing, packaging, and deployment processes.
However, splitting a software system into a set of microservice instances can be challenging due to the need for effective coordination and communication.
This article will delve into messaging in a distributed infrastructure, the concept of message brokers – software applications that enable applications to communicate serially between different microservices using messages, how they work, and their role in communication and coordination between distributed systems or microservices.
Messaging involves the exchange of information between different systems via messages in a loosely coupled manner. Just like you would typically hand over a package to a postal service and trust that the carrier will deliver, so is with messaging. An application passes the messages to a messaging system with the assurance that the information will arrive at its destination.
Most of the communication over the internet happens thanks to the HTTP protocol. The HTTP standard is a stateless protocol built around the request-response communication model where a client sends a request and the server returns a response. For this reason, HTTP is widely used for communication that mainly involves fetching a resource on a service but fails to address concerns such as:
To support the need for these functionalities, other protocols have been built specifically for messaging systems. Most message brokers provide these protocols to address the needs for decoupled systems, high throughput, low latency, and asynchrony. Some of these protocols include:
A message broker provides reliable communication flow by setting up a robust channel for sending, processing, and receiving messages efficiently with scale, resiliency, and fault tolerance in mind. The source application or producer delivers a message to the broker which then translates and sends it to its destination where other applications can consume it (consumers). To make interoperability as seamless as possible, brokers utilize a serial messaging methodology to:
Message transformation and conversion might introduce some performance overhead but having several formats outweighs this in terms of versatility and flexibility.
The publisher is the application that pushes messages to the broker. A publisher writes messages to queues or topics, depending on the communication pattern implemented.
Subscriber is the application that retrieves the message from the broker. They register interest in particular topics/stations of interest and receive a notification when one message is available.
Message brokers often rely on an element known as a message queue to provide reliable message storage and secure delivery. A queue is a data structure that acts as the message buffer for temporary storage until the consumer is ready to consume them. Messages can be ordered in a FIFO manner so that the consumer processes them in a certain order. A queue is a common implementation in a broker like RabbitMQ. With tools like Kafka and Memphis, we have topics and stations respectively. The broadcast mechanism of a queue involves one-to-many communication between the publisher and subscribers. The advantages of using message brokers include the following benefits:
Routing is the mechanism by which messages are directed from publishers to subscribers via topics based on rules. The rules that determine deliveries may include Headers or dynamic filtering based on message content. The broker may give functionality to configure rules and create/ delete topics.
The protocol adapter is the bridge that allows brokers to support different protocols such as MQTT, STOMP, or AMQP. In this way, the broker can support a wide range of applications even though they have different native protocols.
Transformers manipulate the messages as they pass through the broker. Transformers can perform operations such as converting formats, enhancing data, and validating data on messages.
Messages are distributed by various patterns to reach subscribers. These patterns include:
Publisher/ Subscriber: In a Pub/Sub pattern, producers or publishers push messages on a particular topic or endpoint to which consumers or subscribers can selectively subscribe. The producer can broadcast messages without knowing the subscribed consumers while multiple producers can publish on the same topic and messages can be received by multiple subscribers. This is known as a one-to-many relationship between the sender and receiver. This feature is built into the broker to ease integration needs.
Point-to-Point: Point-to-point messaging model is common in one-to-one communication between applications (we have one subscriber and one consumer). The producer writes messages where they are stored until the consumer is ready to receive them. Once received, the consumer can send back an acknowledgment to notify that the message is received. This type of communication is known as point-to-point messaging. Each message in the queue is sent to only one recipient and is consumed only once.
Request/ Reply messaging: This pattern involves publishing a request and then either waiting for a response with a specified time limit or receiving the response asynchronously on a designated subject.
Brokers serve as intermediaries in the communication between the originator of a message (the publisher) and the intended recipient (the subscriber). The broker guarantees message delivery from publishers and manages their distribution to the relevant subscribers, to complete a business process.
On the other hand, events contain information that represents a point-in-time fact or sequence of occurrences. They are descriptions of things that have happened, such as a user action, a change in a system’s state, or the completion of a task that might be of interest to other systems or services. Event propagation systems capture the relevant details, store and propagate them to other services or applications. They mainly work with the pub/sub pattern where services subscribe to the events of interest and respond to them in a timely defined manner. Event propagation platforms will manage data propagation and storage; they are accountable for transmitting and preserving information to other services or systems.
A message broker is highly centralized (but can be scaled with clustering). It receives messages from different sources and propagates them to the appropriate channel, employing asynchronous processing to manage the flow effectively. Therefore, in a distributed infrastructure, brokered messaging happens through this central system. On the other hand, a Service Bus is a highly decentralized system where services and applications interact directly over a shared set of communication interfaces. Service bus practice is common in Service Oriented Architecture (SOA) where direct and immediate communication between services is the main focus.
Message brokers are often more scalable than message buses for handling increased loads. Their centralized messaging logic makes it easier to add more instances, while the service bus may need complex strategies like partitioning into multiple instances for scaling.
Selecting the appropriate tool for your business is critical in driving growth and giving you an advantage over your competitors. We can utilize a message broker for a variety of business purposes, including:
Businesses move to the cloud to tap the ubiquitous and on-demand pool of remote computing resources with minimal management, scalability, and reliability. In cloud-based solutions, message brokers play a critical role in enabling the communication of loosely coupled microservices. In scenarios where services need a response for real-time events like CRUD on databases, brokers can implement an event-driven pattern to publish and consume events for multiple subscribers. All this happens without the need to hard-code hostnames for service discovery.
Handling loads of data streaming, collection, and processing such as:
IoT and embedded systems involve several devices and sensors with diverse capabilities and resource constraints. It is essential to prioritize efficient, low-latency communication using a message broker and various protocols, like MQTT.
Despite its low resource consumption, devices can easily integrate with remote cloud systems to collect, process, and analyze real-time data, with the message broker serving as the central coordinating hub.
As businesses increasingly adopt distributed practices, message brokers have become a critical piece of technology for the communication layer. Broker systems enhance interoperability mechanisms across different platforms, programming languages, and protocols. With a wide range of use cases from cloud-native, serverless, edge computing, or IoT, a broker acts as the standard middleware that provides an interface for communication that scales on demand. Choosing the right message broker is a key point for businesses. Memphis.dev focuses on four pillars: stability, performance and efficiency, observability, and developer experience which maximize the advantages and minimize the disadvantages of the traditional message brokers. You should consider learning more about related concepts such as advanced message queuing protocol, and Java message service.
To get started with a next-generation message broker specifically for real-time data streaming, check out Memphis.dev.