What is a message queue?

Implementation of messaging protocols is a fundamental communication component for modern applications. Different application components require transmitting messages while ensuring reliable and efficient communication. A good example of ensuring reliable message exchange is implementing a pipeline. Once a message is assigned to the pipeline, its destination is already established. This message will always get delivered. Moreover, if the destination is busy or not connected to the queue, the pipeline will hold that message until the destination gets back. You require a message queueing system (pipeline) to implement these advanced queuing operations like message persistence, message priority, scheduled messages, time-to-live, etc.

What is a message queue?

A communication contains a payload of the actual data being transmitted or received. These messages are dispatched by a producer and received by a consumer. On the other side, a message queue is a line (MQ) that stores and manages the flow of these messages in a sequential first-in, first-out (FIFO) order until a consumer retrieves them. It holds the messages waiting to be processed. These messages will then be enqueued and dequeued in the order they are received to ensure asynchronous communication between the sender and the receiver.

In that context, it allows applications to exchange and receive messages through a queue. These messages are stored in a pipeline until the receiving application processes them. It allows asynchronous communication by ensuring a service can transmit a notification to the queue and continue processing. In contrast, another service can receive the message and process it in a timely approach.

A perfect hypothetical example of how message queues (MQ) can be used is an e-commerce app that uses a message queue to process orders. In this case, when a customer places an order, the app dispatches a notification to the queue with the order details. This message payload will include details such as the customer’s shipping address, the products being ordered, and the total cost of the order.

The same application will have another service (fulfillment service) that receives the message from the pipeline and processes the order. While using a message queue, the application can still process other consumers’ incoming orders while the fulfillment service takes care of the orders as they are received. The system remains scalable, as the order service doesn’t need to wait for the fulfillment service to complete the order before processing the incoming orders.

The same application will have another service (fulfillment service) that receives the message from the pipeline and processes the order. While using a message queue, the application can still process other consumers’ incoming orders while the fulfillment service takes care of the orders as they are received. The system remains scalable, as the order service doesn’t need to wait for the fulfillment service to complete the order before processing the incoming orders.

A queue manager functions as a software application that delivers messaging services to various applications. This is to guarantee the accurate delivery of messages to their designated pipelines or their redirection to a different manager if required.

Message queue architecture

A message queue uses a straightforward architecture:

• Producers – It consists of client components that generate messages.
• Consumers – It comprises server destination services that process and store messages in a queue — usually in a buffer or on a storage medium — until they can process and delete them. The consumer connects to the queue and gets the messages to be processed. Messages placed onto the queue are stored until the consumer retrieves them.
• The queue – The central component of the architecture that stores messages.

When a consumer processes a task, it is removed from the queue. However, this process may differ based on the style of message pipeline being used.

Types of message queues

Message queues style differs in characteristics. The major message queues types are:

• Point-to-point queues

Point-to-point queues send messages from a single producer to a single consumer. This method is commonly used for simple and one-to-one communication between services.

An excellent example of a point-to-point queue is communication between a web app and a backend server. In this case, the app sends a message containing a user request to the queue. The server then processes the request and sends a response back to the app. Messages are only produced by one consumer, received by one consumer, and then removed from the queue.

• Publish-subscribe pattern

This approach dispatches messages from a producer (Publishers) to multiple consumers (subscribers). A consumer can subscribe to specific message categories and receive messages that match the respective subscription. The Pub/Sub pattern is essential whenever you want to distribute messages to multiple recipients.

A good example that fits the Pub/Sub pattern is news systems. For example, a financial news system can route messages to multiple subscribers based on specific subscriptions, such as stock prices and market trends. All subscribers who have subscribed to the relevant category will receive the news update.

• Fanout queues

Fanout queues are closely related to Pub/Sub patterns. Messages are sent from a single producer to multiple consumers. However, it uses a fan-out pattern. All connected consumers receive a message sent by the producer simultaneously.

Fanouts a commonly used to create real-time event-driven applications. Take an example of a Social media platform. Any post you add to your timeline is sent to all your followers in real time. The significant advantage of Fanouts is the ability to implement real-time communication. Chat application uses a real-time event-driven architecture to send in group chat communication. All group members receive a message sent by one member.

• Dead letter queues

When a producer sends the message to the queue, the consumer may not be available to receive the dispatched message. In this case, the message will be stored in the queue until its receiver becomes available. Dead letter queues are used to store messages that cannot be delivered to the intended recipient at a given time.

Take an email application as an example, specifically the email delivery service. In this case, if you send an email that fails to be delivered to the intended recipient, it is stored in a dead letter queue for further examination or re-routed to a different recipient.

• Priority queues

Messages stored in a queue are treated using a first-in, first-out (FIFO) approach. This means the first received messages will be processed first, and others will follow in the same order. However, you can use queues to treat messages based on priority levels.

Priority queues are used to store and manage messages with different priority levels. Messages are processed based on priority. Higher-priority messages are processed first, and lower-priority messages come last.

A priority queue allows you to handle time-sensitive and crucial messages. Take an example an app that sells VVIP, VIP, and regular products. In this case, a priority queue can be used to process orders based on the levels you have set.

Benefits of message queuing

Message queues provide tone benefits for modern application development. Based on this post, let’s check out a few advantages you can leverage:

• Message queues strive best at allowing application uncoupling. A single application can be divided into different microservice components. Each component runs independently. These microservices can communicate using message queues without being tightly coupled in a monolith architecture. A producer and consumer don’t necessarily need to be connected simultaneously. Each service is self-contained.
• Decoupled microservices provide application resilience and reliability. If a receiver is not able to process a message at the time it is received, the message can be stored in a queue and processed later.
• It creates fault tolerance models. The message will be re-delivered when the destination node is back online. This is necessary when your application is experiencing high traffic and when down for maintenance.
• A message queue has the capacity to receive a large number of messages while the consumer is still processing them accordingly. This creates scalability advantages. You can easily scale to handle more messages as the system grows. Producers are only assigned to create messages, and consumers take and deliver the messages to the intended destination. This application handles increasing loads without requiring significant changes to the architecture.
• Uncoupling your application and allowing the message queue to handle messages asynchronously improves overall performance. Consumers and producers continue with their tasks without waiting for each other. Each component pushes messages into a pipeline and retrieves them when ready. This allows the producer to free up bandwidth after delivering messages. The consumers always process new incoming messages in the background without clogging the application, and thus performance is improved.
• Message queuing provides a more straightforward monitoring approach. You can get transaction insights such as sent, received, failed, rejected, and retried messages. This provides monitoring statistics to allow you to understand performance and how data flows in your system.
• Uses advanced protocols such as Advanced Message Queuing Protocol (AMQP), AUTO, MQTT, RSS (Really Simple Syndication), Atom, and more to provide flexible and reliable delivery for distributed systems.

Message queues use cases

Message queues can be applied based on the above benefits, and this post has discussed queue types. The everyday use cases examples where they are used include:

• E-commerce application to manage the flow of services such as orders, inventory updates, shipping, and delivery.
• Social media apps use queues to manage notifications.
• Chat applications require queues to create real-time messaging platforms.
• A gaming application can use queues to handle matchmaking.
• The Internet of Things (IoT) uses different services to manage sensor data and device updates. This makes it effortless to implement and manage data flow from different sources and their respective destinations.

Conclusion

Message queues are a powerful tool for creating scalable, fault-tolerant applications. This goes in hand with their ability to decouple operations and enhance overall application reliability. You can use a message pipeline for flexibility and make it easier to scale your app. This makes pipelines a valuable addition to your application architecture.

For an even more solid knowledge, do check out our article on what is a message broker.