gRPC vs Message Broker

Microservices architecture allows large software systems to be broken down into smaller independent units called services.
The services in a microservices-based system need a way to effectively transfer data and events with one another.

There are several ways to connect services together. Usually, the first thing that may come to mind is to create REST endpoints for one service that other services can call.
While this can work in systems with a few services, it’s not scalable or performant in larger systems. This is because REST works based on a blocking request-response mechanism.

A better way to connect microservices is to use a protocol that offers faster request times or use a non-blocking mechanism to get tasks done. Two technologies that enable this are gRPC and message brokers. gRPC sends binary data over the network and has a faster request time, but gRPC is not always the perfect choice as it requires –

• Memory is to be allocated on the receiving end to handle response.
• Not suitable for one producer to many clients pattern.
• The client has to absorb and handle the “back pressure”.

Out of those requirements comes a message broker.

This article goes over the similarities and differences between gRPC and message brokers. You will learn about the pros, cons, and ideal use cases of both technologies.

What is gPRC?

gRPC (which is short for Google Remote Procedure Call) is a communication protocol that is used in place of REST to communicate in a client-server architecture.

The client will make a proto request to the server, and the server responds with a proto response.

image source: https://grpc.io

gRPC uses HTTP 2.0, which is faster than HTTP 1.1, that REST depends on. HTTP 2.0 enforces a binary format by default. This means protocols using HTTP 2.0 need to serialize their data to binary before sending requests over the network. gRPC uses Protobuf, a binary serialization format, for defining data schemas.

gRPC also supports data streaming which allows a single request to return a lot more data than it would typically be in the case of REST. This data streaming can either be server-to-client streaming or bi-direction streaming between client-to-server. Note that the service called the client is the service that makes the initial request, while the server is the service that sends the response.

What is a Message Broker?

A message broker is a server that contains a persistent, append-only log that stores messages.
Ingested data or messages get stored within the log files where they are persistent and available for consumption by the subscribed consumers.

An example use case is an image processing pipeline that converts a large image into smaller images of various sizes.
The conversion task takes an average of 10 seconds per image, but you have a thousand users trying to convert their images into different sizes.
Instead of having direct communication between the different clients with the resizing service, which puts it at risk of a crash, each conversion task gets stored in a queue within the message broker and gets consumed by the conversion server, when its resources allows it to do so. This process prevents the server from being overwhelmed.

Async REST communication In need of a many-to-many message pattern and support of various programming languages? Try Memphis.dev Join our live demo

Similarities between gRPC and Message Brokers

Streaming support

gRPC and message brokers also support streaming. This means data can be sent from the server to the client as soon as the data is introduced. An example of this is sending resized images from the image resizer service to the image watermarking service as soon as the image is resized. The image data can either be queued in a message broker or sent in an RPC call. In either case, the image data is streamed from the image resizer service to the image watermarking service.

Language agnostic

gRPC and message brokers are mostly language agnostic, with support in a variety of languages. You can use gRPC in the most widely used languages like Python, JavaScript, Java, Go, and C#. You can also connect to a message broker like Memphis using SDKs in Python, JavaScript, and Go.

How gPRC differs from a message broker

Pattern

A message broker was invented to enable many-to-many patterns, meaning multiple data producers to multiple consumers at once.
gRPC enables peer-to-peer type of communication or 1:1.

Asynchronous vs request-reply

This one depends on the use case. In case your use case requires a type of communication that requires an immediate response/reply between services, gRPC would be the best fit, but if the communication can take place asynchronously, then a message broker would be the right choice as it decouples the client and server from one another and will not block the client’s operations until a server response is achieved.

Disk storage and Partitioning

A message broker serves as a persistent log data structure store; therefore, it can work on either the main memory or disk storage. This allows messages to be recovered if there is a server outage. A message broker stores data in distributed units (In memphis.dev, it’s called “stations”), and these units are partitioned across multiple brokers. This distributed storage increases the fault tolerance of the system.

gRPC, on the other hand, works with memory because it operates on the client side. This also means that gRPC calls are not persisted to disk. Ideally, gRPC can be combined with a message broker to increase the overall performance of a distributed system.

Stream buffering

gRPC being a messaging protocol, cannot buffer streaming data. This is unlike message brokers that can store millions of streamed messages as they are produced. An example scenario is streaming temperature data from a thermometer in a factory. If real-time processing is required, there has to be some server processing the data as it comes. This streaming process can overwhelm the server, so there needs to be a way to process streams in real time without overwhelming the server. A message broker is an ideal tool to handle this situation. This message broker can be the more traditional Apache Kafka or, the more modern Memphis.dev. Both Kafka and Memphis can be referred to as streaming databases.

Conclusion

While both gRPC and message brokers are used for inter-service communication. gRPC is more suited for inter-service calls, while message brokers are more suited for distributed systems architecture which can operate asynchronously.

In case you find your use case more suitable for a message broker, it can quickly be achieved using Memphis.dev through the open-source version or Memphis.dev cloud.

Are you struggling with schemas? Schemaverse is a revolutionary schema management and enforcement for data streaming. Watch a demo