STAR US

Here is why memphis
should be your next message broker

Contents

Introduction

Times are complicated. Unfortunately, teams are shrinking, while companies still need to push for growth, which means more work with fewer hands. Among the never-ending challenges, tasks, features, and bugs, there is one thing that is constantly growing and becoming more complicated over time - data.

Within data, multiple categories exist, like data streaming, batch processing, pipelines, data warehouse, and databases.

There is a layer that resides in the middle and connects the different components - The message broker.


When to use a message broker?

  • Netflix uses a message broker to ingest their users’ events and spread them across different systems and data stores for ML training, personalizations, analysis, BI, trigger actions, and many more.
  • Uber uses a message broker to update their maps and drivers' locations in real-time, as well as a queue of tasks and orders, BI, and event-driven architecture.
  • Robust task scheduling for stateless workers
  • Queueing
  • Mesh communication between microservices
  • Absorb massive amounts of data and connections in a persistent and protected way

"As the organization scales and in-app real-time ingestion and processing are required,
several challenges arise."

  1. How to ensure governance across multiple data producers?
  2. Monitor the movement of data between different producers and consumers
  3. Horizontal, infinite scale is required
  4. Onboard developers become complicated
  5. Create client wrappers to ensure a single standard across projects
  6. To get ingestion and processing, usually two or more systems are required, like Apache Kafka with Flink and druid
  7. Hard to find talent and a dedicated team to handle Kafka. Highly expensive.
  8. COST$$$

How Memphis solves all of those challenges?

Simplicity.
"What used to take three months in Kafka takes three minutes with Memphis. And with 90% fewer costs in the process."
Memphis has been designed to be deployed as production-ready in 3 minutes, with a ZeroOps mindset, and exceptional DevEx.

Governance.
Schemaverse is a Schema management with versioning, gitops, validation, enforcement, and zero trust. For Avro, protobuf, JSON, and GraphQL.

Monitoring.
Memphis is not a black box. Out-of-the-box monitoring with slack notifications and integration to all major monitoring tools. Each ingested message journey is traced.

Scale-up and out.
Memphis is cloud-native by design, written in Golang, and runs on any kubernetes.
By running over K8S, scaling is easy and infinite. No zookeeper/bookeeper. Memphis uses RAFT.

Onboard developers without client wrappers.
Memphis is designed to be simple and effective, with minimal additions by data engineering teams.
Memphis has been built with a modularity mindset. If more capabilities are needed, your team can add them directly to the system.

Data Ingestion and processing.
Usually, It will require the following flow -
Kafka (For ingestion) -> Flink (For processing) -> Nifi (For transformation)
While with Memphis, you get faster, more resilient,
and true real-time processing with a single tool straight to your app using the cloud’s serverless frameworks.

Cost.
Implementing and managing other message brokers and processing tools will require
a. Expensive, high-compute/memory servers
b. HUGE amount of dev hours
c. Recruit dedicated personnel with unique, hard-to-find talents
d. Amount of maintenance will push you to a managed solution which will cost XXX% more
e. You will get vendor locked
f. Memphis is open-source and 140% less expensive than Kafka and Pulsar, using our proprietary compute scheduler, better resource allocation, small footprint, and simplicity.

 

Installation

Related Articles

share:

We will keep you updated

It's all about data engineering and dev practical materials

We will keep you updated

It's all about data engineering and dev practical materials