Introduction
Times are complicated. Unfortunately, teams are shrinking, while companies still need to push for growth, which means more work with fewer hands. Among the never-ending challenges, tasks, features, and bugs, there is one thing that is constantly growing and becoming more complicated over time - data.
Within data, multiple categories exist, like data streaming, batch processing, pipelines, data warehouse, and databases.
There is a layer that resides in the middle and connects the different components - The message broker.
When to use a message broker?
- Netflix uses a message broker to ingest their users’ events and spread them across different systems and data stores for ML training, personalizations, analysis, BI, trigger actions, and many more.
- Uber uses a message broker to update their maps and drivers' locations in real-time, as well as a queue of tasks and orders, BI, and event-driven architecture.
- Robust task scheduling for stateless workers
- Queueing
- Mesh communication between microservices
- Absorb massive amounts of data and connections in a persistent and protected way
"As the organization scales and in-app real-time ingestion and processing are required,
several challenges arise."
- How to ensure governance across multiple data producers?
- Monitor the movement of data between different producers and consumers
- Horizontal, infinite scale is required
- Onboard developers become complicated
- Create client wrappers to ensure a single standard across projects
- To get ingestion and processing, usually two or more systems are required, like Apache Kafka with Flink and druid
- Hard to find talent and a dedicated team to handle Kafka. Highly expensive.
- COST$$$
How Memphis solves all of those challenges?
Simplicity.
"What used to take three months in Kafka takes three minutes with Memphis. And with 90% fewer costs in the process."
Memphis has been designed to be deployed as production-ready in 3 minutes, with a ZeroOps mindset, and exceptional DevEx.
Governance.
Schemaverse is a Schema management with versioning, gitops, validation, enforcement, and zero trust. For Avro, protobuf, JSON, and GraphQL.
Monitoring.
Memphis is not a black box. Out-of-the-box monitoring with slack notifications and integration to all major monitoring tools. Each ingested message journey is traced.
Scale-up and out.
Memphis is cloud-native by design, written in Golang, and runs on any kubernetes.
By running over K8S, scaling is easy and infinite. No zookeeper/bookeeper. Memphis uses RAFT.
Onboard developers without client wrappers.
Memphis is designed to be simple and effective, with minimal additions by data engineering teams.
Memphis has been built with a modularity mindset. If more capabilities are needed, your team can add them directly to the system.
Data Ingestion and processing.
Usually, It will require the following flow -
Kafka (For ingestion) -> Flink (For processing) -> Nifi (For transformation)
While with Memphis, you get faster, more resilient,
and true real-time processing with a single tool straight to your app using the cloud’s serverless frameworks.
Cost.
Implementing and managing other message brokers and processing tools will require
a. Expensive, high-compute/memory servers
b. HUGE amount of dev hours
c. Recruit dedicated personnel with unique, hard-to-find talents
d. Amount of maintenance will push you to a managed solution which will cost XXX% more
e. You will get vendor locked
f. Memphis is open-source and 140% less expensive than Kafka and Pulsar, using our proprietary compute scheduler, better resource allocation, small footprint, and simplicity.
