STAR US

Memphis vs Apache Pulsar

Min Read

What is Memphis?

​​Memphis{dev} is the only open-source, low-code real-time data processing platform that provides a full ecosystem for in-app streaming use cases using memphis distributed message broker with a paradigm of produce-consume that supports modern in-app streaming pipelines and async communication by removing frictions of management, cost, resources, language barriers, and time for data-oriented developers and data engineers, unlike other message brokers and queues that requires a great amount of code, optimizations, adjustments, and mainly time.

On the whole, Memphis focuses on four main pillars -
Performance, Resiliency, Dev Experience, and Observability

  • Increase performance and scalability.
    Due to replacing naive algorithms with heuristics-based with higher success rates, and RAFT improvements to better suit Kubernetes runtimes.
  • Never lose a message. Build an ecosystem of notification and autonomous capabilities to mitigate poison/unacknowledged messages.
  • Remove noise and complex business logic from the developer/consumers side -
    • Easy-to-use, understand, and onboard management tools like UI, and CLI that provide end-to-end control over every aspect of the cluster for both the maintainers like DevOps and SRE teams, all the way to the developers and data engineers who develop around it.
    • Lightweight SDKs with background configuration to lower the amount of code and design decisions to the minimum. You still have full control over each parameter, but the out-of-the-box configuration with automatic optimizations does the heavy lifting for you.
  • Observability - lags or unacknowledged messages are hard to trace and mitigate! Usually, it requires building a dedicated wrapper, establishing a monitoring stack, crossing information between different consumers/databases/log sinks, and if the issue turned out to live at the application level, consequently restoring the message for validating the event is a pain.

What is Apache Pulsar?

Apache® Pulsar™ is a distributed, open-source pub-sub messaging and streaming platform for real-time workloads, managing hundreds of billions of events per day.

Pulsar is a multi-tenant, high-performance solution for server-to-server messaging. Originally developed by Yahoo, Pulsar is under the stewardship of the Apache Software Foundation.

Key features of Pulsar are listed below:

  • Native support for multiple clusters in a Pulsar instance, with seamless geo-replication of messages across clusters.
  • Very low publish and end-to-end latency.
  • Seamless scalability to over a million topics.
  • A simple client API with bindings for Java, Go, Python and C++.
  • Multiple subscription types (exclusive, shared, and failover) for topics.
  • Guaranteed message delivery with persistent message storage provided by Apache BookKeeper.
  • A serverless lightweight computing framework Pulsar Functions offers the capability for stream-native data processing.
  • A serverless connector framework Pulsar IO, which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar.
  • Tiered Storage offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out.

Quick Comparison Table

Memphis Apache Pulsar
Cloud-Native True True
Quorom Raft Bookeeper / Zookeeper
Delivery guaranty At-least once, Exactly once At-least once, Exactly once
Dead-letter True True
Multi-tenant True True
Stream processing Serverless and complied apps Serverless functions
Schema management Using UI, and SDK SDK
Schema validation True True
Metadata store MongoDB etcd/Zookeeper/RocksDB
Connectors framework Modular, simple to implement True
Events routing True - v0.4.0 False
Distributed security True False
GUI True StreamNative

Bottom-line

Memphis aims to provide cloud-native applications with all the needed in-app requirements for quality data streaming,
focusing on cost-efficiency, deep security for multi-tenant environments, faster data streaming use cases development,
collaboration within the organization between the different data teams, and outstanding management with zero ops.

Apache Pulsar is a mature messaging bus and a message broker with a vast community and users,
but in most cases, it would be used for its broker ability only as it is still highly oriented to java, requires configurations,
a learning curve, and maintenance.

Memphis and Pulsar can provide shared features, but it would probably be faster and easier with Memphis.

share:

We will keep you updated

It's all about data engineering and dev practical materials

We will keep you updated

It's all about data engineering and dev practical materials