Memphis vs Apache Pulsar

Min Read

What is Memphis?

​​Memphis{dev} is the only open-source, low-code real-time data processing platform that provides a full ecosystem for in-app streaming use cases using memphis distributed message broker with a paradigm of produce-consume that supports modern in-app streaming pipelines and async communication by removing frictions of management, cost, resources, language barriers, and time for data-oriented developers and data engineers, unlike other message brokers and queues that requires a great amount of code, optimizations, adjustments, and mainly time.

On the whole, Memphis focuses on four main pillars -
Performance, Resiliency, Dev Experience, and Observability

  • Increase performance and scalability.
    Due to replacing naive algorithms with heuristics-based with higher success rates, and RAFT improvements to better suit Kubernetes runtimes.
  • Never lose a message. Build an ecosystem of notification and autonomous capabilities to mitigate poison/unacknowledged messages.
  • Remove noise and complex business logic from the developer/consumers side -
    • Easy-to-use, understand, and onboard management tools like UI, and CLI that provide end-to-end control over every aspect of the cluster for both the maintainers like DevOps and SRE teams, all the way to the developers and data engineers who develop around it.
    • Lightweight SDKs with background configuration to lower the amount of code and design decisions to the minimum. You still have full control over each parameter, but the out-of-the-box configuration with automatic optimizations does the heavy lifting for you.
  • Observability - lags or unacknowledged messages are hard to trace and mitigate! Usually, it requires building a dedicated wrapper, establishing a monitoring stack, crossing information between different consumers/databases/log sinks, and if the issue turned out to live at the application level, consequently restoring the message for validating the event is a pain.

What is Apache Pulsar?

Apache® Pulsar™ is a distributed, open-source pub-sub messaging and streaming platform for real-time workloads, managing hundreds of billions of events per day.

Pulsar is a multi-tenant, high-performance solution for server-to-server messaging. Originally developed by Yahoo, Pulsar is under the stewardship of the Apache Software Foundation.

Key features of Pulsar are listed below:

  • Native support for multiple clusters in a Pulsar instance, with seamless geo-replication of messages across clusters.
  • Very low publish and end-to-end latency.
  • Seamless scalability to over a million topics.
  • A simple client API with bindings for Java, Go, Python and C++.
  • Multiple subscription types (exclusive, shared, and failover) for topics.
  • Guaranteed message delivery with persistent message storage provided by Apache BookKeeper.
  • A serverless lightweight computing framework Pulsar Functions offers the capability for stream-native data processing.
  • A serverless connector framework Pulsar IO, which is built on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar.
  • Tiered Storage offloads data from hot/warm storage to cold/long-term storage (such as S3 and GCS) when the data is aging out.

Quick Comparison Table

Memphis Apache Pulsar
Cloud-Native True True
Quorom Raft Bookeeper / Zookeeper
Delivery guaranty At-least once, Exactly once At-least once, Exactly once
Dead-letter True True
Multi-tenant True True
Stream processing Serverless and complied apps Serverless functions
Schema management Using UI, and SDK SDK
Schema validation True True
Metadata store MongoDB etcd/Zookeeper/RocksDB
Connectors framework Modular, simple to implement True
Events routing True - v0.4.0 False
Distributed security True False
GUI True StreamNative


Memphis aims to provide cloud-native applications with all the needed in-app requirements for quality data streaming,
focusing on cost-efficiency, deep security for multi-tenant environments, faster data streaming use cases development,
collaboration within the organization between the different data teams, and outstanding management with zero ops.

Apache Pulsar is a mature messaging bus and a message broker with a vast community and users,
but in most cases, it would be used for its broker ability only as it is still highly oriented to java, requires configurations,
a learning curve, and maintenance.

Memphis and Pulsar can provide shared features, but it would probably be faster and easier with Memphis.


We will keep you updated

It's all about data engineering and dev practical materials

We will keep you updated

It's all about data engineering and dev practical materials