Memphis{dev} is an open-source real-time data processing platform that provides end-to-end support for in-app streaming use cases using Memphis distributed message broker.
“It took me three minutes to do in Memphis what took me a week and a half in Kafka״
Memphis{dev} enables building next-generation applications that require large volumes of streamed and enriched data, modern protocols, zero ops, rapid development, extreme cost reduction, and a significantly lower amount of dev time for data-oriented developers and data engineers.
Four critical pillars –
Memphis key features (current and future) –
In a federated data platform, which responsibilities are distributed between stakeholders, teams, and sources, it’s harder to control establish a single standard. This is where the data contracts concept comes into play. Why do data contracts matter? Because (a) they provide insights into who owns what data products. (b) they support setting standards and managing your data pipelines with confidence. They also provide crucial information on what data is being consumed, by whom, and for what purpose. Bottom line: data contracts are essential for robust data management!
The very basic building block to control and ensure the quality of data that flows through your organization between the different owners is by defining well-written schemas and data models.
Why?
Formats
A quick overview of the most popular formats
Serialization process
Serialization converts a data object—a combination of code and data—into a series of bytes that saves the object’s state in an easily transmittable form.
The opposite process is called deserialization.
Serialization is basically represented as a function, each data format with its own implementation, and in case the structure and content of a given data do not match the defined schema in the .proto/.avro/.json struct, the serialization process fails, and therefore, the message will not be sent to the broker.
That process establishes the message’s initial validation before it reaches the broker itself, using the client cache to store the schema locally.
Memphis Schemaverse provides a robust schema store and schema management layer on top of memphis broker without a standalone compute or dedicated resources. With a unique and modern UI and programmatic approach, technical and non-technical users can create and define different schemas, attach the schema to multiple stations and choose if the schema should be enforced or not.
Memphis’ low-code approach removes the serialization part as it is embedded within the producer library.
Schemaverse supports versioning, GitOps methodologies, and schema evolution.
Also includes –
Step 1: Create a new schema (Currently only available through Memphis GUI)
Step 2: Attach Schema
Head to your station, and on the top-left corner, click on “+ Attach schema”
Step 3: Code example in node.js
Memphis abstracts the need for external serialization functions and embeds them within the SDK.
Producer (Protobuf example)
Consumer (Requires .proto file to decode messages)