Memphis Approach
A Kafka producer publishes or writes data to different partitions’ topics.
Producers know which data should be written to which partition and broker.
However, the broker and partition are not required to be specified by the user.
Understanding the events under which a producer fails to produce messages for the cluster is necessary for handling errors, retrying the events with particular failures, and creating recovery mechanisms.
The possible scenarios where the producer might fail are as follows:
- Kafka cluster is unavailable.
- The “acks” setting on the Kafka producer is set to “all” and brokers are unavailable.
To solve these problems, we can temporarily store the message in a secondary store DB/Cache and retry the messages from the secondary store to write the message to the main topic.
A Memphis producer is a source application or service that sends data or messages to the station.
Several objects are created when a user configures a client connection to Memphis.
A connection is an open socket that connects the client to Memphis.
Only required once when the client/application is first launched.
A producer entity must be declared to write data/messages into Memphis.
The producer might fail when the cluster or station is not available.
Therefore, to resolve the producer failure, we can temporarily store the message in a secondary store DB/Cache and retry the messages from the secondary store to write the message to the main topic.