What is the difference between SQS vs RabbitMQ vs Memphis?
Avatar Idan Asulin
Apr 16, 2023
14 15 16
What is the difference between RabbitMQ vs ZeroMQ vs Memphis?
Avatar Idan Asulin
Apr 16, 2023
10 11 12
What is the difference between Kafka vs ZeroMQ vs Memphis?
Avatar Yaniv Ben Hemo
Apr 16, 2023
2 3 4
What is the difference between Azure Event Hub vs Kafka vs Memphis?
Avatar Idan Asulin
Apr 16, 2023
5 6 7
How does ActiveMQ vs RabbitMQ vs Kafka vs Memphis work?
Avatar Idan Asulin
Apr 16, 2023
7 8 9
What is ActiveMQ vs IBM MQ vs Memphis?
Avatar Yaniv Ben Hemo
Apr 16, 2023
3 4 5
What is IBM MQ?
Avatar Yaniv Ben Hemo
Apr 16, 2023
4 5 6
What is an mq server?
Avatar Idan Asulin
Apr 16, 2023
3 4 5
How does mq messaging take place?
Avatar Idan Asulin
Apr 16, 2023
2 3 4
Kafka message consumed multiple times?
3 4 5
Memphis Approach

A consumer works to consume data from a specific topic.
Consumers work in a group where all the topics’ partitions are distributed among the group’s consumers.
As new members join the group and old members leave, the partitions are reassigned so that each member receives a proportionate share of the partitions.
It is referred to as rebalancing the group.
The Kafka message consumption error occurs because a timeout occurs before consumer can process and commit the message.
When your Kafka consumer commits, it essentially acknowledges receipt of the previous message, advancing the offset and thus proceeding to the next message.
However, if that timeout is exceeded, the consumer’s commit is ineffective because it occurs too late; the next time the consumer requests a message, it receives the same message.

There are multiple solutions to this problem.
Firstly, we can increase session.timeout.ms to 30000 to give the consumer more time to process the messages.
Secondly, we can reduce the max.poll.records=20 value.
So that the consumer has fewer messages to work on before the timeout occurs.
Lastly, we can allow enable.auto.commit, which is probably not the best solution because, as it says in the documentation of Kafka:

“If we allowed offsets to auto commit, as in the previous example, messages would be consumed after the consumer delivered them.
As a result, our process could fail after reading messages into our in-memory buffer before inserting them into the database.”

Reference: https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html

Memphis Approach

Memphis’ added layer of a consumer group allows for message ordering control.
In addition, it avoids duplicate processing of messages within the same type of consumers, as each message is delivered parallel to all consumer groups.
Data is not kept in message brokers indefinitely like in Kafka but kept for a set period based on conditions such as ingested time, size, and the number of messages within a station.