A consumer works to consume data from a specific topic.
Consumers work in a group where all the topics’ partitions are distributed among the group’s consumers.
As new members join the group and old members leave, the partitions are reassigned so that each member receives a proportionate share of the partitions.
It is referred to as rebalancing the group.
The Kafka message consumption error occurs because a timeout occurs before consumer can process and commit the message.
When your Kafka consumer commits, it essentially acknowledges receipt of the previous message, advancing the offset and thus proceeding to the next message.
However, if that timeout is exceeded, the consumer’s commit is ineffective because it occurs too late; the next time the consumer requests a message, it receives the same message.
There are multiple solutions to this problem.
Firstly, we can increase session.timeout.ms to 30000 to give the consumer more time to process the messages.
Secondly, we can reduce the max.poll.records=20 value.
So that the consumer has fewer messages to work on before the timeout occurs.
Lastly, we can allow enable.auto.commit, which is probably not the best solution because, as it says in the documentation of Kafka:
“If we allowed offsets to auto commit, as in the previous example, messages would be consumed after the consumer delivered them.
As a result, our process could fail after reading messages into our in-memory buffer before inserting them into the database.”
Reference: https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
Memphis’ added layer of a consumer group allows for message ordering control.
In addition, it avoids duplicate processing of messages within the same type of consumers, as each message is delivered parallel to all consumer groups.
Data is not kept in message brokers indefinitely like in Kafka but kept for a set period based on conditions such as ingested time, size, and the number of messages within a station.