A Kafka broker receives messages from producers and stores them on a disc with a unique offset.
A Kafka broker enables consumers to retrieve messages by topic, partition, and offset.
If a Kafka broker goes offline for an extended time, it will likely return online but be far behind in logs.
Therefore, it will need to catch up on logs from the remaining brokers before being reintroduced into the In-Sync Replica (ISR).
During normal operation, replicas communicate with one another.
However, if one broker is significantly behind, we adjust settings to enable Kafka to spend more resources on keeping replicas up to date.
Because all partition replicas on the broker become unavailable during a broker outage, the availability of the affected partitions is determined by the existence and status of their other replicas.
If a partition has no additional replicas, it becomes inaccessible.
The settings on which they depend are:
- Min.isync.replicas
- Default.replication.factor
- Unclean.leader.election.enable
When the broker goes down, however, Kafka does not create a new replica.
You will have to wait for the broker to become available, but if you need a replica, you can use the reassignment tool kafka-reassign-partitions.sh to move partitions to available brokers.
In addition, when you restart the broker, it will attempt to reconnect.
Once completed, whether it remains a follower or becomes the leader is determined by whether it is the appropriate replica.
Reference: https://jaceklaskowski.gitbooks.io/apache-kafka/content/kafka-brokers.html
A Memphis message broker is a software that uses an architectural pattern to transport, transform, and route messages.
It mediates application communication by reducing the mutual awareness that applications must have of each other to exchange messages, effectively implementing decoupling.
The message broker’s primary function is to decouple services by handling communication between them.
Memphis tries to handle this situation better because of its asynchronized task management.
In addition, since there are multiple recipients for a single message, systems can receive the message from any recipient if one of the clients’ message brokers goes down.
Reference: https://docs.memphis.dev/memphis/memphis/concepts/message-broker