What is the difference between SQS vs RabbitMQ vs Memphis?
Avatar Idan Asulin
Apr 16, 2023
14 15 16
What is the difference between RabbitMQ vs ZeroMQ vs Memphis?
Avatar Idan Asulin
Apr 16, 2023
10 11 12
What is the difference between Kafka vs ZeroMQ vs Memphis?
Avatar Yaniv Ben Hemo
Apr 16, 2023
2 3 4
What is the difference between Azure Event Hub vs Kafka vs Memphis?
Avatar Idan Asulin
Apr 16, 2023
5 6 7
How does ActiveMQ vs RabbitMQ vs Kafka vs Memphis work?
Avatar Idan Asulin
Apr 16, 2023
7 8 9
What is ActiveMQ vs IBM MQ vs Memphis?
Avatar Yaniv Ben Hemo
Apr 16, 2023
3 4 5
What is IBM MQ?
Avatar Yaniv Ben Hemo
Apr 16, 2023
4 5 6
What is an mq server?
Avatar Idan Asulin
Apr 16, 2023
3 4 5
How does mq messaging take place?
Avatar Idan Asulin
Apr 16, 2023
2 3 4
Kafka data loss
2 3 4
Memphis Approach

We can recover lost events if the data source lives in the stack’s persistence layer.
In terms of Ingestion latency, increasing CPU resources and fine-tuning batch sizes result in a significant increase in throughput, reducing total lag from several hours to a few minutes.

Most importantly, we can make changes in Kafka to solve this problem too, which are as follows:

  1. Changing Kafka offset reset policy in several applications to the earliest.
    For example, suppose an ingestion lag is observed again due to spikes in data traffic.
    In that case, we attempt to load the oldest available record in Kafka to reduce the blast radius associated with data loss.
    It is especially useful in services where auto commits are disabled, and idempotency is considered.
  2. Extending Kafka retention policy by several days to allow for system re-stabilization and to give development teams enough time to work on potential fixes to issues without losing data.
    Where, of course, the retention policy can be reduced or reverted to its default or custom setting.
Memphis Approach

Each station in Memphis uses a stream file to store the station’s messages.
The user can choose whether the streaming file is saved in memory or on a file from where the user can retrieve it.