Memphis Approach
To send large messages with Kafka, you’ll have to adjust a few properties.
On the consumer side, you’ll have to adjust fetch.message.max.bytes, which determines the largest message that a consumer can fetch and increase max.partition.fetch.bytes to receive bigger messages.
On the broker side, you need to change three properties. The first is the replica.fetch.max.bytes, which allows replicas in the brokers to send messages in the cluster and ensure that the messages are correctly replicated. If this property is set to a small value, the message won’t be replicated and since it won’t be committed, the consumer won’t be able to see it.
The second property you need to adjust is message.max.bytes, which is the largest message that a broker can receive from a producer. The third is the max.message.bytes property per topic, which is the largest message that can be appended to the topic. Keep in mind that message.max.bytes should be smaller or equal to the value set for the replica.fetch.max.bytes property.
Finally, on the producer end, you need to increase max.request.size to send larger messages.
You don’t need to do make any changes if you want to send large messages with Memphis. All you have to do is just produce the message and Memphis will take care of the rest.