Kafka sender

Kafka Sender

The Kafka Sender is used to send messages to a topic in a Kafka cluster. It uses an incoming ConnectMessage to create a Kafka message using the payload of the selected message part. Optionally, you can define a Kafka message key and headers that are mapped from the ConnectMessage.

Each Kafka topic is chopped up in separate partitions which will contain part of the messages in the topic. The decision of which topic partition a message is sent to is decided by the Kafka Sender. To determine this the producer uses either an explicit topic partition identifier, the message key or a default implementation specific strategy (in this order of precedence).

Configuring a Kafka Sender requires a list of bootstrap servers. These are the addresses of Kafka brokers in the cluster which the producer will contact during startup.

We assume that the desired message payload, the message key value and any message header values are all text data. The Target Encoding can be used to specify the desired character encoding, e.g. UTF-8. It is important to keep in mind that Kafka stores all data as raw binary data. So if after sending a message using this producer there is logic that subsequently reads this data from the topic, it should have knowledge of the used character encoding in order to correctly interpret the data.

The Kafka Sender has an Acknowledgement setting which affects message delivery semantics. It currently has three settings, ONE, NONE and ALL. NONE means that the producer does not expect any kind of acknowledgement of the Kafka broker. ONE means that the producer expects only an acknowledgement of the Kafka broker that is the leader of the topic partition. ALL means that the producer expects an acknowledgement only after the sent message has been replicated across all topic partition replicas (not just the topic partition leader). Please note that the exact meaning of the ALL setting can be altered by the min.insync.replicas setting of the broker.

The Kafka Sender can be further customized by providing a custom properties file. These properties are available in the official Kafka docs, see https://kafka.apache.org/documentation/#producerconfigs. Note that custom properties should be used with care because they will overwrite other behavior.

In the table below, you will find an explanation of these properties. All attributes with a ‘*’ are mandatory.