Kafka sender
Kafka Sender
The Kafka Sender is used to send messages to a topic in a Kafka cluster. It uses an incoming ConnectMessage to create a Kafka message using the payload of the selected message part. Optionally, you can define a Kafka message key and headers that are mapped from the ConnectMessage.
Each Kafka topic is chopped up in separate partitions which will contain part of the messages in the topic. The decision of which topic partition a message is sent to is decided by the Kafka Sender. To determine this the producer uses either an explicit topic partition identifier, the message key or a default implementation specific strategy (in this order of precedence).
Configuring a Kafka Sender requires a list of bootstrap servers. These are the addresses of Kafka brokers in the cluster which the producer will contact during startup.
We assume that the desired message payload, the message key value and any message header values are all text data. The Target Encoding can be used to specify the desired character encoding, e.g. UTF-8. It is important to keep in mind that Kafka stores all data as raw binary data. So if after sending a message using this producer there is logic that subsequently reads this data from the topic, it should have knowledge of the used character encoding in order to correctly interpret the data.
The Kafka Sender has an Acknowledgement setting which affects message delivery semantics. It currently has three settings, ONE, NONE and ALL. NONE means that the producer does not expect any kind of acknowledgement of the Kafka broker. ONE means that the producer expects only an acknowledgement of the Kafka broker that is the leader of the topic partition. ALL means that the producer expects an acknowledgement only after the sent message has been replicated across all topic partition replicas (not just the topic partition leader). Please note that the exact meaning of the ALL setting can be altered by the min.insync.replicas setting of the broker.
The Kafka Sender can be further customized by providing a custom properties file. These properties are available in the official Kafka docs, see https://kafka.apache.org/documentation/#producerconfigs. Note that custom properties should be used with care because they will overwrite other behavior.
In the table below, you will find an explanation of these properties. All attributes with a ‘*’ are mandatory.
Attribute | Description |
MessagePart In | Name of the MessagePart in the incoming ConnectMessage that is to be used by this component. |
Topic Specification Type | The manner in which to define the target topic of the outgoing Kafka message. |
Fixed Topic Value | Only available if Topic Specification Type is set to FIXED. The name of the target Kafka topic. |
Topic Header | Only available if Topic Specification Type is set to HEADER. Name of the ConnectMessage header which contains the name of the target Kafka topic. |
Topic Expression | Only available if Topic Specification Type is set to EXPRESSION. The SpEL expression determining the name of the target Kafka topic. |
Message Key Specification Type | The manner in which to define the message key in the outgoing Kafka message. |
Fixed Message Key Value | Only available if Message Key Specification Type is set to FIXED. Fixed message key value for all outgoing Kafka messages. |
Message Key Header | Only available if Message Key Specification Type is set to HEADER. Name of the ConnectMessage header which contains the Kafka message key value. |
Message Key Expression | Only available if Message Key Specification Type is set to EXPRESSION. The SpEL expression determining the Kafka message key value. |
Partition ID Specification Type | The manner in which to define the target Kafka topic partition ID. |
Fixed Partition ID | Only available if Partition ID Specification Type is set to FIXED. Fixed Kafka topic partition ID. |
Partition ID Header | Only available if Partition ID Specification Type is set to HEADER. Name of the ConnectMessage header which contains the Kafka topic partition ID. |
Partition ID Expression | Only available if Partition ID Specification Type is set to EXPRESSION. The SpEL expression determining the Kafka topic partition ID. |
Kafka Bootstrap Servers* | A comma-separated list of Kafka brokers. |
Target Encoding | The target encoding used for the key, value and header values in a Kafka message. |
Mapped Headers | A comma-separated list of headers that are mapped from the ConnectMessage to the Kafka message. |
Acknowledgement | The acknowledgement setting for the Kafka producer. |
Kafka Producer Properties Filename | The name of the Java properties file containing additional config for the Kafka Sender. |