The KafkaConnector component is used to send messages to the Kafka Broker and retrieve messages from it. The KafkaConnector component can work either in synchronous or asynchronous mode to send the messages to the Kafka Broker.

Configuration

Component Configuration panel in the CPS (Configuration Property Sheet) allows customization in the component.

Figure 1: KafkaConnector CPS

Attributes

Connector Mode

The way in which the Kafka Connector acts; either as a Producer or a Consumer.

SEND

Selecting this mode makes the connector act as a producer, to send messages as Topics to the Kafka Broker.

RECEIVE

Selecting this mode makes the connector act as a consumer, to consume messages from the Kafka Broker.

Send Mode

This mode is enabled only when the Connector Mode is set to "Send.".

SYNCHRONOUS

When this mode is selected, the Kafka Connector acting as a producer sends requests and waits for the server reply. In this case, errors arising, if any, will be logged in the component and sent to the Exception port as per the configuration set in the Error Handling Configuration panel.

ASYNCHRONOUS

When this mode is selected, the Kafka Connector acting as a producer sends requests and does not wait for the server reply. The callback interface is used to log error messages in error logs and not sent to the Exception port.

Connection URL

The connection URL of the Kafka Connector which is being used to send or receive the messages.

Poll Interval (in ms)

The maximum interval of time in milliseconds that the Kafka Consumer's poll method takes to return from the call to Kafka Broker. It returns immediately when unattended messages are available; if not available, it returns null on timeout.

Empty Fetch Threshold

The maximum number of times call for records from Kafka Consumer's poll method to Kafka Broker can return empty. On breaching this limit, the consumer closes.

Topic Name

The name of the topic through which the messages are sent by the producer or through which the messages are consumed by the consumer.

Additional Properties

The default properties in the additional properties of the Kafka consumer (Connector Mode - Receive) can be changed as per requirement.

Figure 2: Additional Properties dialog box for Kafka consumer (Connector Mode - Receive) with prepopulated data

Please refer to the link below to add additional properties than the default ones to the Kafka Consumer:
https://kafka.apache.org/documentation/#newconsumerconfigs

The default properties in the additional properties of the Kafka consumer (Connector Mode - Send) can be changed as per requirement.

Figure 3: Additional Properties dialog box for Kafka consumer (Connector Mode - Send) with prepopulated data

Please refer to the link below to add additional properties than the default ones to the Kafka Producer:
https://kafka.apache.org/documentation/#producerconfigs

Expert Properties

Enable the Expert Properties view to configure these properties.

Figure 4: KafkaConnector CPS with the Expert Properties enabled and highlighted

Error handling configuration

Please refer the Error Handling section in Common Configurations page for documentation.

Pre Processing XSL Configuration

Pre Processing XSL configuration can be used to transform request message before processing it. Click the ellipses button against the property to configure the properties.

Refer to the Pre/Post Processing XSL Configuration section under the Common Configurations page for details regarding Pre Processing XSL configuration and Post Processing XSL configuration (below).

Post Processing XSL Configuration 

Post Processing XSL configuration can be used to transform the response message before sending it to the output port.

Process Message Based on Property

The property helps components to skip certain messages from processing.

Refer to the Process Message Based On a Property section under the Common Configurations page.

Threadpool Configuration

 This property is used when there is a need to process messages in parallel within the component, still maintaining the sequence from the external perspective. 

Refer to the Threadpool Configuration section under the Common Configurations page.

Functional Demonstration

Scenario

Configure two KafkaConnector microservices as producer and consumer following descriptions in the Component description and send a message. Connect a Feeder microservice to KafkaProducer and connect the KafkaProducer output and the KafkaConsumer input.

Connect Display microservices to the Kafka microservice output ports to view the respective outputs.

Figure 5: Event process to demonstrate the functions of KafkaProducer and KafkaConsumer

In this example, the input injected using Feeder is sent via the KafkaProducer, which is in turn received by the KafkaConsumer; the message will be displayed on Consumer_Display.

Input

Provide the following input in the Feder text field:

Figure 6: Feeder input sent to the KafkaProducer

Output

KafkaProducer Display window displays the acknowledgment message.

Figure 7: Output received on the KafkaProducer display

KafkaConsumer Display window displays the message recived from the KafkaProducer.

Figure 8: Output received on the KafkaConsumer display