The KafkaConnector component is used to send messages to the Kafka Broker and retrieve messages from it. The KafkaConnector component can work either in synchronous or asynchronous mode to send the messages to the Kafka Broker.

Configuration

Component Configuration panel in the CPS (Configuration Property Sheet) allows customization in the component.

Figure 1: KafkaConnector CPS

Attributes

Connector Mode

The way in which the Kafka Connector acts; either as a Producer or a Consumer.

SEND

Selecting this mode makes the connector act as a producer, to send messages as Topics to the Kafka Broker.

RECEIVE

Selecting this mode makes the connector act as a consumer, to consume messages from the Kafka Broker.

Listen Mode

This mode is enabled only when the Connector Mode is set to receive. This gives the way in which the connector consumes message from the Kafka Broker.

SYNCHRONOUS

When this mode is selected the Kafka Connector acting as a  consumer explicitly requests that a message be delivered and then consumes it.

ASYNCHRONOUS

When this mode is selected the Kafka Connector acting as a consumer automatically consumes the message without waiting for any reply.

Connection URL

The connection URL of the Kafka Connector which is being used to send or receive the messages.

Poll Interval (in ms)

The time interval in milliseconds until which the Kafka Connector acting as the consumer fetches messages from the Kafka Broker.

Topic Name

The name of the topic through which the messages are sent by the producer or through which the messages are consumed by the consumer.

Additional Properties

The default properties in the additional properties of the Kafka consumer (Connector Mode - Receive) can be changed as per requirement.

Figure 2: Additional Properties dialog box for Kafka consumer (Connector Mode - Receive) with prepopulated data

Please refer to the link below to add additional properties than the default ones to the Kafka Consumer:
https://kafka.apache.org/documentation/#newconsumerconfigs

The default properties in the additional properties of the Kafka consumer (Connector Mode - Send) can be changed as per requirement.

Figure 3: Additional Properties dialog box for Kafka consumer (Connector Mode - Send) with prepopulated data

Please refer to the link below to add additional properties than the default ones to the Kafka Producer:
https://kafka.apache.org/documentation/#producerconfigs

Expert Properties

Enable the Expert Properties view to configure these properties.

Figure 4: KafkaConnector CPS with the Expert Properties enabled and highlighted

Error handling configuration

Please refer the Error Handling section in Common Configurations page for documentation.

Pre Processing XSL Configuration

Pre Processing XSL configuration can be used to transform request message before processing it. Click the ellipses button against the property to configure the properties.

Refer to the Pre/Post Processing XSL Configuration section under the Common Configurations page for details regarding Pre Processing XSL configuration and Post Processing XSL configuration (below).

Post Processing XSL Configuration 

Post Processing XSL configuration can be used to transform the response message before sending it to the output port.

Process Message Based on Property

The property helps components to skip certain messages from processing.

Refer to the Process Message Based On a Property section under the Common Configurations page.

Threadpool Configuration

 This property is used when there is a need to process messages in parallel within the component, still maintaining the sequence from the external perspective. 

Refer to the Threadpool Configuration section under the Common Configurations page.

Functional Demonstration

Scenario

Configure two KafkaConnector microservices as producer and consumer following descriptions in the Component description and send a message. Connect a Feeder microservice to KafkaProducer and connect the KafkaProducer output and the KafkaConsumer input.

Connect Display microservices to the Kafka microservice output ports to view the respective outputs.

Figure 5: Event process to demonstrate the functions of KafkaProducer and KafkaConsumer

In this example, the input injected using Feeder is sent via the KafkaProducer, which is in turn received by the KafkaConsumer; the message will be displayed on Consumer_Display.

Input

Provide the following input in the Feder text field:

Figure 6: Feeder input sent to the KafkaProducer

Output

KafkaProducer Display window displays the acknowledgment message.

Figure 7: Output received on the KafkaProducer display

KafkaConsumer Display window displays the message recived from the KafkaProducer.

Figure 8: Output received on the KafkaConsumer display