Even though in the documentation, its says “In addition to the deadletter queues provided by Kafka Connect, the Stream Reactor sink connectors support Error Policies to handle failure scenarios.” I tried dead letter queues with influxdb sink connector. Anyone can guide me how to use dead letter queue along with influxdb sink connector.
Hi Omal,
It would help to get the details from you on what you tried.
The DLQ support is part of the Kafka Connect framework, not the connectors. It can be enabled by using these entries in the connector config:
errors.tolerance=all
errors.log.enable=true
errors.log.include.messages=true
errors.deadletterqueue.topic.name=example.deadletterqueue
errors.deadletterqueue.context.headers.enable=true
errors.deadletterqueue.topic.replication.factor=1
The errors tolerated by Connect are related to the following:
- key conversion
- value conversion
- transformation (single message transform)
- header conversion
Apart from the transformation, the other three are related to serialisation, i.e. invalid payload type.
These are the scenarios for which messages are routed to a DLQ. These operations happen outside of the connector plugin code; the Connect framework handles them.
If an error happens in the connector plugin, there’s no mechanism, as of today, to route them to the DLQ. And for these errors, as you shared in the screenshot, there is an option to:
- THROW - will set the connector in a failed state
- NOOP - ignores the exception
- RETRY - will attempt the operation
You might ask: why isn’t the connector plugin also supporting DLQ? To support it, the connector config would need to be aware of the Kafka connection itself, and at the moment, this can only happen via the configuration. It would be an anti-pattern to allow passing the Kafka connection in the Connector config.