Retry connectors on failure

Apache Kafka Connectors
1

How can I configure a connector to retry on failure?

2

In addition to the dead letter queues provided by Kafka Connect, the Lenses Stream Reactor sink connectors support Error Policies to handle failure scenarios.

The sinks have three error policies that determine how failed writes to the target database are handled. These error policies allow you to control the behaviour of the sink if it encounters an error when writing records to the target system. Since Kafka retains the records, subject to the configured retention policy of the topic, the sink can ignore the error, fail the connector or attempt redelivery

Throw

Any error in writing to the target system will be propagated up and processing is stopped. This is the default behaviour.

Noop

Any error in writing to the target database is ignored and processing continues.

Keep in mind This can lead to missed errors if you don’t have adequate monitoring. Data is not lost as it’s still in Kafka subject to Kafka’s retention policy. The sink currently does not distinguish between integrity constraint violations and or other exceptions thrown by any drivers or target systems.

Retry

Any error in writing to the target system causes the RetryIterable exception to be thrown. This causes the Kafka Connect framework to pause and replay the message. Offsets are not committed. For example, if the table is offline it will cause a write failure, and the message can be replayed. With the Retry policy, the issue can be fixed without stopping the sink.

The documentation can be found here. Using Error Policies | Lenses Docs

Additionally, with the connector, you can configure a retry for some connectors if the underlying driver or SDK supports it, for example the Azure Service Bus connector, or AWS S3 and GCP. For example for GCP Storage | Lenses Docs

  1. connect.gcpstorage.error.policy
  2. connect.gcpstorage.retry.interval
  3. connect.gcpstorage.http.max.retries
  4. connect.gcpstorage.http.retry.interval
  5. connect.gcpstorage.http.retry.timeout.multiplier

Also you can configure the deadletter queues , this is provided by the Connect Framework not Lenses.