We want Kafka Connect GCS connector to stop processing when GCS is down and resume from the same offset when GCS becomes available again, how can we set achieve that?
Hi,
The GCS connector has a retry mechanism inside the connector plugin. This will control how many times the connector itself tries to upload files. If exhausted the connector will error and stop.
connect.gcpstorage.retry.interval - Specifies the interval (in milliseconds) between retry attempts by the connector.
connect.gcpstorage.http.max.retries - Sets the maximum number of retries for the underlying HTTP client when interacting with GCP.
connect.gcpstorage.http.retry.interval - Specifies the retry interval (in milliseconds) for the underlying HTTP client.
connect.gcpstorage.http.retry.timeout.multiplier - Specifies the change in delay before the next retry or poll
In Addition, the Connector support error policies, which trigger a redeliver of messages to the plugin if a Retryable exception is thrown. Using Error Policies | Lenses Docs
And the Kafka Connect framework now has builtin retry policies. Kafka Connect Deep Dive – Error Handling and Dead Letter Queues | Confluent
So you can tweak the connector configuration for retrying for transistent network errors, set a retry policy on the whole connector (e.g. THROW and stop on error, RETRY or NO OP) and set error handling provided by the framework.
Regards,
Andrew