S3 Sink Connector - Consume from/Back-up Multiple Topics using Regex

Apache Kafka Connectors
1

Hey there,

Is it possible to deploy a Lenses S3 Sink Connector that uses a Regex pattern to configure a connector to consume from any number of topics that match the pattern?
I want to deploy one configuration that will back up any new topics created that fit the naming convention/pattern, without making a change to my connector configuration.

Thanks.

2

Hey Alan,

Yes this is possible. Please see the example S3 sink connector config below that fulfills this requirement:


connector.class=io.lenses.streamreactor.connect.aws.s3.sink.S3SinkConnector
key.converter.schemas.enable=false
aws.secret.key=Abstracted
connect.s3.kcql=insert into `backblaze1:version2` select * from `*` STOREAS `json` WITH_FLUSH_COUNT = 50
aws.access.key=Abstracted
value.converter.schemas.enable=false
name=Backblaze-s3-sink
connect.s3.aws.region=us-east-1
value.converter=org.apache.kafka.connect.json.JsonConverter
topics.regex=.*backblaze.*
aws.auth.mode=Credentials
key.converter=org.apache.kafka.connect.json.JsonConverter

This example configuration will consume from any topic that contains the string “backblaze”.

Do note that the connector task will need to be restarted in order for the connector to pick up on any newly created topics.