Is it possible to deploy a Lenses S3 Sink Connector that uses a Regex pattern to configure a connector to consume from any number of topics that match the pattern?
I want to deploy one configuration that will back up any new topics created that fit the naming convention/pattern, without making a change to my connector configuration.
Yes this is possible. Please see the example S3 sink connector config below that fulfills this requirement:
connect.s3.kcql=insert into `backblaze1:version2` select * from `*` STOREAS `json` WITH_FLUSH_COUNT = 50
This example configuration will consume from any topic that contains the string “backblaze”.
Do note that the connector task will need to be restarted in order for the connector to pick up on any newly created topics.