Lenses S3 Sink Connector: Regex to restore multiple Kafka topics in one configuration

Apache Kafka Connectors
,
1

Following a discussion here: Slack

I’d like to have the possibility of restoring topics in s3 to topics named the same (or possibly other) using regex.

For example it’s possible to backup topics using

"INSERT INTO my-bucket:topics SELECT * FROM `*` STOREAS `JSON` PROPERTIES ( 'store.envelope'=true, 'flush.size'=1000000, 'flush.interval'=30, 'flush.count'=5000)"

however reversing the query isn’t currently supported.

For my specific use case, restore currently requires some manual work to create the queries for each topic I want to back up, and whilst there are some architecture changes that could be made to simplify this, a regex restore would make things much quicker.

2

Hi @dplater

Thanks for the question! This can be supported by having multiple KCQL statements separated by a ;. Currently there’s no way to wildcard the target topic. This could be something we implement in the future as it sounds like it would be useful!