I’m trying to perform backup and restore on topics using the Lenses sink and source connectors for S3, but I’m encountering errors like:
org.apache.kafka.common.errors.RecordTooLargeException and java.lang.OutOfMemoryError: Java heap space.
What would be the correct configuration to perform the backup and restore of a topic?
In Lenses 5.3, you don’t need to worry about manually deploying sinks or sources for backing up a topic and dealing with these errors. There’s a built-in feature to deploy the S3 connector for backup/restore use case with the optimized configuration.
You need to ensure your Kafka Connect nodes are running with an IAM user with access to your target S3 bucket. Then go to the explorer menu, select your topic, click on the Explore menu, and choose whether you want to back up or restore the topic. For more details, you can refer to the documentation through this link:
Using Kafka Connect and AWS S3 for backing up and restoring Kafka topics | Lenses.io Documentation.
There are also some blog posts on the Lenses website about backups. You can check them out here:
https://lenses.io/blog/2023/09/lenses-5.3-kafka-topic-back-up-kafka-s3-connector/
and here: https://lenses.io/blog/2023/09/backup-restore-kafka-topics-s3/.