S3 source connector doesn't pull files from the bucket

Apache Kafka Connectors
1

I have an S3 source connector configured to consume files from an S3 bucket. I checked the Connect logs and didn’t see anything there. What should I do next?

2

Hello Edward_k,

A few things to check:

  • The logs of the Kafka Connect Cluster to see if any error messages appear.
  • Access permission to the bucket, so it’s necessary to check the IAM roles. Test with the AWS CLI client with the same role if necessary.
  • Ensure he bucket address used is correct in your KCQL.
  • That the Kafka Connect Cluster has the appropriate amount of resources for its correct operation.
  • If there are any Glacier or Deep Archive objects in the bucket. Depending on the version of the Connector you are using, these buckets may be skipped from processing.
  • Take a look at the Stream Reactor GitHub repository; any fixes and improvements will appear there: Releases · lensesio/stream-reactor · GitHub