S3 Source connector - How can I test partition Regex without running a connector?

Apache Kafka Connectors
1

I am trying to set up the S3 Source connector but am finding it difficult to craft a regular expression to extract the partitions.

I can only run it via the connector which is expensive in terms of time, it has a long feedback loop to find that my masterfully crafted regex has failed.

2

I would recommend if you want to test out the regex without the long feeback loop of running the connector you can do this by cloning the Stream Reactor source code and writing a simple test:

  "RegexPartitionExtractor" should "extract path from real path" in {
    val path = "bucket/prefix/topicname/partition=0/topicname+0+0000000000.json"
    val partitionExtractor = new RegexPartitionExtractor("(?i)^(?:.*)\\/partition=([0-9]*)\\/.*$")
    partitionExtractor.extract(path) should be(Some(0))
  }

You can then run this and get immediate feedback on whether your regex will match the files in your bucket.