S3 Source CSV to Avro

Apache Kafka Connectors
,
1

I have some data stored in S3 in CSV format and I’m looking to use the S3 source connector to ingest this data into Kafka.

The Kafka topic I’m looking to ingest into has an Avro schema stored in my Confluent schema registry and expects data in the Avro format.

The KCQL I’m using is:

      insert into my_topic select * from
      my_bucket:prefix STOREAS `CSV_WithHeaders`

I’m using the io.confluent.connect.avro.AvroConverter converter and have been looking at possible options SMTs to use to transform the CSV data to a format the schema expects as I’m currently seeing an Incompatible schema error.

Is there any guidance or examples on how to transform the CSV data to input into Kafka in Avro format?