JSON to parquet/AVRO sink using S3 Connector

Apache Kafka Connectors
1

I would like to know if it’s possible to sink JSON messages into parquet using the S3 sink?

2

Hello Edward_k,

Yes, you can indeed convert JSON messages to Parquet/AVRO, but you’ll need to add some parameters like store.envelope=true and JsonConverter:

connect.s3.kcql=INSERT INTO bucket:folder SELECT * FROM topic STOREAS `PARQUET` PROPERTIES('store.envelope'=true)
value.converter=org.apache.kafka.connect.json.JsonConverter

Check the documentation for more details on message conversion and storage format: Kafka to AWS S3 open source connector | Lenses.io Documentation

Cheers!