Corrupted XML messages from sourcing XML data from S3 Bucket with S3 Connector

Apache Kafka Connectors
1

I’m sourcing XML data from an S3 bucket using the Lenses S3 Source Connector with the configuration:

connector.class=io.lenses.streamreactor.connect.aws.s3.source.S3SourceConnector
connect.s3.kcql=insert into `radar` select * from `shippositions:radar` STOREAS `Text` PROPERTIES('read.text.mode'='StartEndTag', 'read.text.start.tag'='<smgcsdata:SMGCSFlightObject>', 'read.text.end.tag'='</smgcsdata:SMGCSFlightObject>','read.text.trim'='true')
connect.s3.aws.auth.mode=Credentials
name=test-radar
connect.s3.aws.access.key=xxxx
connect.s3.aws.region=eu-west-3
connect.s3.aws.secret.key=xxxxx

But when I explore them in Lenses, events look corrupted:

image
image1674×853 214 KB

2

Converter defaults to AVRO so adds data at the beginning of your message. You need to set the key & value converters to StringConverter such as:

connector.class=io.lenses.streamreactor.connect.aws.s3.source.S3SourceConnector
connect.s3.kcql=insert into `radar` select * from `shippositions:radar` STOREAS `Text` PROPERTIES('read.text.mode'='StartEndTag', 'read.text.start.tag'='<smgcsdata:SMGCSFlightObject>', 'read.text.end.tag'='</smgcsdata:SMGCSFlightObject>','read.text.trim'='true')
connect.s3.aws.auth.mode=Credentials
name=test-radar
connect.s3.aws.access.key=xxxx
connect.s3.aws.region=eu-west-3
connect.s3.aws.secret.key=xxxx
value.converter=org.apache.kafka.connect.storage.StringConverter
key.converter=org.apache.kafka.connect.storage.StringConverter