I’m sourcing XML data from an S3 bucket using the Lenses S3 Source Connector with the configuration:
connector.class=io.lenses.streamreactor.connect.aws.s3.source.S3SourceConnector
connect.s3.kcql=insert into `radar` select * from `shippositions:radar` STOREAS `Text` PROPERTIES('read.text.mode'='StartEndTag', 'read.text.start.tag'='<smgcsdata:SMGCSFlightObject>', 'read.text.end.tag'='</smgcsdata:SMGCSFlightObject>','read.text.trim'='true')
connect.s3.aws.auth.mode=Credentials
name=test-radar
connect.s3.aws.access.key=xxxx
connect.s3.aws.region=eu-west-3
connect.s3.aws.secret.key=xxxxx
But when I explore them in Lenses, events look corrupted:
Converter defaults to AVRO so adds data at the beginning of your message. You need to set the key & value converters to StringConverter such as:
connector.class=io.lenses.streamreactor.connect.aws.s3.source.S3SourceConnector
connect.s3.kcql=insert into `radar` select * from `shippositions:radar` STOREAS `Text` PROPERTIES('read.text.mode'='StartEndTag', 'read.text.start.tag'='<smgcsdata:SMGCSFlightObject>', 'read.text.end.tag'='</smgcsdata:SMGCSFlightObject>','read.text.trim'='true')
connect.s3.aws.auth.mode=Credentials
name=test-radar
connect.s3.aws.access.key=xxxx
connect.s3.aws.region=eu-west-3
connect.s3.aws.secret.key=xxxx
value.converter=org.apache.kafka.connect.storage.StringConverter
key.converter=org.apache.kafka.connect.storage.StringConverter