Not correctly deserialising XML data when sourced from S3 Source Connector

Apache Kafka Connectors
1

I’m trying to process this sort of data in an S3 bucket with the Lenses S3 Source connector

<?xml version="1.0" encoding="UTF-8"?>
<smgcsdata:Envelope xmlns:smgcsdata="http://xx.esb.xxxxx.com" xmlns:env="http://envelope.esb.xxxxx.com"
                    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                    xsi:schemaLocation="http://smgcs.esb.xxxxxxx.com SMGCSData_2.xsd">
   <smgcsdata:Header>
      <env:MessageSentDateTime>2012-10-12T14:59:35Z</env:MessageSentDateTime>
      <env:MessageSequenceNumber>4153829</env:MessageSequenceNumber>
      <env:MessageType>SMGCS</env:MessageType>
      <env:SourceSystemID>SMGCS</env:SourceSystemID>
      <env:DestinationSystemID>SAM</env:DestinationSystemID>
   </smgcsdata:Header>
   <smgcsdata:Body>
      <smgcsdata:SMGCSFlightObject>
         <smgcsdata:AircraftMovement>
            <smgcsdata:TimeOfDay>14:59:35</smgcsdata:TimeOfDay>
            <smgcsdata:Position>
               <smgcsdata:XSGN>+</smgcsdata:XSGN>
               <smgcsdata:XPOS>44547</smgcsdata:XPOS>
               <smgcsdata:YSGN>-</smgcsdata:YSGN>
               <smgcsdata:YPOS>6484</smgcsdata:YPOS>
            </smgcsdata:Position>
            <smgcsdata:Velocity>
               <smgcsdata:XSGN>-</smgcsdata:XSGN>
               <smgcsdata:XVEL>18</smgcsdata:XVEL>
               <smgcsdata:YSGN>-</smgcsdata:YSGN>
               <smgcsdata:YVEL>31</smgcsdata:YVEL>
            </smgcsdata:Velocity>
            <smgcsdata:TrackInformation>
               <smgcsdata:TrackSizeClass>Default</smgcsdata:TrackSizeClass>
               <smgcsdata:Target>Aircraft</smgcsdata:Target>
               <smgcsdata:VerticalMovementIndicator>Default</smgcsdata:VerticalMovementIndicator>
            </smgcsdata:TrackInformation>
            <smgcsdata:ModeC-Code>
               <smgcsdata:CodeValidated>Code validated</smgcsdata:CodeValidated>
               <smgcsdata:GarbledCode>Default</smgcsdata:GarbledCode>
               <smgcsdata:Sign>+</smgcsdata:Sign>
               <smgcsdata:Altitude>0</smgcsdata:Altitude>
            </smgcsdata:ModeC-Code>
         </smgcsdata:AircraftMovement>
         <smgcsdata:AircraftMovement>
            <smgcsdata:TimeOfDay>14:59:35</smgcsdata:TimeOfDay>
            <smgcsdata:Position>
               <smgcsdata:XSGN>-</smgcsdata:XSGN>
               <smgcsdata:XPOS>1080</smgcsdata:XPOS>
               <smgcsdata:YSGN>-</smgcsdata:YSGN>
               <smgcsdata:YPOS>72</smgcsdata:YPOS>
            </smgcsdata:Position>
            <smgcsdata:Velocity>
               <smgcsdata:XSGN>+</smgcsdata:XSGN>
               <smgcsdata:XVEL>0</smgcsdata:XVEL>
               <smgcsdata:YSGN>+</smgcsdata:YSGN>
               <smgcsdata:YVEL>0</smgcsdata:YVEL>
            </smgcsdata:Velocity>
            <smgcsdata:TrackInformation>
               <smgcsdata:TrackSizeClass>Default</smgcsdata:TrackSizeClass>
               <smgcsdata:Target>Default</smgcsdata:Target>
               <smgcsdata:VerticalMovementIndicator>Default</smgcsdata:VerticalMovementIndicator>
            </smgcsdata:TrackInformation>
            <smgcsdata:ModeC-Code>
               <smgcsdata:CodeValidated>Code validated</smgcsdata:CodeValidated>
               <smgcsdata:GarbledCode>Default</smgcsdata:GarbledCode>
               <smgcsdata:Sign>+</smgcsdata:Sign>
               <smgcsdata:Altitude>0</smgcsdata:Altitude>
            </smgcsdata:ModeC-Code>
         </smgcsdata:AircraftMovement>
         <smgcsdata:AircraftMovement>
            <smgcsdata:TimeOfDay>14:59:35</smgcsdata:TimeOfDay>
            <smgcsdata:Position>
               <smgcsdata:XSGN>-</smgcsdata:XSGN>
               <smgcsdata:XPOS>938</smgcsdata:XPOS>
               <smgcsdata:YSGN>-</smgcsdata:YSGN>
               <smgcsdata:YPOS>254</smgcsdata:YPOS>
            </smgcsdata:Position>
            <smgcsdata:Velocity>
               <smgcsdata:XSGN>+</smgcsdata:XSGN>
               <smgcsdata:XVEL>0</smgcsdata:XVEL>
               <smgcsdata:YSGN>+</smgcsdata:YSGN>
               <smgcsdata:YVEL>0</smgcsdata:YVEL>
            </smgcsdata:Velocity>
            <smgcsdata:TrackInformation>
               <smgcsdata:TrackSizeClass>Default</smgcsdata:TrackSizeClass>
               <smgcsdata:Target>Default</smgcsdata:Target>
               <smgcsdata:VerticalMovementIndicator>Default</smgcsdata:VerticalMovementIndicator>
            </smgcsdata:TrackInformation>
            <smgcsdata:ModeC-Code>
               <smgcsdata:CodeValidated>Code validated</smgcsdata:CodeValidated>
               <smgcsdata:GarbledCode>Default</smgcsdata:GarbledCode>
               <smgcsdata:Sign>+</smgcsdata:Sign>
               <smgcsdata:Altitude>0</smgcsdata:Altitude>
            </smgcsdata:ModeC-Code>
         </smgcsdata:AircraftMovement>
      </smgcsdata:SMGCSFlightObject>
   </smgcsdata:Body>
</smgcsdata:Envelope>

I’m using the configuration:

connector.class=io.lenses.streamreactor.connect.aws.s3.source.S3SourceConnector
connect.s3.kcql=insert into `radar` select * from `shippositions:radar` STOREAS `Text` PROPERTIES('read.text.mode'='StartEndTag', 'read.text.start.tag'='<smgcsdata:SMGCSFlightObject>', 'read.text.end.tag'='</smgcsdata:SMGCSFlightObject>','read.text.trim'='true')
connect.s3.aws.auth.mode=Credentials
name=test-radar
connect.s3.aws.access.key=xxxxxx
connect.s3.aws.region=eu-west-3
connect.s3.aws.secret.key=xxxxxx
value.converter=org.apache.kafka.connect.storage.StringConverter
key.converter=org.apache.kafka.connect.storage.StringConverter

I see the data as STRING in Lenses SQL Studio:

image
image1333×566 93.5 KB

But when I convert the Topic serialisation to AVRO/AVRO, no events are rendered

2

Because of the namespace smgcsdata the XML cannot be parsed correctly. This might need to be removed by an SMT or alternatively you could use a Lenses SQL Processor to remove it.