Lenses ADLS Sink Connector - timestamp partitioning problem

Apache Kafka Connectors
1

Hello!

I am relatively new to Kafka Connect and Stream Reactor/Lenses and I’m trying to get the adls sink connector to work. I am trying to follow the documentation and the files are writing correctly, but the PARTITIONBY doesn’t appear to be working properly, instead of splitting the directory structure into yyyy/MM/dd/HH it is writing it like: source_ts_ms=1744042084001/file.json (source_ts_ms being a field in a Debezium messages’ payload - a datatype of int-64).

I think this is because it is falling back to the long format as it can’t convert it to a timestamp. I’ve tried using wallclock too but get something similar or something like event_timestamp=[missing].

Can anyone inform me on what I’m doing wrong? I can send the configuration if needed. Thanks! :slightly_smiling_face:

2

For anyone else having this issue, I solved it using the InsertRecordTimestampHeaders SMT located here: kafka-connect-smt/InsertRecordTimestampHeaders.md at main · lensesio/kafka-connect-smt.

“transforms.Ts.type”:“io.lenses.connect.smt.header.InsertRecordTimestampHeaders”,
“transforms.ts.timezone”: “Europe/London”,
“transforms.Ts.year.format”:“yyyy”,
“transforms.Ts.month.format”:“MM”,
“transforms.Ts.day.format”:“dd”,
“transforms.Ts.hour.format”:“HH”

PARTITIONBY _header.year, _header.month, _header.day in the KCQL

3

Thanks for sharing this @J-Harmson !

1 Like