Adding KCQL properties delays S3 sink

Hi Team!

After adding the properties PROPERTIES(‘store.envelope’=true,‘partition.include.keys’=false) WITHPARTITIONER=Values WITH_FLUSH_SIZE=5242880 WITH_FLUSH_INTERVAL=30 WITH_FLUSH_COUNT=500 to KCQL insert, I see the records are sinked to s3 every hour. Without adding the PROPERTIES to insert statement, data sinks as per WITH_FLUSH_INTERVAL. What could be the issue?


Hi shaik_ahmed,

Our documentation here provides the KCQL syntax:

INSERT INTO bucketAddress[:pathPrefix]
FROM kafka-topic
[PARTITIONBY (partition[, partition] ...)]
[STOREAS storage_format]
[WITH_FLUSH_SIZE = flush_size]
[WITH_FLUSH_INTERVAL = flush_interval]
[WITH_FLUSH_COUNT = flush_count]

At the moment the keywords order matter . In your case properties is set before the flushing options so the flushing options are ignored by the grammar tokenizer.


1 Like

Thank you for the documentation!