Adding KCQL properties delays S3 sink

Lenses
1

Hi Team!

After adding the properties PROPERTIES(‘store.envelope’=true,‘partition.include.keys’=false) WITHPARTITIONER=Values WITH_FLUSH_SIZE=5242880 WITH_FLUSH_INTERVAL=30 WITH_FLUSH_COUNT=500 to KCQL insert, I see the records are sinked to s3 every hour. Without adding the PROPERTIES to insert statement, data sinks as per WITH_FLUSH_INTERVAL. What could be the issue?

Thanks,
Samdan

2

Hi shaik_ahmed,

Our documentation here provides the KCQL syntax:

INSERT INTO bucketAddress[:pathPrefix]
SELECT *
FROM kafka-topic
[PARTITIONBY (partition[, partition] ...)]
[STOREAS storage_format]
[WITH_FLUSH_SIZE = flush_size]
[WITH_FLUSH_INTERVAL = flush_interval]
[WITH_FLUSH_COUNT = flush_count]
[PROPERTIES(
  'property.1'=x,
  'property.2'=x,
)]

At the moment the keywords order matter . In your case properties is set before the flushing options so the flushing options are ignored by the grammar tokenizer.

Stefan

1 Like
3

Thank you for the documentation!