KCQL Syntax Parse Failure - GCP Storage - Wildcards in Target

General
1

I am using the GCS Sink. I need some help writing a KCQL query to put data in GCS hourly buckets named after Kafka topics

I am getting the following exception:

java.lang.IllegalArgumentException: Invalid syntax.failed to parse at line 1 due to extraneous input '*' expecting SELECT at io.lenses.kcql.Kcql.parse(Kcql.java:706

I’m using this query:

connect.gcpstorage.kcql = INSERT INTO `your-bucket-name:stage-parquet/`*`` SELECT * FROM `*` PARTITIONBY _header.date STOREAS PARQUET PROPERTIES ('partition.include.keys'=false, 'flush.count'=50, 'flush.interval'=600)
2

The error message indicates that the wildcard in the target bucket isn’t correct KCQL syntax.

To dynamically use the topic name as a partition, include it in the PARTITIONBY clause. Please try the following query:

INSERT INTO `your-bucket-name:stage-parquet/` SELECT * FROM `*` PARTITIONBY _topic, _header.date STOREAS PARQUET PROPERTIES ('partition.include.keys'=false, 'flush.count'=50, 'flush.interval'=600)

This should resolve the syntax error and correctly partition your data by topic and date.