I have use case where a single processor monitors 2 two separate topics, combines the results into one single output topic. Both source topics having the same key field.
If both source topics receive data within a window, the processor combines the data from the two source topics into the output topic.
If either source topic receives data but the other does not within a window (no longer than 5 minutes), create an event in the output topic using the key, the data received and default values for data from the other source topic.
Any matching events incoming after the initial window are to be ignored for another period of time, say 24 hours based on the key value.
Neither source topic is the primary, they are both equal.
How would I go about creating an SQL Processor(s) to handle this scenario?
Hi wkolbenschlag,
I am afraid such a construct is impossible to express via Lenses SQL, I am not sure other SQL streaming records can express that.
For this very specific usecase, I would code using the Kafka streams API directly. You will have to manually handle the state and create the logic of ignoring any sequential events once a match happened. This is a departure from how frameworks like Kafka Streams, Flink behave from my experience.
Best wishes,
Stefan