Hello
I have been working with Kafka topics that contain deeply nested JSON structures, and while Lenses SQL has been great for querying; I am running into challenges when trying to extract and flatten specific fields for analysis.
In particular, when arrays and nested objects are mixed, the queries start to get complicated, and performance sometimes drops noticeably.
I am wondering what the best practices are for handling these scenarios. Should I pre-process the JSON before ingestion, or are there advanced SQL functions within Lenses that can efficiently flatten and query this type of data? Checked JSON | Lenses Docs guide related to this and found it quite informative. When I reviewed what is javascript, I noticed its simple JSON handling, and it made me think about whether Lenses SQL supports comparable transformations.
Has anyone here designed an efficient strategy for working with deeply nested JSON in Lenses SQL, particularly when balancing readability, maintainability, and query performance?
Any tips, examples, or real-world use cases would be really helpful for anyone facing similar challenges.
Thank you !!