Is it possible to improve the performance of queries in the Lenses SQL Explorer.
Hello ejhon_s,
Let’s consider as a starting point that your Lenses uses the amount of memory and CPU that allow it to process your workloads. However, the necessary amount of memory and CPU doesn’t necessarily mean it will run faster. Improving the performance of Lenses needs to take into account several factors that are not directly related to Lenses but contribute to performance enhancement. These include:
- Parallelization of query execution: query.parallel
- The amount of data extracted per consumer: max.poll.records
- The maximum amount of extracted data: max.size
- Filtering data using a time period: _meta.timestamp
- Optimized nodes for CPU-intensive use
- Increasing the number of CPUs used in the helm chart
- Using newer versions of Lenses
- Using the latest Java
Despite these tips, it’s essential to consider the network bandwidth where Kafka is located, the type of disk it uses, and the amount of data you intend to fetch, the cloud you are using to host your infrastructure, for example, AWS, offers Graviton3 ARM-based instances (C7g or C7gn), which are promoted for their CPU performance.