“max size reached” message in Lenses SQL Studio

Lenses SQL
1

Using the SQL Studio, my query is displaying the following message:

“Query is finished, max size reached (27 records in 21.37 seconds)”

Here is my query:

select * from user-events-apps where _meta.timestamp > now()-'48h' and eventName="Main User";

How can I avoid hitting the max size limit and have the search return more data?

2

Hello ejhon_s,

This is normal behaviour to ensure a single person cannot consume too many resources when querying data in Kafka with Lenses.

When the SQL Snapshot engine performs queries, by default, it scans only 200MB of data in the topic. To execute the query on more data from the topic, you need to configure the “max.size” parameter, for example:

SET max.size = '1g';
SELECT * FROM user-events-apps WHERE 
_meta.timestamp > now()-'48h' AND eventName="Error User";

There are other configurations that can be made in the SQL Studio, take a look at them as there might be other features that could be useful to you, link:

To increase the amount of data used in the query affects Kafka processing when retrieving messages, increases network bandwidth consumption between Kafka and Lenses, and extends Lenses ‘ response time due to processing more data. Change with caution to avoid negatively impacting your environment.