No information for some partitions and consumer groups

Lenses
1

Hello. It appears that Lenses is missing information for some of my partitions. This has the following effects:

  • I cannot see data in the explore page for the affected topics, or I cannot perform queries for these topics in SQL Studio
  • Consumer groups that are consuming from these topics miss the consumer lag, start and latest offsets information for the affected partitions
  • Lenses reports these partitions as having zero messages

Any advice?

2

This may be an indication of a performance problem. Usually, it will not show until you have a few thousands of partitions and hundreds of consumers.

To work around this issue, you can try to increase the intervals that Lenses is querying the Kafka cluster for information and the allowed memory usage.

You can start by doubling these intervals in lenses.conf (default values are 10000):

lenses.interval.summary=20000
lenses.interval.consumers.refresh.ms=20000
lenses.interval.partitions.messages=20000
lenses.akka.request.timeout.ms=20000

To learn more about these↑ settings, check here.

Then to increase allowed memory usage, use this environment variable:

LENSES_HEAP_OPTS="-Xmx10g -Xms5g"

If you are using the Helm Charts, you can try these values (tested with Lenses 5.4):

resources:
  requests:
    cpu: 4
    memory: 12Gi
  limits:
    cpu: 6
    memory: 16Gi
lenses:
  append:
    conf: |-
      lenses.interval.summary=20000
      lenses.interval.consumers.refresh.ms=20000
      lenses.interval.partitions.messages=20000
      lenses.akka.request.timeout.ms=20000
  jvm:
    heapOpts: -Xmx10g -Xms5g

Cheers!

3 Likes