Hi, we have a few connectors sink + source running on production, some are based on high loaded topics and some are not. We see that in case we define two processes for lenses connector - there is a big impact if one high volumned topics is only processed by one connector and if we restart the connector and then the same topic is divided in partitions between the two connectors - the cpu is better.
Is there a way to define how to divide the partitions between the two connectors? Or every restart it will be defined differently and there is no control?
Kafka Connect dynamically assigns partitions to tasks, and this assignment can change on restart. There’s no built-in way to manually control how partitions are distributed between connector instances. However, if you’re seeing better CPU usage when partitions are more evenly split, increasing tasks.max can help distribute the load more effectively.
It would be useful to know which connectors you’re referring to, as some may have configuration options that influence partition assignment. Let me know, and I can provide more specific guidance.
Hi, regarding what you said on the max.tasks - even if we increase it - we still see that on some cases a full topic is going to one worker. And if one topic is very heavy it is problematic.
We have Source and Sink Connectors:
“connector.class”: “io.lenses.streamreactor.connect.aws.s3.sink.S3SinkConnector”,
“connector.class”: “io.lenses.streamreactor.connect.aws.s3.source.S3SourceConnector”,
Both are consuming high cpu
If we have about 6-10 different connectors
They consume about 80% cpu
The S3SinkConnector which is consuming from kafka is consuming from about 10 kafka topics.
Some of the topics are more heavy loaded then others
Also looks like tasks are not well balanced between workers. Not just partitions.
I have 2 workers - and one has more tasks then the other
Do you recomend using [connect.protocol] (Apache Kafka)
to better balance?