I see that overtime on kafka lenses connect s3sink, cpu starts low but then increases.
I suspect it is related to the Hashmap that contains keys per partition written to s3 and over time the writers for each partition is not closed and occumlates - causing this state of map to get bigger and cpu is increased when all these writers are working during flush and are not closed.
Restart solves the problem so it looks like a state that is increasing
and impacting the cpu.
How can i configure some ttl or close operation on the writers to be closed and removed from this map let’s say after a day with no new data arriving to their partition?
Something like rotate.interval that will close the writers on flush or
writer.inactive.ttl to remove them after time out
From what i see in lenses stream reactor code - close of writers is only callked when
restart the connector
When the worker is stopped or crashes
when task rebalancing happens (e.g. another worker joins or leaves the group)
Does this make sense? This causes a problem with cpu
Any ideas?