How to Publish Metrics from Lenses Kafka Connect Http Sink Connector to Prometheus?

I’m using the Lenses Kafka Connect Http Sink Connector and need to publish operational metrics (like 2xx/4xx/5xx error counts, API response times, etc.) to Prometheus. These metrics will help in setting up dashboards and alerts for operational excellence.

The connector version is 8.1.16, and I am using Kafka version 3.4. The configuration includes properties like retry settings, batching, request content, and HTTP endpoints. Here’s a sample of the configuration:

connector.class: io.lenses.streamreactor.connect.http.sink.HttpSinkConnector
errors.tolerance: all
connect.http.retries.max.retries: 3
connect.http.retries.on.status.codes: 408,429,500,502,503,504
connect.http.method: POST
connect.http.endpoint: https://example.com/api
connect.http.request.headers: Content-Type: application/json

Is there a way to directly expose these metrics to Prometheus from the connector? If not, what would be the best practice for integrating these metrics into a Prometheus monitoring setup?

The Lenses Kafka Connect Http Sink Connector does not directly publish metrics to Prometheus. However, starting with version 8.1.24, the connector supports exposing metrics through JMX, which can be scraped by Prometheus using a JMX exporter.

Steps to Enable Metrics for Prometheus:

  1. Upgrade the Connector:

Download version 8.1.24 or later from the Stream Reactor releases page.

  1. Enable JMX Metrics in the Connector:

In the new release, the connector exposes the following metrics via JMX under the MBean:

io.lenses.streamreactor.connect.http.sink:type=metrics,name=$sinkName.

Metrics include:

• 2xxCount, 4xxCount, 5xxCount, OtherErrorsCount

• P50RequestTimeMs, P95RequestTimeMs, P99RequestTimeMs (computed over a 5-minute time window)

  1. Set Up a Prometheus JMX Exporter:

Configure a JMX exporter (e.g., prometheus/jmx_exporter) to scrape these JMX metrics. This step involves:

• Running the JMX exporter as a sidecar or as part of your Kafka Connect process.

• Updating your Prometheus configuration to scrape metrics from the JMX exporter endpoint.

  1. Monitor and Validate:

Once the setup is complete, validate that the metrics are visible in Prometheus. You can create dashboards and alerts based on these metrics.

Why Not Direct Integration?

Direct Prometheus integration from the connector would lead to tight coupling with Prometheus, making it an anti-pattern for connectors. Using JMX ensures flexibility, allowing teams to use different monitoring systems without modifying the connector.

For further details on the release and configuration, refer to the release notes.