K2K standalone replication example on Community Edition with two localhost Kafka clusters

Lenses
1

I want to test Lenses K2K replication between two clusters connected to Lenses Community edition.

Does anyone have some good k2k pipeline examples I can copy from?

2

I’ve got Community Edition connected to two localhost (fast-data-dev) Kafka clusters.

I’ve given the names:

us-prod
us-dev

(you can copy my docker-compose config from this github repo). My environment looks like this:

image
image1642×920 49.4 KB

I’m copying telecom_italia_data topic from kafka-cluster-1(prod) cluster to copy.telecom_italia_data on kafka-cluster-2(dev) cluster.

This is my pipelines/telecom_k2K_pipeline.yaml k2k (version 0.5) pipeline file:

name: "telecom-italia-replication"
source:
  kafka:
    common:
      "bootstrap.servers": "kafka-cluster-1:9092"
      "security.protocol": "PLAINTEXT"
    consumer:
      "group.id": "k2k.my-first-k2k"
  registry:
    config:
       "schema.registry.url": "http://kafka-cluster-1:8081"
target:
  kafka:
    common:
      "bootstrap.servers": "kafka-cluster-2:9092"
      "security.protocol": "PLAINTEXT"
  registry:
    config:
       "schema.registry.url": "http://kafka-cluster-2:8081"
replication:
  - source:
      topic: "telecom_italia_data"
  - sink:
      topic:
        prefix: "copy."
      partition: source
coordination:
  kafka:
    commit:
      group: "k2k.telecom-italia-replication.coordination"
      topic: "__k2k_consumer-offsets"
features:
  exactlyOnce: enabled
  schemaMapping: enabled
  autoCreateControlTopics: enabled
  autoCreateTopics: enabled

license:
  token: xxxxx-64cd-4b2e-a560-433ac2719fb3
  acceptEula: true

You may need to email k2k@lenses.io to get the license token.

…and then docker command:

docker run --rm -it \
  --network lenses_community_edition_2_kafkas_lenses-network \
  -v "$(pwd)/pipelines:/pipelines" \
  -e K2K_PIPELINE_FOLDER="/pipelines" \
  -e OTEL_SERVICE_NAME="k2k" \
  -e OTEL_METRICS_EXPORTER=console \
  -e OTEL_TRACES_EXPORTER=none \
  --name k2k \
  lensesio/k2k:0.5 \
  k2k start -f /pipelines/telecom_k2K_pipeline.yaml -t

You should see:

image
image1720×1200 171 KB