Trying to get Lenses K2K replicator to work to replicate from one MSK Serverless instance to another. Via IAM authentication. Here’s my configuration:
name: "gui-test-k2k"
coordination:
kafka:
commit:
group: "k2k.prod-to-dr.coordination" #required
topic: "__k2k_consumer-offsets" #optional
syncTimeout: "10 seconds" #optional
batchSize: 100 #optional
batchTimeout: "5 seconds" #optional
consumer:
"group.id": "demo-k2k-coordination"
"client.id": "test-coordination"
"security.protocol": "SASL_SSL"
"sasl.mechanism": "AWS_MSK_IAM"
"sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;"
"sasl.client.callback.handler.class": "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
connection:
servers: "boot-xxxxxxx.c2.kafka-serverless.eu-west-3.amazonaws.com:9098"
source:
kafka:
consumer:
"client.id": "demo-k2k"
"security.protocol": "SASL_SSL"
"sasl.mechanism": "AWS_MSK_IAM"
"sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;"
"sasl.client.callback.handler.class": "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
connection:
servers: "boot-xxxxxxx.c2.kafka-serverless.eu-west-3.amazonaws.com:9098"
target:
kafka:
producer:
"security.protocol": "SASL_SSL"
"sasl.mechanism": "AWS_MSK_IAM"
"sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;"
connection:
servers: "boot-xxxxxxx.c2.kafka-serverless.eu-west-3.amazonaws.com:9098"
replication:
- source:
name: source #required
topic: #required
- "mysource-topic-1"
- sink:
name: sink #required
partition: source #required
topic:
prefix: "gui."
suffix: ".copy"
…but I’m getting the error:
Exception while evaluating challenge [Caused by javax.security.auth.callback.UnsupportedCallbackException: Unrecognized SASL ClientCallback]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.
Full stack trace:
Starting k2k_k2k_1 ...
Starting k2k_k2k_1 ... done
Attaching to k2k_k2k_1
k2k_1 | 12:43:21.073 [io-compute-blocker-1] INFO io.lenses.k2k.main.Main -
k2k_1 | YOUR USE OF AND ACCESS TO THE [LENSES K2K SOFTWARE] (THE "SOFTWARE") IS
k2k_1 | SUBJECT TO THE TERMS AND CONDITIONS OF THE END USER LICENSE AGREEMENT FOR
k2k_1 | LENSES.IO COMMUNITY EDITION FOUND AT: https://lenses.io/legals/eula (THE "EULA").
k2k_1 | BY INSTALLING OR USING THE SOFTWARE IN WHOLE OR IN PART, YOU AGREE TO BE BOUND
k2k_1 | BY THE EULA AND THE CELONIS PRIVACY NOTICE FOUND AT: HTTPS://WWW.CELONIS.COM/PRIVACY-NOTICE/.
k2k_1 | WE MAY IN OUR DISCRETION CHANGE THESE TERMS, THE PRIVACY NOTICE OR ANY ASPECT OF
k2k_1 | THE SOFTWARE, WITHOUT NOTICE TO YOU. IF YOU DO NOT AGREE TO ANY CHANGES, YOU
k2k_1 | MUST CEASE USING THE SOFTWARE AND DESTROY ALL COPIES.
k2k_1 |
k2k_1 | 12:43:21.854 [io-compute-blocker-1] DEBUG io.lenses.k2k.main.Main - Evaluating pipeline [name: gui-test-k2k]
k2k_1 | 12:43:22.015 [io-compute-blocker-1] DEBUG io.lenses.k2k.main.Main - producer instantiated:
k2k_1 | "retries"="2147483647"
k2k_1 | "bootstrap.servers"="boot-xxxxxxx.c2.kafka-serverless.eu-west-3.amazonaws.com:9098"
k2k_1 | "sasl.jaas.config"="software.amazon.msk.auth.iam.IAMLoginModule required;"
k2k_1 | "sasl.mechanism"="AWS_MSK_IAM"
k2k_1 | "security.protocol"="SASL_SSL"
k2k_1 |
k2k_1 | 12:43:22.626 [io-compute-blocker-1] DEBUG io.lenses.k2k.main.Main - Instantiated producer.
k2k_1 | 12:43:25.021 [io-compute-1] DEBUG io.lenses.k2k.main.Main - consumer instantiated:
k2k_1 | "auto.offset.reset"="earliest"
k2k_1 | "group.id"="demo-k2k-coordination"
k2k_1 | "client.id"="test-coordination"
k2k_1 | "sasl.client.callback.handler.class"="software.amazon.msk.auth.iam.IAMClientCallbackHandler"
k2k_1 | "sasl.jaas.config"="software.amazon.msk.auth.iam.IAMLoginModule required;"
k2k_1 | "sasl.mechanism"="AWS_MSK_IAM"
k2k_1 | "security.protocol"="SASL_SSL"
k2k_1 | "bootstrap.servers"="boot-xxxxxxx.c2.kafka-serverless.eu-west-3.amazonaws.com:9098"
k2k_1 | "enable.auto.commit"="false"
k2k_1 |
k2k_1 | 12:43:25.419 [kafka-producer-network-thread | producer-1] ERROR o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node -1 (boot-xxxxxxx.c2.kafka-serverless.eu-west-3.amazonaws.com/172.31.32.225:9098) failed authentication due to: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Exception while evaluating challenge [Caused by javax.security.auth.callback.UnsupportedCallbackException: Unrecognized SASL ClientCallback]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.
k2k_1 | 12:43:25.429 [kafka-producer-network-thread | producer-1] WARN o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Bootstrap broker boot-xxxxxxx.c2.kafka-serverless.eu-west-3.amazonaws.com:9098 (id: -1 rack: null) disconnected