Lenses K2K - "Unrecognized SASL ClientCallback]) occurred when evaluating SASL token received from the Kafka Broker"

Trying to get Lenses K2K replicator to work to replicate from one MSK Serverless instance to another. Via IAM authentication. Here’s my configuration:

name: "gui-test-k2k"
coordination:
  kafka:
    commit:
      group: "k2k.prod-to-dr.coordination"     #required
      topic: "__k2k_consumer-offsets"          #optional
      syncTimeout: "10 seconds"                #optional
      batchSize: 100                           #optional
      batchTimeout: "5 seconds"                #optional
    consumer:
      "group.id": "demo-k2k-coordination"
      "client.id": "test-coordination"
      "security.protocol": "SASL_SSL"
      "sasl.mechanism": "AWS_MSK_IAM"
      "sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;"
      "sasl.client.callback.handler.class": "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
    connection:
      servers: "boot-xxxxxxx.c2.kafka-serverless.eu-west-3.amazonaws.com:9098"
source:
  kafka:
    consumer:
      "client.id": "demo-k2k"
      "security.protocol": "SASL_SSL"
      "sasl.mechanism": "AWS_MSK_IAM"
      "sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;"
      "sasl.client.callback.handler.class": "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
    connection:
      servers: "boot-xxxxxxx.c2.kafka-serverless.eu-west-3.amazonaws.com:9098"
target:
  kafka:
    producer:
      "security.protocol": "SASL_SSL"
      "sasl.mechanism": "AWS_MSK_IAM"
      "sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;"
    connection:
      servers: "boot-xxxxxxx.c2.kafka-serverless.eu-west-3.amazonaws.com:9098"
replication:
  - source:
      name: source                             #required
      topic:                                   #required
        - "mysource-topic-1"
  - sink:
      name: sink                               #required
      partition: source                        #required
      topic:
        prefix: "gui."
        suffix: ".copy"

…but I’m getting the error:

Exception while evaluating challenge [Caused by javax.security.auth.callback.UnsupportedCallbackException: Unrecognized SASL ClientCallback]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.

Full stack trace:

Starting k2k_k2k_1 ... 
Starting k2k_k2k_1 ... done
Attaching to k2k_k2k_1
k2k_1 | 12:43:21.073 [io-compute-blocker-1] INFO io.lenses.k2k.main.Main - 
k2k_1 |  YOUR USE OF AND ACCESS TO THE [LENSES K2K SOFTWARE] (THE "SOFTWARE") IS
k2k_1 |  SUBJECT TO THE TERMS AND CONDITIONS OF THE END USER LICENSE AGREEMENT FOR
k2k_1 |  LENSES.IO COMMUNITY EDITION FOUND AT: https://lenses.io/legals/eula (THE "EULA").
k2k_1 |  BY INSTALLING OR USING THE SOFTWARE IN WHOLE OR IN PART, YOU AGREE TO BE BOUND
k2k_1 |  BY THE EULA AND THE CELONIS PRIVACY NOTICE FOUND AT: HTTPS://WWW.CELONIS.COM/PRIVACY-NOTICE/.
k2k_1 |  WE MAY IN OUR DISCRETION CHANGE THESE TERMS, THE PRIVACY NOTICE OR ANY ASPECT OF
k2k_1 |  THE SOFTWARE, WITHOUT NOTICE TO YOU. IF YOU DO NOT AGREE TO ANY CHANGES, YOU
k2k_1 |  MUST CEASE USING THE SOFTWARE AND DESTROY ALL COPIES.
k2k_1 | 
k2k_1 | 12:43:21.854 [io-compute-blocker-1] DEBUG io.lenses.k2k.main.Main - Evaluating pipeline [name: gui-test-k2k]
k2k_1 | 12:43:22.015 [io-compute-blocker-1] DEBUG io.lenses.k2k.main.Main - producer instantiated:
k2k_1 |  "retries"="2147483647"
k2k_1 |  "bootstrap.servers"="boot-xxxxxxx.c2.kafka-serverless.eu-west-3.amazonaws.com:9098"
k2k_1 |  "sasl.jaas.config"="software.amazon.msk.auth.iam.IAMLoginModule required;"
k2k_1 |  "sasl.mechanism"="AWS_MSK_IAM"
k2k_1 |  "security.protocol"="SASL_SSL"
k2k_1 | 
k2k_1 | 12:43:22.626 [io-compute-blocker-1] DEBUG io.lenses.k2k.main.Main - Instantiated producer.
k2k_1 | 12:43:25.021 [io-compute-1] DEBUG io.lenses.k2k.main.Main - consumer instantiated:
k2k_1 |  "auto.offset.reset"="earliest"
k2k_1 |  "group.id"="demo-k2k-coordination"
k2k_1 |  "client.id"="test-coordination"
k2k_1 |  "sasl.client.callback.handler.class"="software.amazon.msk.auth.iam.IAMClientCallbackHandler"
k2k_1 |  "sasl.jaas.config"="software.amazon.msk.auth.iam.IAMLoginModule required;"
k2k_1 |  "sasl.mechanism"="AWS_MSK_IAM"
k2k_1 |  "security.protocol"="SASL_SSL"
k2k_1 |  "bootstrap.servers"="boot-xxxxxxx.c2.kafka-serverless.eu-west-3.amazonaws.com:9098"
k2k_1 |  "enable.auto.commit"="false"
k2k_1 | 
k2k_1 | 12:43:25.419 [kafka-producer-network-thread | producer-1] ERROR o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node -1 (boot-xxxxxxx.c2.kafka-serverless.eu-west-3.amazonaws.com/172.31.32.225:9098) failed authentication due to: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Exception while evaluating challenge [Caused by javax.security.auth.callback.UnsupportedCallbackException: Unrecognized SASL ClientCallback]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.
k2k_1 | 12:43:25.429 [kafka-producer-network-thread | producer-1] WARN o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Bootstrap broker boot-xxxxxxx.c2.kafka-serverless.eu-west-3.amazonaws.com:9098 (id: -1 rack: null) disconnected

Ensure you put the

"sasl.client.callback.handler.class": "software.amazon.msk.auth.iam.IAMClientCallbackHandler"

for producer, consumer & coordinator.

name: "gui-test-k2k"
coordination:
  kafka:
    commit:
      group: "k2k.prod-to-dr.coordination"     #required
      topic: "__k2k_consumer-offsets"          #optional
      syncTimeout: "10 seconds"                #optional
      batchSize: 100                           #optional
      batchTimeout: "5 seconds"                #optional
    consumer:
      "group.id": "demo-k2k-coordination"
      "client.id": "test-coordination"
      "security.protocol": "SASL_SSL"
      "sasl.mechanism": "AWS_MSK_IAM"
      "sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;"
      "sasl.client.callback.handler.class": "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
    connection:
      servers: "boot-xxxxx.c2.kafka-serverless.eu-west-3.amazonaws.com:9098"
source:
  kafka:
    consumer:
      "client.id": "demo-k2k"
      "security.protocol": "SASL_SSL"
      "sasl.mechanism": "AWS_MSK_IAM"
      "sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;"
      "sasl.client.callback.handler.class": "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
    connection:
      servers: "boot-xxxxx.c2.kafka-serverless.eu-west-3.amazonaws.com:9098"
target:
  kafka:
    producer:
      "security.protocol": "SASL_SSL"
      "sasl.mechanism": "AWS_MSK_IAM"
      "sasl.jaas.config": "software.amazon.msk.auth.iam.IAMLoginModule required;"
      "sasl.client.callback.handler.class": "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
    connection:
      servers: "boot-xxxxx.c2.kafka-serverless.eu-west-3.amazonaws.com:9098"
replication:
  - source:
      name: source                             #required
      topic:                                   #required
        - "mysource-topic-1"
  - sink:
      name: sink                               #required
      partition: source                        #required
      topic:
        prefix: "gui."
        suffix: ".copy"