Insert message does not work for JSON topic in Lenses - "does not have a schema for the Value"

Lenses
1

Lenses.io created this topic as part of the seeding process for our community

We have a Kafka topic named test, with a key type of string, and value type of JSON.
When we try to insert messages into a valid JSON message, like the one below, Lenses gives out the following error:

Cannot insert data. `TableName(test)` does not have a schema for the Value.

This is what we use as the entry:

[
  {
    "key": "user1",
    "value": {
      "name": "John Doe",
      "email": "john.doe@example.com",
      "address": {
          "street": "123 Main St",
          "city": "Anytown",
          "state": "CA",
          "zip": "12345"
      },
      "phone_numbers": [
          {"type": "home", "number": "555-1234"},
          {"type": "work", "number": "555-5678"}
      ]
    }
  }
]
2 Likes
2

Starting with Lenses 5.0, we consciously decided to enforce schema validation for JSON-serialized keys and values, just like we do for their AVRO counterparts.

Whilst this choice may be inconvenient sometimes, it will protect you from inserting messages that can break your streaming pipelines or consumer code.

In your case, you must visit the Schema tab of the test topic and add a schema like the one below for the topic’s value field. The schema must be in AVRO format:

{
  "type": "record",
  "name": "Person",
  "namespace": "example",
  "fields": [
    {
      "name": "name",
      "type": "string"
    },
    {
      "name": "email",
      "type": "string"
    },
    {
      "name": "address",
      "type": {
        "type": "record",
        "name": "Address",
        "fields": [
          {
            "name": "street",
            "type": "string"
          },
          {
            "name": "city",
            "type": "string"
          },
          {
            "name": "state",
            "type": "string"
          },
          {
            "name": "zip",
            "type": "string"
          }
        ]
      }
    },
    {
      "name": "phone_numbers",
      "type": {
        "type": "array",
        "items": {
          "type": "record",
          "name": "PhoneNumber",
          "fields": [
            {
              "name": "type",
              "type": "string"
            },
            {
              "name": "number",
              "type": "string"
            }
          ]
        }
      }
    }
  ]
}

Best,
Marios

3 Likes
3

If it’s easier, you can create the Lenses schema in Lenses using a SQL CREATE TABLE command. Eg:

CREATE TABLE 
   test
     (_key string, name string, email string, address.street string, address.city string,address.state string, address.zip string, phone_numbers.type string[], phone_numbers.number string[])
FORMAT 
   (string, json)
PROPERTIES 
   (partitions=1, compacted=false);

And then insert

INSERT INTO test
 (_key, name, email, address.street, address.city, address.state, address.zip, phone_numbers.type, phone_numbers.number)
   VALUES
   ("user1","John Doe","john.doe@example.com","123 Main St","Anytown", "CA", "12345", ["home","work"], ["555-1234","555-5678"])

(not exactly the same array structure for phone_numbers, maybe someone else can correct this)

1 Like
4

Lenses 5.5.4 now has the option to disable schema validation when inserting JSON messages via Lenses