# Location of stored data in S3 with the Lenses connector

**URL:** <https://ask.lenses.io/t/location-of-stored-data-in-s3-with-the-lenses-connector/201>\
**Category:** Apache Kafka Connectors\
**Tags:** kafka-connect\
**Created:** [April 9, 2024, 9:12pm UTC](https://ask.lenses.io/t/location-of-stored-data-in-s3-with-the-lenses-connector/201 "2024-04-09T21:12:41Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![cwoodruff](https://dub1.discourse-cdn.com/flex017/user_avatar/ask.lenses.io/cwoodruff/32/111_2.png) [@cwoodruff](https://ask.lenses.io/u/cwoodruff)\
**Post date:** [April 9, 2024, 9:12pm UTC](https://ask.lenses.io/t/location-of-stored-data-in-s3-with-the-lenses-connector/201/1 "2024-04-09T21:12:41Z")

</div>

My sink connector with the following configuration started successfully. The SinkRecordSendRate and SinkRecordReadRate metrics showed data was flowing through the connector to the s3 destination, but when I checked S3 I did not see the data. I expected to see a folder named after the topic being backed up.

connector.class = io.lenses.streamreactor.connect.aws.s3.sink.S3SinkConnector  
s3.region = us-east-1  
key.converter.schemas.enable = false  
connect.s3.kcql = INSERT INTO $BUCKET\_NAME SELECT \* FROM lensestestTopic STOREAS JSON  
connect.s3.aws.region = us-east-1  
schema.compatibility = NONE  
tasks.max = 2  
topics = lensestestTopic  
schema.enabled = false  
errors.log.enable = true  
errors.tolerance = “all”  
errors.log.include.messages = true  
config.action.reload = “restart”  
errors.deadletterqueue.topic.name = “dlq\_file\_sink”  
errors.deadletterqueue.topic.replication.factor = “1”  
value.converter = org.apache.kafka.connect.storage.StringConverter  
key.converter = org.apache.kafka.connect.storage.StringConverter"

The service execution role had these permissions.

```
    {
        "Action": [
            "s3:ListBucket",
            "s3:GetBucketLocation",
            "s3:DeleteObject",
            "s3:PutObject",
            "s3:GetObject",
            "s3:AbortMultipartUpload",
            "s3:ListMultipartUploadParts",
            "s3:ListBucketMultipartUploads"
        ],
        "Effect": "Allow",
        "Resource": [
            "arn:aws:s3:::*",
            "arn:aws:s3:::*/*"
         ]

```

I did see the three .indexes objects in s3 for the connector, but that was all.

Was an additional configuration required for data to be placed in the folder mentioned above?

Where did the data get stored in s3? There is no sign of any new data being written to the bucket.

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![cwoodruff](https://dub1.discourse-cdn.com/flex017/user_avatar/ask.lenses.io/cwoodruff/32/111_2.png) [@cwoodruff](https://ask.lenses.io/u/cwoodruff)\
**Post date:** [April 10, 2024, 4:43pm UTC](https://ask.lenses.io/t/location-of-stored-data-in-s3-with-the-lenses-connector/201/2 "2024-04-10T16:43:10Z")

</div>

I checked the s3 bucket today and saw the data. Not sure why it took so long to see the results since there were only 260 lines of text in that particular topic.

---

<div class="post-metadata">

**Author:** ![Simon\_Duchene\_lenses](https://dub1.discourse-cdn.com/flex017/user_avatar/ask.lenses.io/simon_duchene_lenses/32/104_2.png) [@Simon\_Duchene\_lenses](https://ask.lenses.io/u/Simon_Duchene_lenses)\
**Post date:** [April 15, 2024, 8:55pm UTC](https://ask.lenses.io/t/location-of-stored-data-in-s3-with-the-lenses-connector/201/3 "2024-04-15T20:55:08Z")

</div>

Hello, your config is incorrect, this is why you have this beahvior, please refer to [Kafka to AWS S3 open source connector | Lenses.io Documentation](https://docs.lenses.io/5.4/connectors/sinks/s3sinkconnector/) for examples.  
for example your topic name shouldn’t be $BUCKET\_NAME, or another example is you should enable the converter, since you have txt data to convert to json, etc…  
I hope this helps

---

<div class="post-metadata">

**Author:** ![Simon\_Duchene\_lenses](https://dub1.discourse-cdn.com/flex017/user_avatar/ask.lenses.io/simon_duchene_lenses/32/104_2.png) [@Simon\_Duchene\_lenses](https://ask.lenses.io/u/Simon_Duchene_lenses)\
**Post date:** [April 15, 2024, 9:47pm UTC](https://ask.lenses.io/t/location-of-stored-data-in-s3-with-the-lenses-connector/201/4 "2024-04-15T21:47:46Z")

</div>

a working config example on my lab is

> connector.class=io.lenses.streamreactor.connect.aws.s3.sink.S3SinkConnector  
> connect.s3.kcql=INSERT INTO `mybucket:myprefix` SELECT \* FROM `mytopic` STOREAS `JSON` WITH\_FLUSH\_SIZE = 1000000 WITH\_FLUSH\_INTERVAL = 3600 PROPERTIES (‘store.envelope’ = true)  
> topics=mytopic  
> tasks.max=1  
> name=demo\_sink\_s3\_toJSON  
> value.converter=org.apache.kafka.connect.storage.StringConverter  
> key.converter=org.apache.kafka.connect.storage.StringConverter