On prem s3 support by the lenses s3 connector and corresponding config

Apache Kafka Connectors
1

Hi Team,

I am trying to use the lenses s3 source connector to connect with an on-prem s3 deployment

I am not sure where to put in the config specific to on-prem s3 like the domain name e.g. http://s3ext.orgname.com/
I tried to run with the current config and the error I get is
software.amazon.awssdk.services.s3.model.S3Exception: The AWS Access Key Id you provided does not exist in our records

Is this because the key ID I supplied is not in the aws-s3 records?
Is there a specific way to generate the creds with on-prem s3 that will be accepted by the connector?

Can you please suggest and provide an example? It will be really helpful.

Thanks.

2

Hi Anup,

“I tried to run with the current config”

What is the configuration you are running? I don’t know what “current” means:
It looks like the Access Key Id is not valid.

Is there a specific way to generate the creds with on-prem s3 that will be accepted by the connector?

The connector uses AWS Java library, if the connection is valid and the credentials are valid, the connector will work.

3

Hi stheppi

My current config is specific to an on-prem deployment of s3.
s3 buckets can be hosted by implementing the s3 protocol on-prem and not directly in the AWS cloud.
I figured the AWS Java lib is used for the connection.

My question is if the connector supports on-prem deployments for s3 or just the AWS-hosted s3?
If it does support on-prem s3 deployment, are there properties to specify that the connection attempt should be made with on-prem s3 using a domain name and not with AWS s3.

Here is the config for reference.

{
	"name": "aws-s3SourceConnector",
	"config": {
		"connector.class": "io.lenses.streamreactor.connect.aws.s3.source.S3SourceConnector",
		"tasks.max": "1",
		"connect.s3.kcql": "insert into test select * from anups-test-bucket:csv_data STOREAS `CSV_WithHeaders`",
		"connect.s3.aws.region": "ap-south-1",
		"connect.s3.aws.secret.key": "1WPRQMuBx5Rh/hirsY56vdEJaNmE04aoPU9pyZ1K",
		"connect.s3.aws.access.key": "AKIAJPERYWLWE56T3NEQ",
		"connect.s3.aws.auth.mode": "Credentials",
		"value.converter": "org.apache.kafka.connect.storage.StringConverter"
	}
}
4

The connector allows to specify a custom URL for S3 via:

connect.s3.custom.endpoint=http://s3ext.orgname.com/
connect.s3.vhost.bucket=true