Hi everyone,
I'm using the DatalakeSinkConnector (StreamReactor v11) to write Parquet files
to Azure Data Lake Storage Gen2, and I'm trying to set up Snowpipe (Snowflake
automated ingest) to load them automatically.
The problem is that Snowpipe on ADLS Gen2 requires a BlobCreated event with
api=FlushWithClose to trigger ingestion — CreateFile events alone are explicitly
ignored, and BlobRenamed events are not supported at all.
After testing both configurations, here's what I observed via Azure Event Grid:
**connect.datalake.exactly.once.enable: true**
The connector writes to a .temp-upload staging directory and renames the file
atomically to the final destination. The final file arrives as a BlobRenamed
event (api=RenameFile) → Snowpipe never sees it.
**connect.datalake.exactly.once.enable: false**
The connector writes directly to the final path via CreateFile + AppendFile,
but the file is never closed with FlushFile(close=true). Even with all Event Grid
filters removed, only BlobCreated(CreateFile) events are generated — no
FlushWithClose ever appears → Snowpipe ignores them.
The Snowflake documentation is explicit on this:
https://docs.snowflake.com/en/user-guide/data-load-snowpipe-auto-azure
Has anyone found a workaround for this? Is there a connector config I'm missing
that forces FlushWithClose on file commit?
I will also open a GitHub issue to request it as a feature — just wanted to
check first if this is a known limitation or if someone has solved it already.
Thanks