SNOW-1887537: GET only works from AWS, not from Azure or GCP

[tlarsendataguy]

Please answer these questions before submitting your issue.
In order to accurately debug the issue this information is required. Thanks!

1. What version of .NET driver are you using?
   4.2.0

2. What operating system and processor architecture are you using?
   macOS Sequoia 15.1, ARM (M2 processor)

3. What version of .NET framework are you using?
   .NET 8.0

4. What did you do?
   I ran a GET query to download a file from an Internal stage using SnowflakeDbCommand.ExecuteReader().

5. What did you expect to see?
   I expected the file to download. Instead, the query threw the following exception:

Snowflake.Data.Client.SnowflakeDbException : Error: IO operation failed. Error: The given key 'sfcdigest' was not present in the dictionary. SqlState: , VendorCode: 270058, QueryId: 01b9c349-0000-c5e4-000b-748301a32946
  ----> System.Collections.Generic.KeyNotFoundException : The given key 'sfcdigest' was not present in the dictionary.
   at Snowflake.Data.Core.SFFileTransferAgent.execute() in /Users/tlarsen/Documents/Repositories/snowflake-connector-net/Snowflake.Data/Core/FileTransfer/SFFileTransferAgent.cs:line 242
   at Snowflake.Data.Core.SFStatement.ExecuteSqlWithPutGet(Int32 timeout, String sql, String trimmedSql, Dictionary`2 bindings, Boolean describeOnly) in /Users/tlarsen/Documents/Repositories/snowflake-connector-net/Snowflake.Data/Core/SFStatement.cs:line 493
   at Snowflake.Data.Core.SFStatement.Execute(Int32 timeout, String sql, Dictionary`2 bindings, Boolean describeOnly, Boolean asyncExec) in /Users/tlarsen/Documents/Repositories/snowflake-connector-net/Snowflake.Data/Core/SFStatement.cs:line 463
   at Snowflake.Data.Client.SnowflakeDbCommand.ExecuteInternal(Boolean describeOnly, Boolean asyncExec) in /Users/tlarsen/Documents/Repositories/snowflake-connector-net/Snowflake.Data/Client/SnowflakeDbCommand.cs:line 452
   at Snowflake.Data.Client.SnowflakeDbCommand.ExecuteDbDataReader(CommandBehavior behavior) in /Users/tlarsen/Documents/Repositories/snowflake-connector-net/Snowflake.Data/Client/SnowflakeDbCommand.cs:line 267
   at ThisWay.SnowflakeConnector.DownloadFile(String fileName, String saveTo, CancellationToken token) in /Users/tlarsen/Documents/Repositories/thisway/src/SnowflakeConnector.cs:line 59
   at test.SnowflakeConnectorTest.TestDownloadFile() in /Users/tlarsen/Documents/Repositories/thisway/test/SnowflakeConnectorTest.cs:line 31
   at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.BlockUntilCompleted()
   at NUnit.Framework.Internal.MessagePumpStrategy.NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaiter)
   at NUnit.Framework.Internal.AsyncToSyncAdapter.Await[TResult](Func`1 invoke)
   at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(Func`1 invoke)
   at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context)
   at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)
   at NUnit.Framework.Internal.Execution.SimpleWorkItem.<>c__DisplayClass3_0.<PerformWork>b__0()
   at NUnit.Framework.Internal.ContextUtils.<>c__DisplayClass1_0`1.<DoIsolated>b__0(Object _)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at NUnit.Framework.Internal.ContextUtils.DoIsolated(ContextCallback callback, Object state)
   at NUnit.Framework.Internal.ContextUtils.DoIsolated[T](Func`1 func)
   at NUnit.Framework.Internal.Execution.SimpleWorkItem.PerformWork()
--KeyNotFoundException
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at Snowflake.Data.Core.FileTransfer.StorageClient.SFSnowflakeAzureClient.HandleFileHeaderResponse(SFFileMetadata& fileMetadata, BlobProperties response) in /Users/tlarsen/Documents/Repositories/snowflake-connector-net/Snowflake.Data/Core/FileTransfer/StorageClient/SFSnowflakeAzureClient.cs:line 173
   at Snowflake.Data.Core.FileTransfer.StorageClient.SFSnowflakeAzureClient.GetFileHeader(SFFileMetadata fileMetadata) in /Users/tlarsen/Documents/Repositories/snowflake-connector-net/Snowflake.Data/Core/FileTransfer/StorageClient/SFSnowflakeAzureClient.cs:line 119
   at Snowflake.Data.Core.SFFileTransferAgent.initFileMetadata(List`1 files) in /Users/tlarsen/Documents/Repositories/snowflake-connector-net/Snowflake.Data/Core/FileTransfer/SFFileTransferAgent.cs:line 689
   at Snowflake.Data.Core.SFFileTransferAgent.execute() in /Users/tlarsen/Documents/Repositories/snowflake-connector-net/Snowflake.Data/Core/FileTransfer/SFFileTransferAgent.cs:line 199

6. Can you set logging to DEBUG and collect the logs?

The issue seems to arise from line 175 of SFSnowflakeAzureClient.cs:
digest = response.Metadata["sfcdigest"],

The digest property is being set in a FileHeader constructor from the Metadata dictionary. The Metadata dictionary never receives this value, and it doesn't appear the digest property of FileHeader is used anywhere else in the codebase.

When I remove line 175, the GET query successfully downloads the file.

My Snowflake account is on the Azure cloud, in the East US 2 region. I don't know if this issue is limited to Azure cloud, or whether it affects other cloud clients.

[sfc-gh-dszmolka]

hi and thank you for raising this issue with us, also for the pointers! will check it

[sfc-gh-dszmolka]

unfortunately, i can confirm this is indeed a bug, and not just on Azure but GCP platform too; GET fails with a similar exception due to how we want to retrieve the digest for the file, as you pointed out. Thank you again for reporting this issue !

We're going to fix it, and I'll keep this thread posted on the progress.

[sfc-gh-dszmolka]

PR prepared at #1081
edit: and already merged, will be part of the next release