Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doesn't handle keyvault throttling #219

Open
scottmcburney opened this issue Dec 4, 2023 · 3 comments
Open

Doesn't handle keyvault throttling #219

scottmcburney opened this issue Dec 4, 2023 · 3 comments

Comments

@scottmcburney
Copy link

When a keyvault throttles the signing process, the sign tools hangs. It would be better if it could detect this situation, sleep an amount of time, and retry.
Below is the output from an AzDO pipeline that was throttled.

Unhandled exception. System.AggregateException: One or more errors occurred. (Error calling Key Vault) (Error calling Key Vault)
---> System.Security.Cryptography.CryptographicException: Error calling Key Vault
---> Azure.RequestFailedException: Service request failed.
Status: 429 ()

Content:
{"error":{"code":"Throttled","message":"Request was not processed because too many requests were received. Reason: VaultRequestTypeLimitReached"}}

Headers:
Cache-Control: no-cache
Pragma: no-cache
x-ms-keyvault-region: REDACTED
x-ms-client-request-id: 991a7dda-ca08-4415-81d3-3f8f898ae410
x-ms-request-id: 0e57042c-b600-47ba-bfac-27238ec41a3a
x-ms-keyvault-service-version: REDACTED
x-ms-keyvault-network-info: REDACTED
X-Content-Type-Options: REDACTED
Strict-Transport-Security: REDACTED
Date: Mon, 04 Dec 2023 15:18:07 GMT
Content-Length: 146
Content-Type: application/json; charset=utf-8
Expires: -1

at Azure.Security.KeyVault.KeyVaultPipeline.SendRequest(Request request, CancellationToken cancellationToken)
at Azure.Security.KeyVault.KeyVaultPipeline.SendRequest[TContent,TResult](RequestMethod method, TContent content, Func`1 resultFactory, CancellationToken cancellationToken, String[] path)
at Azure.Security.KeyVault.Keys.Cryptography.RemoteCryptographyClient.Sign(SignatureAlgorithm algorithm, Byte[] digest, CancellationToken cancellationToken)
at Azure.Security.KeyVault.Keys.Cryptography.CryptographyClient.Sign(SignatureAlgorithm algorithm, Byte[] digest, CancellationToken cancellationToken)
at RSAKeyVaultProvider.RSAKeyVault.SignHash(Byte[] hash, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding) in //RSAKeyVaultProvider/RSAKeyVault.cs:line 45
--- End of inner exception stack trace ---
at RSAKeyVaultProvider.RSAKeyVault.SignHash(Byte[] hash, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding) in /
/RSAKeyVaultProvider/RSAKeyVault.cs:line 45
at AzureSign.Core.AuthenticodeKeyVaultSigner.SignCallback(IntPtr pCertContext, IntPtr pvExtra, UInt32 algId, Byte[] pDigestToSign, UInt32 dwDigestToSign, CRYPTOAPI_BLOB& blob) in /_/src/AzureSign.Core/AuthenticodeKeyVaultSigner.cs:line 239

@philsayersCDS
Copy link

I'm seeing the same problem.
I'm signing about 35 files in a single call to AzureSignTool.
My build server logs are showing this: "Signing failed with error 801901AD".

This seems to be the same problem as #135

When Azure Key Vault's API responds with a 429 status code, there needs to be a retry based on the time info (RetryAfter header?) included in the response.
Azure has recommendations for retry behavior here:
https://learn.microsoft.com/en-us/azure/key-vault/general/overview-throttling

Additionally Service limits are available here:
https://learn.microsoft.com/en-us/azure/key-vault/general/service-limits

@thomas-rabiller
Copy link

Added PR to try and properly handle throttling in Azure Keyvault #237

@fabricioferreira
Copy link

fabricioferreira commented Aug 21, 2024

Any updates on this? The PR looks good. Can we have it merged, pretty please?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants