DurableTaskClient throwing RpcException "An attempt was made to access a socket in a way forbidden by its access permissions"

[dfaivre]

It appears that randomly, running in our Azure live environment, most (any?) calls made using the DurableTaskClient are failing with an RpcException:

Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error connecting to subchannel.", DebugException="System.Net.Sockets.SocketException: An attempt was made to access a socket in a way forbidden by its access permissions.")
 ---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport.TryConnectAsync(ConnectContext context)
   --- End of inner exception stack trace ---
   at Grpc.Net.Client.Balancer.Internal.ConnectionManager.PickAsync(PickContext context, Boolean waitForReady, CancellationToken cancellationToken)
   at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
   at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.GetInstancesAsync(String instanceId, Boolean getInputsAndOutputs, CancellationToken cancellation)
...

If I run the functions app locally (configured to point to the same live environment) I do not get the error.

I haven't restarted our Azure deployed functions app yet because I opened a support request and wanted them to be able to have a reproducible environment.

Azure Support Request ID: 2411070040008596

[etheone]

Experiencing the same issue. Any leads?

[dfaivre]

Experiencing the same issue. Any leads?

No. Azure support request is getting no where fast. I have to reboot the the Functions App whenever the issue pops up. :(

[dfaivre]

@etheone I did post the same issue to another repo though: Azure/azure-functions-durable-extension#2970