'AMQP: link closed' errors when using Event Hub Premium

[ranelkayam]

Expected Behavior

Sending events to Azure Event Hub premium via SDK

Actual Behavior

Ever since we migrated from Event Hub Standard to Event Hub Premium, we started experiencing sporadic errors of "AMQP: link closed".
Upgrading to 'github.com/Azure/azure-event-hubs-go/v3 v3.6.1 (github.com/Azure/azure-amqp-common-go/v4/aad)' only made the problem exponentially worse, and struck down our ability to send events completely

Environment

• OS: Azure AKS
• Go version: 1.16.15
• Version of Library: github.com/Azure/azure-event-hubs-go/v3 v3.3.20

[akhudairymicrosoft]

Sharing tips that we got on another email thread by GO SDK team.

azure-event-hubs-go library is deprecated - we have a new library out at https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/messaging/azeventhubs. The new library has better error recovery, more efficient use of connections, better diagnostics, etc.. There are sender scenarios in the old library that are more or less impossible for clients to work around.

There's a migration guide for moving from the old to the new: https://github.com/Azure/azure-sdk-for-go/blob/main/sdk/messaging/azeventhubs/migrationguide.md

Regarding the old library azure-event-hubs-go:
The older library does a less granular recovery than the newer library. When it encounters an error with a sender.Send it closes the connection, which will cause any senders that share that connection to also close.We might be seeing a close on some senders but the original cause hasn't been identified yet

There isn't a built-in way to activate better logging but following pieces of code could help:

Here's an example logger built for a test. You can just create a copy of this file in your app, and possibly change it to redirect to your logging system. Note, the logging that comes out of this can be pretty verbose.

https://github.com/Azure/azure-event-hubs-go/blob/master/internal/stress/stderrTracer.go

And here's the lines that show how to add it so it'll be used by the Event Hubs library:
https://github.com/Azure/azure-event-hubs-go/blob/07e5a11a61336dfb2c5d6276c0298bb4e2d38625/internal/stress/throttling/throttling.go#L34C1-L38C35"