Dotmim.Sync.SyncException: Invalid operation. The connection is closed | Retry stategy?

[badsyntax]

Hi

I'm getting a random db connection issue when syncing. Both the server and the client are Azure SQL server databases. It's unclear if this is a network issue or otherwise. I get this error when running on a build server, and I haven't been able to replicate locally.

Here's what I know:

• When running EF migrations (to multiple clients) on the same build server, i never receive the db connection error.
• When running Dotmim.Sync (to multiple clients), I sometimes (not always) get a Dotmim.Sync.SyncException: Invalid operation. The connection is closed. error
  • (It might sync successfully to a few clients before this error is thrown on the next client)
• If I re-run the sync, then I might or might not get the same error

It appears like a network issues, but then i don't understand why migrations always succeed, so perhaps I'm doing something wrong with the sync.

Can you provide some more information on the default connection retry strategy? Will it try to re-connect (eg due to network issues)? A long shot, but can you suggest any areas for me to debug what might be causing this? Many thanks in advance for any pointers.

Here's the stack trace:

Dotmim.Sync.SyncException: Invalid operation. The connection is closed.
 ---> System.InvalidOperationException: Invalid operation. The connection is closed.
   at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__207_0(Task`1 result)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Dotmim.Sync.BaseOrchestrator.InternalReadSyncTableChangesAsync(ScopeInfo scopeInfo, SyncContext context, Nullable`1 excludintScopeId, SyncTable syncTable, BatchInfo batchInfo, Boolean isNew, Nullable`1 lastTimestamp, DbConnection connection, DbTransaction transaction, CancellationToken cancellationToken, IProgress`1 progress)
   at Dotmim.Sync.BaseOrchestrator.<>c__DisplayClass135_0.<<InternalGetChangesAsync>b__1>d.MoveNext()
--- End of stack trace from previous location ---
   at Dotmim.Sync.BaseOrchestrator.InternalGetChangesAsync(ScopeInfo scopeInfo, SyncContext context, Boolean isNew, Nullable`1 fromLastTimestamp, Nullable`1 toNewTimestamp, Nullable`1 excludingScopeId, Boolean supportsMultiActiveResultSets, String batchRootDirectory, String batchDirectoryName, DbConnection connection, DbTransaction transaction, CancellationToken cancellationToken, IProgress`1 progress)
   at Dotmim.Sync.RemoteOrchestrator.InternalApplyThenGetChangesAsync(ScopeInfoClient cScopeInfoClient, ScopeInfo cScopeInfo, SyncContext context, ClientSyncChanges clientChanges, DbConnection connection, DbTransaction transaction, CancellationToken cancellationToken, IProgress`1 progress)
   at Dotmim.Sync.RemoteOrchestrator.InternalApplyThenGetChangesAsync(ScopeInfoClient cScopeInfoClient, ScopeInfo cScopeInfo, SyncContext context, ClientSyncChanges clientChanges, DbConnection connection, DbTransaction transaction, CancellationToken cancellationToken, IProgress`1 progress)
   at Dotmim.Sync.RemoteOrchestrator.InternalApplyThenGetChangesAsync(ScopeInfoClient cScopeInfoClient, ScopeInfo cScopeInfo, SyncContext context, ClientSyncChanges clientChanges, DbConnection connection, DbTransaction transaction, CancellationToken cancellationToken, IProgress`1 progress)
   --- End of inner exception stack trace ---
   at Dotmim.Sync.RemoteOrchestrator.InternalApplyThenGetChangesAsync(ScopeInfoClient cScopeInfoClient, ScopeInfo cScopeInfo, SyncContext context, ClientSyncChanges clientChanges, DbConnection connection, DbTransaction transaction, CancellationToken cancellationToken, IProgress`1 progress)
   at Dotmim.Sync.SyncAgent.SynchronizeAsync(String scopeName, SyncSetup setup, SyncType syncType, SyncParameters parameters, CancellationToken cancellationToken, IProgress`1 progress)
   at Dotmim.Sync.SyncAgent.SynchronizeAsync(String scopeName, SyncSetup setup, SyncType syncType, SyncParameters parameters, CancellationToken cancellationToken, IProgress`1 progress)

[badsyntax]

I have done a lot more debugging on this, and I've discovered the following:

• I was able to replicate this issue locally
  • I ran the sync in a continuous loop and eventually I got the exception Invalid operation. The connection is closed.
• I tried up-scaling my Azure SQL database, to see if it was db resource related - this made no difference
• I hacked in a RetryLogicProvider into the SqlConnection objects in the library - this made no difference
• I noticed there was some conditional logic in the library depending if MultipleActiveResultSets is enabled. Disabling MultipleActiveResultSets in my connection strings resolved this issue.

Are you aware of any issues related to MARS and async connection handling? I'm running the sync on MacOS, .NET 6.

[Mimetis]

No idea honestly.
I wonder if you don't have too many connections opened at some point, and you are reaching the max pool connection ?