Problems with MFA under some load

[ghosftflc]

Description of the working infrastructure

1. 2 DNS records in cloudflare (round robin) for the service name
2. 2 haproxy in two data centers, to which the records from cloudflare refer
3. 2 backends on each haproxy. The backend is an adfs+mfa server.
4. Storage - MSSQL (a separate server in one of the data centers).

Problems are observed when the load on the service increases (approximately after 25-30 users per minute):

1. The service response time increases sharply (visible from haproxy statistics)
2. In the adfs-mfa server logs, errors like "Error on Check Remote Service method: adfs02.xxxxxxx.xxx => This request operation sent to net.tcp://adfs02.xxxxxxx.xxx:5987/ReplayService did not receive a reply within the configured timeout (00:00:15). ..."
3. Errors in responses to users:

• "The operation completed successfully", but there is no login to the target service.
• "Error during authentication process! One Time Password has been played before" when entering the correct code once.
• "Incorrect identification, please try again." when entering the correct code.

To find the cause of the problem, we prepared a test stand and conducted many load tests.
I will not provide the entire list of possible combinations, I will only describe the infrastructure options in which the service worked normally without any errors for users and visible problems:

Infrastructure option 1

1. 1 DNS record in cloudflare for the service name
2. 1 haproxy in one data center, referenced by the record from cloudflare
3. 1 backend on haproxy. The backend is an adfs+mfa server (there is one registered in the mfa cluster).
4. Storage - MSSQL (a separate server in the data center).

Infrastructure option 2

1. 1 DNS record in cloudflare for the service name
2. 1 haproxy in one data center, referenced by the record from cloudflare
3. 1 backend on haproxy. The backend is an adfs+mfa server, but there are two registered servers in the mfa cluster. The second one is enabled, but requests from haproxy do not go to it.
4. Storage - MSSQL (separate server in the data center).

Please help me understand the causes of the problem.
Perhaps this is the expected behavior of the service?
Please suggest proven recommended schemes with balancing of this mfa service.

[redhook62]

Hi, @ghosftflc

I can assure you that it is not with 20 or 30 simultaneous connections that performance issues appear.
First, as indicated in the documentation

• TCP port 5987 must be open on each ADFS server (use Set-MFAFirewallRules on each server)
• The fqdn of each server must respond in the DNS (using entries in the host files is acceptable and sometimes recommended)
• Routing between your ADFS servers must be fast...
  You can disable Anti-Replay to disable caller synchronization between your different servers.
  and then restart the MFA service.

regards

[ghosftflc]

Thanks for the recommendations. Please remind me how I can disable Anti-Replay?

[redhook62]

In the Snappin MMC

https://github.com/neos-sdi/adfsmfa/wiki/07-Security#managing-mfa-security-with-mmc

[ghosftflc]

Do I need the "Replay protection level" to be set to "Disable"?

[redhook62]

Yes, Of course

regards