Serious and growing hack on easyengine sites - coming from local IP

[ssuess]

I just migrated about 6 sites to a brand new server with easyengine 4 setup. The migration went well but now it seems I have a disturbing and growing hack that I am trying unsuccessfully to track down. Here is what is happening:

• With increasing frequency across my sites there are failed login attempts seemingly originating from 172.19.0.1 (an IP local to the server)
• Wordfence and other software won't allow me to block this IP (which I guess makes sense since it is local)
• When I run a command to get the IPs of all my docker containers, this exact IP is not present (but several others in the range are like 172.19.0.3, 172.19.0.6 etc. All of these are on nginx containers.
• The logs show calls from 172.19.0.1 to xmlrpc.php and wp-cron.php, which then result in the failed logins (that my activity viewer is showing). Fortunately so far these seem to be against non existent usernames.
• There are no file changes I can detect in any of my sites, even as new ones on the server are added to these hack attempts. If they were coming from one of the IPs on any of the nginx containers, I would shut down that site. But disabling a few sites and waiting does not seem to stop these attacks.
• looking through the logs on ANY of the sites I migrated shows no evidence of these attacks before migrating to easyengine, so I am really at a loss.
  I am getting increasingly desperate to try to figure out why this is happening and how to stop it, but I can't even figure out how to shutdown 172.19.0.1 which seems to be the top level easyengine address (it is pingable on the server itself).

[ssuess]

I guess I would assume the ee global_nginx_proxy would hold that IP, but it seems to be at 172.19.0.8. Maybe there is a bug in this proxy that allows certain requests to not have their correct IP forwarded to the other sites, and instead be passed 172.19.0.1?

[ssuess]

Even if I disable ALL sites (leaving only the proxy and redis running apparently) I get spoofed IPs in the logs like this:

172.19.0.1 - - [30/Oct/2021:10:10:14 +0000] "HEAD /feed/ HTTP/1.1" 404 0 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0" "-"

and sometimes error log entries like this:

2021/10/30 10:00:21 [error] 28#28: *13 "/etc/nginx/html/feed/index.html" is not found (2: No such file or directory), client: 172.19.0.1, server: , request: "GET /feed/ HTTP/1.1", host: "mysite.net"

How is this possible??

[ssuess]

I believe I found the problem and if I am correct it is a quite serious one for EE, pointing either to some problem with off the shelf, default ee setup or somehow something I have missed. But here it is in a nutshell:

• On EE machines that have IPv6 enabled, an IPv6 request will get translated by the proxy to internal local IPv4
• This allows all number of attempted hacks to bypass security mechanisms, BECAUSE THEY APPEAR TO BE LOCAL
  I will report this in issues as a bug, but here is how I figured it out and tested it:
• ran an ipv6 reachability test (https://ipv6-test.com/validate.php) on a couple of my sites (one ee, one manual setup on another machine) which both passed.
• When i check the logs on the ee machine (in the logs in /opt/easyengine/services/nginx-proxy/logs/access.log) that test shows up as coming from 172.19.0.1
• When I check the logs on the manually configured apache machine, that test shows up as coming from 2001:41d0:701:1100::29c8
  So clearly either my machine (although I setup a fresh ee machine to test just this from scratch with only one site) or the proxy setup for ee does not properly forward or deal with IPv6, and it really needs to if we are to avoid hacking attempts like this. I will report this in issues separately, but just for anyone that was following this.

[digit-c]

@ssuess Was this submitted and fixed?

[ssuess]

#1621