Cross-host facts. Modern hosts. No firewall.

[drjrkuhn]

I fixed a few things on the omero-three-node deployment sample. This worked for me with three freshly installed Centos 7 VMs. Fixes include:

• Ansible now prefers underscores to dashes in host names, omero-database => omero_database etc
• hostvars are not available for hosts outside of the current -hosts: block. So made a common block to gather connection addresses.
• Gathering hostvars from the first host in a group works better in my hands. Also, used the ansible_default_ipv4 address.
• Centos7 firewall blocks cross-host traffic. Turned off the firewalls for these internal connection through the ome.iptables_raw role. Future versions should specifically allow psql, omero, and web traffic on each specific host type via ome.iptables_raw or ansible.posix.firewalld rather than this blanket firewall turn-off.
• The omero-server needs the psql user and password to connect to the remote database. I'm not sure how secure this user:password combination of omero:omero will work now that the database and omero-server on on different machines.

[pwalczysko]

@joshmoore I am afraid I cannot test this easily - I am failing on step 1: Have 3 VMs and know how to run ansible against all three of them - for now, all I did was having one single docker (not VM) and inside that docker an ansible which I used with --connection=local flag. That has the strong reason that I did not know any better - if you can lend me a hand with the basic setup, I can run the playbook, but atm not sure where to start.

[drjrkuhn]

I should probably learn molecule and vagrant to write proper tests. It's all part of my learning process.

Maybe I should drop the pull request and resubmit once I have something that can be properly tested.

[joshmoore]

Hi @drjrkuhn. 💯 for the learning process and for anything we can do to help. If you are looking to get into molecule & vagrant anyway, that's definitely a great way to test this.

From our POV, don't feel that you must close before doing all of that though. The changes look sane, but it'd be good to have one verification (automated or manual) of the iptables before merging.

[pwalczysko]

So I have created a Vagrantfile with the help of @dominikl (thank you very much) as follows (no aspiration on beauty of the file, it is rough):

Vagrantfile

Vagrant.configure("2") do |config|
	
    config.vm.define "web" do |web|
        web.vm.box = "centos/7"
        web.vm.network "private_network", ip: "192.168.56.100", :name => 'vboxnet0', :adapter => 2
        web.vm.network "forwarded_port", guest: 80, host: 8080, auto_correct: true
        
        web.vm.provider "virtualbox" do |vb|
            vb.name = "web"
            vb.memory = "2048"
            vb.cpus = "2"
        end
    end
    
    config.vm.define "postgres" do |postgres|
        postgres.vm.box = "centos/7"
        postgres.vm.network "private_network", ip: "192.168.56.101", :name => 'vboxnet0', :adapter => 2
        postgres.vm.network "forwarded_port", guest: 443, host: 8443, auto_correct: true
        
        postgres.vm.provider "virtualbox" do |vb|
            vb.name = "postgres"
            vb.memory = "2048"
            vb.cpus = "2"
        end
    end

    config.vm.define "server" do |server|
        server.vm.box = "centos/7"
        server.vm.network "private_network", ip: "192.168.56.102", :name => 'vboxnet0', :adapter => 2
        server.vm.network "forwarded_port", guest: 4063, host: 4063, auto_correct: true
        server.vm.network "forwarded_port", guest: 4064, host: 4064, auto_correct: true
        
        server.vm.provider "virtualbox" do |vb|
            vb.name = "server"
            vb.memory = "20048"
            vb.cpus = "2"
        end
    end

	config.vm.provision "shell", inline: <<-SHELL
	  sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/g' /etc/ssh/sshd_config
	  echo 'MaxAuthTries 100' >> /etc/ssh/sshd_config    
	  systemctl restart sshd.service
          yum makecache fast && yum install -y /usr/bin/python /usr/bin/python2-config sudo yum-plugin-ovl bash iproute ca-certificates && sed -i 's/plugins=0/plugins=1/g' /etc/yum.conf && yum clean all
	SHELL
end

Then

vagrant up

This creates three VMs in VBox with IPs as specified in the Vagrantfile into which I can ssh such as

ssh  [email protected] #analogically for 101,102 

The IPs are in order: ...100 = web, ...101 = postgres, ...102 = server

Then I created a hosts.yml file:

[omero_server]
192.168.56.102

[omero_web]
192.168.56.100

[omero_database]
192.168.56.101

and ran the ansible command (my requirements are already satisfied) with this PR checked out:

ansible-playbook --ask-become -e ansible_ssh_user=vagrant  -i hosts.yml playbook.yml

which runs fine for the postgres VM (web is not reached, because it comes later in the playbook), then crashes on the omero_server task below (Edit: surprisingly, this error is first time only, this means if I rerun the playbook the second time not changing anything, this task is marked as green (i.e. OK, no changes needed) and the playbook crashes later, on the omero-server restart)

TASK [ome.omero_server : omero server | install requirements] ******************
changed: [192.168.56.102]

TASK [ome.omero_server : omero server | install omero] *************************
[WARNING]: Module remote_tmp /opt/omero/server/.ansible/tmp did not exist and
was created with a mode of 0700, this may cause issues when running as another
user. To avoid this, create the remote_tmp dir with the correct permissions
manually

fatal: [192.168.56.102]: FAILED! => {"changed": true, "cmd": ["/opt/omero/server/venv3/bin/omego", "install", "--release", "5.6.5", "--sym", "OMERO.server", "--ice", "3.6", "--no-start", "--no-web", "--ignoreconfig", "--omerocli", "/opt/omero/server/venv3/bin/omero", "-qq", "--dbhost", "10.0.2.15", "--dbuser", "omero", "--dbname", "omero", "--dbpass", "omero", "--managedb", "--rootpass", "omero"], "delta": "0:00:24.488595", "end": "2022-10-08 16:29:20.191931", "msg": "non-zero return code", "rc": 30, "start": "2022-10-08 16:28:55.703336", "stderr": "OMERO.py version:\nOMERO.server version:\n5.6.5-ice36-b233\n2022-10-08 16:29:20,170 [omego.extern] ERROR Failed [0.006 s]\n2022-10-08 16:29:20,171 [    omego.db] ERROR Non-zero return code\ncommand: psql -v ON_ERROR_STOP=on -d omero -h 10.0.2.15 -U omero -w -A -t -c \\conninfo\nreturn code: 2\nstdout: b''\nstderr: b'psql: could not connect to server: Connection refused\\n\\tIs the server running on host \"10.0.2.15\" and accepting\\n\\tTCP/IP connections on port 5432?\\n'", "stderr_lines": ["OMERO.py version:", "OMERO.server version:", "5.6.5-ice36-b233", "2022-10-08 16:29:20,170 [omego.extern] ERROR Failed [0.006 s]", "2022-10-08 16:29:20,171 [    omego.db] ERROR Non-zero return code", "command: psql -v ON_ERROR_STOP=on -d omero -h 10.0.2.15 -U omero -w -A -t -c \\conninfo", "return code: 2", "stdout: b''", "stderr: b'psql: could not connect to server: Connection refused\\n\\tIs the server running on host \"10.0.2.15\" and accepting\\n\\tTCP/IP connections on port 5432?\\n'"], "stdout": "5.12.1\nERROR: Database connection check failed", "stdout_lines": ["5.12.1", "ERROR: Database connection check failed"]}

This invites two questions:

1. Is my workflow correct, and mainly my port forwarding makes sense ?
2. Is the error caused by the present PR ?
3. Is the workflow such as above really testing the usage of ome.iptables_raw role, as I actually know my IPs upfront (I do not know any better how to call the VMs otherwise, happy to hear suggestions please) ?

Any ideas @drjrkuhn @joshmoore please ?

[pwalczysko]

Just to finish the review: Please react to #9 (comment) - no hurry on our side though of course :)