Openstack Gitlab Runner Setup

We're now setting up a set of Gitlab runners for vg on the GI's Openstack local cloud.

To make a new runner:

• Make sure you have an SSH key set up in Openstack
• Get Openstack YAML file from http://gicloud.prism/dashboard/project/api_access/
• Put at ~/config/openstack/clouds.yaml
• Edit in your Openstack password under auth as a password key, since it can't remember a cookie or anything.
• Install the Openstack CLI:

pip install --user python-openstackclient

• Deploy a runner VM and connect to it:

SSH_KEY_NAME=anovak-swords
SERVER_NAME=anovak-gitlab-runner-3

openstack --os-cloud openstack server create --image ubuntu-22.04-LTS-x86_64 --flavor m1.huge --key-name ${SSH_KEY_NAME} --wait ${SERVER_NAME}
while true ; do
    IP_ID=$(openstack --os-cloud openstack floating ip list --long --status DOWN --network ext-net --format value --column ID | head -n1)
    while [[ "${IP_ID}" == "" ]] ; do
        openstack --os-cloud openstack floating ip create ext-net
        IP_ID=$(openstack --os-cloud openstack floating ip list --long --status DOWN --network ext-net --format value --column ID | head -n1)
    done
    openstack --os-cloud openstack server add floating ip ${SERVER_NAME} ${IP_ID} || continue
    break
done
INSTANCE_IP="$(openstack --os-cloud openstack floating ip show ${IP_ID} --column floating_ip_address --format value)"
sleep 60
ssh ubuntu@${INSTANCE_IP}

• On the VM, become root:

sudo su -

• Set up the machinery to keep Docker images and scratch space on the instance's ephemeral storage that mounts at /mnt (this code is taken form Toil and would be Apache licensed):

cat >/usr/local/bin/ephemeral-setup.sh <<'EOF'
#!/bin/bash
set -x
directories=(lib/docker)
for directory in "${directories[@]}"; do
    mkdir -p /mnt/ephemeral/var/$directory
    mkdir -p /var/$directory
    mount --bind /mnt/ephemeral/var/$directory /var/$directory
done
EOF
cat >/etc/systemd/system/ephemeral-setup.service <<'EOF'
[Unit]
Description=bind mounts ephemeral directories
Before=docker.service

[Service]
Type=oneshot
Restart=no
ExecStart=/usr/bin/bash /usr/local/bin/ephemeral-setup.sh

[Install]
WantedBy=multi-user.target
EOF

systemctl start ephemeral-setup.service

• Go to https://ucsc-ci.com/groups/vgteam/-/runners/new and make a new runner that is paused and runs untagged jobs. Name it so you can match it up with the VM you made.
• Set the runner token in your environment on the server:

RUNNER_TOKEN=!!!PASTE!TOKEN!HERE!!!

• Set up the Gitlab runner on the server:

sudo bash -c "export DEBIAN_FRONTEND=noninteractive; curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | bash"
sudo DEBIAN_FRONTEND=noninteractive apt update && sudo DEBIAN_FRONTEND=noninteractive apt upgrade -y
sudo DEBIAN_FRONTEND=noninteractive apt install -y docker.io gitlab-runner
sudo gitlab-runner register --non-interactive --url https://ucsc-ci.com --token $RUNNER_TOKEN --limit 4 --executor docker --docker-privileged --docker-memory 25G --docker-image docker:dind
sudo sed -i 's/concurrent = 1/concurrent = 4/g' /etc/gitlab-runner/config.toml
sudo gitlab-runner restart