error creating pidfile: open /var/run/carbon-relay-ng.pid: read-only file system

[ljurk]

With the current master-image we got the errror error creating pidfile: open /var/run/carbon-relay-ng.pid: read-only file system and carbon-relay-ng was unable to start. Our container runs with podman and is marked as read_only, the solution was to add /var/run as a tmpfs so the container could write there(--mount type=tmpfs,destination=/var/run).
I suspect this error is related to using the distroless image(e216f17). Does anyone have a clue why the distroless-based image wants to write /var/run/carbon-relay-ng.pid and the alpine-based image didn't?
It's fine for us to add the mount, we are just curious what happened.

The container is controlled via systemd, this is our fixed unit file, which is created via ansibles podman_container-module:

[Unit]
Description=Podman container-carbon_relay.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStartPre=/bin/rm \
        -f %t/%n.ctr-id
ExecStart=/usr/bin/podman container run \
        --cidfile=%t/%n.ctr-id \
        --cgroups=no-conmon \
        --rm \
        --sdnotify=conmon \
        --replace \
        --name carbon_relay \
        --network podman-dns \
        --mount type=tmpfs,destination=/var/spool/carbon-relay-ng \
        --mount type=tmpfs,destination=/var/run \
        --read-only=True \
        --read-only-tmpfs=True \
        --cap-drop ALL \
        --security-opt no-new-privileges \
        --label io.containers.autoupdate=registry \
        --label PODMAN_SYSTEMD_UNIT=container-carbon_relay.service \
        --publish 2003:2003 \
        --publish 2004:2004 \
        --volume /opt/carbon_relay/conf.ini:/conf/carbon-relay-ng.ini \
        --volume /etc/timezone:/etc/timezone:ro \
        --volume /etc/localtime:/etc/localtime:ro \
        --detach=True docker.io/grafana/carbon-relay-ng:master
ExecStop=/usr/bin/podman stop \
        --ignore -t 10 \
        --cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm \
        -f \
        --ignore -t 10 \
        --cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all

[Install]
WantedBy=default.target

[jesusvazquez]

👋

First of all thanks for posting here the working configuration for you 👏

/var/run is typically a symbolic link to /run, here is an example

❯ docker run -it alpine  ls -l /var/run
lrwxrwxrwx    1 root     root             6 Dec  5 12:20 /var/run -> ../run

However /var/run has been for a long time the place to create the pid files. New linux systems are just writing them to /run and thats probably what Alpine was doing.

The new distroless image is based on debian and it must be writing the pid file to /var/run hence the issue you found.

Let me know if there is anything else you'd like to do in this issue.

[mztud]

Thank you for the explanation.
In the initial example the container is started with --read-only-tmpfs which mounts /run as a writeable tmpfs.
so even without the extra --mount option, this should not have bin a problem? 🤔

see https://docs.podman.io/en/latest/markdown/podman-run.1.html#read-only-tmpfs