Pi 3B Bullseye: Anthias update hangs at "TASK [system : Remove libc6-dev]"

[traeu]

Overview of the Issue

I use multiple Anthias-installations. Normally, I don't update them regularly when everything works.
Recently, I tried to update two of my screens with latest anthias, but I experience problems.
My Pis run on Raspberry Pi OS Bullseye, all apt updates are installed.
I tried to start the update-process by running the install-script again, but the script freezes at "TASK [system : Remove libc6-dev]". While it freezes, I noticed that the system is lagging in general, for example, another ssh-session responds only delayed while the install-script runs in another ssh-session.
A bit strange:
Last thursday, I only got this problem on one of my Pis, not on both. Both run bullseye, both had all apt updates made. But one would run through this libc6-dev task quickly, while the other one hangs at it.
I thought, I will try my luck again on monday. Now today, I try to run the install-scripts on both Pis again, and now on both Pis the install-script freezes at this libc6 remove task.

Reproduction Steps

• Have a running anthias installation on debian bullseye that was not upgraded for some time (date of last update unfortunately unknown)
• install all system updates: sudo apt update, sudo apt upgrade
• run the anthias-installscript again: bash <(curl -sL https://install-anthias.srly.io)
• Options for installing: Yes, No, Yes (I don't want to use network management because with it enabled, distributing DNS-server over DHCP did not work for me)
• it freezes at task [system : Remove libc6-dev]

Environment

• Raspberry Pi Hardware Version: Raspberry Pi 3B, Raspberry Pi OS Bullseye 64 bit
• Raspberry Pi Network Setup: Ethernet
• Anthias Version: unknown. Was not updated for some time, now I cannot complete the update process. I don't know if parts of the update succeeded and what version is running exactly now.

[nicomiguelino]

@traeu, I'll look into this issue closely. Is your device still operational even after you halt the update.
In the meantime, if it's still working, you can download backup files via the web UI (Settings > Get Backup).

[nicomiguelino]

@traeu I tried reproducing the issue, tweaked things so that I can install an older version of Anthias (48e427b to be specific). The installation went fine. Then I ran bash <(curl -sL https://install-anthias.srly.io/). Mine was not stuck at the libc6-dev part. It could be that yours was way older.

If it wouldn't be a hassle on your part, you can backup the data first, then re-install Anthias.
Alternatively, you can comment out those steps (see https://github.com/Screenly/Anthias/blob/master/ansible/roles/system/tasks/main.yml#L180-L191 for details) in the Pi, then run the installer script again.

[traeu]

hello @nicomiguelino, thank you for your quick response!
Anthias seems to be operational after the script hangs and I forced it to quit with ctrl-c. I also tried to run the update-container script after my failed upgrade-attempt, that also worked successfully.
I also tried to install/re-install and also remove this libc6-dev package with apt before running the script. But that made no difference.

Is there a way how I can make a fresh Anthias installation and remove an existing installation, without reinstalling the whole Raspberry Pi OS?
Wiping the SD card and doing a clean install would be my next option to try, I was hoping that I can avoid it...
But no big deal, especially as this looks like a problem on my side and not like a common problem.

What would be the best way to modify the main.yml locally before executing the installer script? Can I somehow pause the installer script after it downloaded all files, then modify it, and then resume the install-script?

Thanks for your help!

[nicomiguelino]

@traeu I have only tried the following once yesterday:

• As mentioned previously, I commented the following (higlighted) in https://github.com/Screenly/Anthias/blob/master/ansible/roles/system/tasks/main.yml#L180-L191:

Anthias/ansible/roles/system/tasks/main.yml

Lines 180 to 191 in 5bb4123

	- name: Remove libc6-dev
	ansible.builtin.apt:
	name: libc6-dev
	state: absent
	when: not cdefs_exist
	- name: Install libc6-dev
	ansible.builtin.apt:
	name: libc6-dev
	state: present
	update_cache: true
	when: not cdefs_exist

• You can SSH to your Pi and edit the file using the text editor of your choice (e.g., vim, nano VS Code via remote connection).
• After editing, you can run the install script again.

I hope this helps.

[traeu]

Today I choose the easiest solution and wiped all SD cards, re-installed Raspberry Pi OS (legacy/bullseye 64bit lite) and made fresh installs of Anthias. As expected, everything works perfectly.
Before wiping, I tried upgrading one of my other Pis. Installed apt updates and re-run the anthias install script. This also worked without problems. That must mean, that two of my Pis had some other problems, and in general, upgrading works also for older installations. But while I was at it, I didn't want to risk anything and I made fresh installs on all Pis, even if upgrading may would have worked perfectly for all my Pis except for the two that made trouble with this libc6 package.

I guess I can close this issue now, it was clearly some problem on my side, even if I will never know why exactly it failed...

Thanks again for your quick help and your work on this amazing software!

Edit: I also tried to comment out the removal and installation of libc6-dev in main.yml.
After that, installation/upgrading also worked, the script was able to run until it was done, no freezes.
But if I enabled this removal and installation of libc6-dev again in main.yml after a first successful run while it was commented out, the same problems appeared again.
Because I just didn't want to permanently use modified local files that are not in sync with your files on github, I choose to made fresh installs of everything.

[kroam]

Hey @traeu what file did you edit to comment out the libc6-dev lines?
I have run the bash <(curl -sL https://install-anthias.srly.io/) on my Raspberry PI Zero 2 but it failed at ["TASK [system : Remove libc6-dev]. I then edited /home/pi/screenly/ansible/roles/system/tasks/main.yml and commented out the lines for removing and installing libc6-dev and saved the file.
Then i ran ~/screenly/bin/install.sh but the installation still hangs at remove libc6-dev also after running the install.sh the file main.yml is back to its original state.

[nicomiguelino]

@kroam, you can comment out the following lines inside ansible/roles/system/tasks/main.yml:

Anthias/ansible/roles/system/tasks/main.yml

Lines 180 to 191 in 5bb4123

	- name: Remove libc6-dev
	ansible.builtin.apt:
	name: libc6-dev
	state: absent
	when: not cdefs_exist
	- name: Install libc6-dev
	ansible.builtin.apt:
	name: libc6-dev
	state: present
	update_cache: true
	when: not cdefs_exist

[traeu]

@kroam later, I found out by accident what the cause of my problem was. I used the option "force_turbo=1" in config.txt, because only with this I could get 4K output to work on my Raspi 3B. But this unofficial hack seems to come with unexpected downsides. Thats why after reinstalling (and using stock config.txt at first) the problem was gone and installation was successful. Later, after I tested 4K again and noticed that the problem reappeared, I found out that this turbo-setting was causing the installation process to hang at this remove libc6-dev.

[nicomiguelino]

@traeu, does removing force_turbo=1 fixes the issue where you're stuck at removing libc6-dev?

[traeu]

@nicomiguelino yes for me this worked. As soon as I commented out this line in the config file (and rebooted of course), the installation process did not hang anymore at this step of removing libc6-dev.

[nicomiguelino]

@traeu, @kroam, thanks. I'm still testing the upgrade process, given the details that you provided. Don't hesitate to fork Anthias and create a pull request if you feel like it and are comfortable with the changes.

[kroam]

On my Raspberry PI Zero 2 i don't use the "force_turbo=1" in config.txt. its a stock Raspberry Pi OS 64bit Bookworm

[kroam]

Maybe a dumb question but when i git clone anthias to my pi and change the main.yml to comment out the lines for removing and installing libc6-dev, then run the /bin/install.sh file, the installer still hangs at remove libc6-dev.
Am i missing something?

[nicomiguelino]

@kroam, no, it's not a dumb question. Glad you asked.
The script always pulls the latest changes from the master branch, causing your changes to be overwritten.
Here's what you can do:

• Go to the ~/screenly directory.
• Modify ./bin/install.sh. Take a look at the following snippet:
  

  Anthias/bin/install.sh

  Lines 192 to 194 in e3cfeb1

  	sudo -u ${USER} ${SUDO_ARGS[@]} ansible localhost \
  	-m git \
  	-a "repo=$REPOSITORY dest=${ANTHIAS_REPO_DIR} version=${BRANCH} force=yes"

• You can either:
  • Set force to no instead; or
  • Just remove/comment out those lines.

I hope it helps. If not, please let me know.