Apps fail to migrate when upgrading to Electric Eel

[sync-by-unito]

Steps: I'm running 24.04.2.5. Three apps are installed & working correctly: syncthing, emby, and frigate. All three have been updated to the most recent version available. Next, I upgrade to 24.10.0.2.

Expected Result: After the upgrade, all three apps should have been migrated and are up and running.

Actual Result: Only one of my apps (frigate) migrated successfully. The other two (syncthing, emby) do not appear in the list of Applications after the upgrade. When looking at /var/log/app_migrations.log it's clear that there is some kind of issue when interpreting the config files:

% cat app_migrations.log
[2024/12/15 21:04:02] (DEBUG) app_migrations.migrate():235 - Migration details for 'system-update--2024-12-16_03:00:37' backup on 'isildur' pool
[2024/12/15 21:04:02] (DEBUG) app_migrations.migrate():241 - 'syncthing' app failed to migrate successfully: 'Failed to migrate config: Failed to migrate config: Traceback (most recent call last):\n File "/mnt/.ix-apps/truenas_catalog/trains/stable/syncthing/1.1.1/migrations/migrate_from_kubernetes", line 51, in \n print(yaml.dump(migrate(yaml.safe_load(f.read()))))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/mnt/.ix-apps/truenas_catalog/trains/stable/syncthing/1.1.1/migrations/migrate_from_kubernetes", line 19, in migrate\n "additional_envs": config["syncthingConfig"].get("additionalEnvs", []),\n ~~~~~~^^^^^^^^^^^^^^^^^^^\nKeyError: 'syncthingConfig'\n'
[2024/12/15 21:04:02] (DEBUG) app_migrations.migrate():241 - 'emby' app failed to migrate successfully: 'Failed to migrate config: Failed to migrate config: Traceback (most recent call last):\n File "/mnt/.ix-apps/truenas_catalog/trains/stable/emby/1.2.5/migrations/migrate_from_kubernetes", line 49, in \n print(yaml.dump(migrate(yaml.safe_load(f.read()))))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/mnt/.ix-apps/truenas_catalog/trains/stable/emby/1.2.5/migrations/migrate_from_kubernetes", line 18, in migrate\n "additional_envs": config["embyConfig"].get("additionalEnvs", []),\n ~~~~~~^^^^^^^^^^^^^^\nKeyError: 'embyConfig'\n'
[2024/12/15 21:04:02] (DEBUG) app_migrations.migrate():241 - 'frigate' app failed to migrate successfully: 'App with same name is already installed'

Session ID: 3f0904bb-7b91-1ffc-ffaf-42014ddfc8be

[sync-by-unito]

➤ Bug Clerk commented:

Thank you for submitting this TrueNAS Bug Report! So that we can quickly investigate your issue, please attach a Debug file and any other information related to this issue through our secure and private upload service below. Debug files can be generated in the UI by navigating to System > Advanced > Save Debug.

https://ixsystems.atlassian.net/servicedesk/customer/portal/15/group/37/create/153

[sync-by-unito]

➤ ShannonC commented:

I did check the “attach debug” box… do you still need a debug file?

[sync-by-unito]

➤ Bonnie Follweiler commented:

Good Morning ShannonC.

It did upload a debug file so there is no need to upload a second one. I have moved this ticket into our queue to review.

An engineering representative will update with any further questions or details in the near future.

[sync-by-unito]

➤ Stavros Kois commented:

Can you please clarify what versions was installed for Emby and Syncthing?
The errors you are getting suggests that Emby is at least 11 months old and Syncthing is at least 10 months old.

Thanks

[sync-by-unito]

➤ ShannonC commented:

Yes, copied & pasted info from the dashboard:

Emby:

App Version:
4.8.10.0
Chart Version:
2.0.13

Syncthing:

App Version:
1.28.1
Chart Version:
2.0.13

It looks to me like those are the right versions, I see 2.0.13 for both in here:

https://github.com/truenas/charts/tree/master/charts/emby ( https://github.com/truenas/charts/tree/master/charts/emby|smart-link )

https://github.com/truenas/charts/tree/master/charts/syncthing ( https://github.com/truenas/charts/tree/master/charts/syncthing|smart-link )

[sync-by-unito]

➤ Stavros Kois commented:

I suppose you are now booted on 24.04.

Can you run those commands on the truenas shell?

midclt call chart.release.get_instance emby | jq '.config'
midclt call chart.release.get_instance syncthing | jq '.config'Thanks

[sync-by-unito]

➤ ShannonC commented:

Here you go:

% midclt call chart.release.get_instance emby | jq '.config'
{
"embyConfig": {
"additionalEnvs": []
},
"embyGPU": {
"amd.com/gpu": 0,
"gpu.intel.com/i915": 0,
"nvidia.com/gpu": 0
},
"embyID": {
"group": 3001,
"user": 8675309
},
"embyNetwork": {
"hostNetwork": false,
"webPort": 9096
},
"embyStorage": {
"additionalStorages": [
{
"hostPathConfig": {
"aclEnable": false,
"hostPath": "/mnt/isildur/media/video"
},
"mountPath": "/srv/media/video",
"readOnly": true,
"type": "hostPath"
},
{
"hostPathConfig": {
"aclEnable": false,
"hostPath": "/mnt/isildur/media/music"
},
"mountPath": "/srv/media/music",
"readOnly": true,
"type": "hostPath"
}
],
"config": {
"ixVolumeConfig": {
"aclEnable": false,
"datasetName": "ix-emby_config"
},
"type": "ixVolume"
}
},
"global": {
"ixChartContext": {
"addNvidiaRuntimeClass": false,
"hasNFSCSI": true,
"hasSMBCSI": true,
"isInstall": false,
"isStopped": false,
"isUpdate": true,
"isUpgrade": false,
"kubernetes_config": {
"cluster_cidr": "172.16.0.0/16",
"cluster_dns_ip": "172.17.0.10",
"service_cidr": "172.17.0.0/16"
},
"nfsProvisioner": "nfs.csi.k8s.io",
"nvidiaRuntimeClassName": "nvidia",
"operation": "UPDATE",
"smbProvisioner": "smb.csi.k8s.io",
"storageClassName": "ix-storage-class-emby",
"upgradeMetadata": {}
}
},
"image": {
"pullPolicy": "IfNotPresent",
"repository": "emby/embyserver",
"tag": "4.8.10.0"
},
"ixCertificateAuthorities": {},
"ixCertificates": {},
"ixChartContext": {
"addNvidiaRuntimeClass": false,
"hasNFSCSI": true,
"hasSMBCSI": true,
"isInstall": false,
"isStopped": false,
"isUpdate": true,
"isUpgrade": false,
"kubernetes_config": {
"cluster_cidr": "172.16.0.0/16",
"cluster_dns_ip": "172.17.0.10",
"service_cidr": "172.17.0.0/16"
},
"nfsProvisioner": "nfs.csi.k8s.io",
"nvidiaRuntimeClassName": "nvidia",
"operation": "UPDATE",
"smbProvisioner": "smb.csi.k8s.io",
"storageClassName": "ix-storage-class-emby",
"upgradeMetadata": {}
},
"ixExternalInterfacesConfiguration": [],
"ixExternalInterfacesConfigurationNames": [],
"ixVolumes": [
{
"hostPath": "/mnt/isildur/ix-applications/releases/emby/volumes/ix_volumes/ix-emby_config"
}
],
"release_name": "emby",
"resources": {
"limits": {
"cpu": "2",
"memory": "2Gi"
}
}
}

% midclt call chart.release.get_instance syncthing | jq '.config'
{
"global": {
"ixChartContext": {
"addNvidiaRuntimeClass": false,
"hasNFSCSI": true,
"hasSMBCSI": true,
"isInstall": false,
"isStopped": false,
"isUpdate": false,
"isUpgrade": true,
"kubernetes_config": {
"cluster_cidr": "172.16.0.0/16",
"cluster_dns_ip": "172.17.0.10",
"service_cidr": "172.17.0.0/16"
},
"nfsProvisioner": "nfs.csi.k8s.io",
"nvidiaRuntimeClassName": "nvidia",
"operation": "UPGRADE",
"smbProvisioner": "smb.csi.k8s.io",
"storageClassName": "ix-storage-class-syncthing",
"upgradeMetadata": {
"newChartVersion": "2.0.13",
"oldChartVersion": "2.0.12",
"preUpgradeRevision": 13
}
}
},
"image": {
"pullPolicy": "IfNotPresent",
"repository": "syncthing/syncthing",
"tag": "1.28.1"
},
"ixCertificateAuthorities": {},
"ixCertificates": {},
"ixChartContext": {
"addNvidiaRuntimeClass": false,
"hasNFSCSI": true,
"hasSMBCSI": true,
"isInstall": false,
"isStopped": false,
"isUpdate": false,
"isUpgrade": true,
"kubernetes_config": {
"cluster_cidr": "172.16.0.0/16",
"cluster_dns_ip": "172.17.0.10",
"service_cidr": "172.17.0.0/16"
},
"nfsProvisioner": "nfs.csi.k8s.io",
"nvidiaRuntimeClassName": "nvidia",
"operation": "UPGRADE",
"smbProvisioner": "smb.csi.k8s.io",
"storageClassName": "ix-storage-class-syncthing",
"upgradeMetadata": {
"newChartVersion": "2.0.13",
"oldChartVersion": "2.0.12",
"preUpgradeRevision": 13
}
},
"ixExternalInterfacesConfiguration": [],
"ixExternalInterfacesConfigurationNames": [],
"ixVolumes": [
{
"hostPath": "/mnt/isildur/ix-applications/releases/syncthing/volumes/ix_volumes/ix-syncthing_config"
}
],
"podOptions": {
"dnsConfig": {
"options": []
}
},
"release_name": "syncthing",
"resources": {
"limits": {
"cpu": "2000m",
"memory": "3Gi"
}
},
"syncthingConfig": {
"additionalEnvs": []
},
"syncthingID": {
"group": 3002,
"user": 3002
},
"syncthingNetwork": {
"hostNetwork": false,
"tcpPort": 20978,
"udpPort": 20979,
"webPort": 20910
},
"syncthingStorage": {
"additionalStorages": [
{
"hostPathConfig": {
"aclEnable": false,
"hostPath": "/mnt/isildur/backup/syncthing/carey"
},
"mountPath": "/srv/carey",
"readOnly": false,
"type": "hostPath"
},
{
"hostPathConfig": {
"aclEnable": false,
"hostPath": "/mnt/isildur/backup/syncthing/schuyler"
},
"mountPath": "/srv/schuyler",
"readOnly": false,
"type": "hostPath"
}
],
"config": {
"ixVolumeConfig": {
"aclEnable": false,
"datasetName": "ix-syncthing_config"
},
"type": "ixVolume"
}
}
}

[sync-by-unito]

➤ Stavros Kois commented:

Hello ShannonC , logs suggest that it tries to use the system-update--2024-12-16_03:00:37 backup, which also seems to be the latest.
This backup seems to contain older versions. Did you upgrade the apps after the backup?

[sync-by-unito]

➤ ShannonC commented:

I upgraded the apps before performing the upgrade, per the instructions. So, if that backup was auto-created during the upgrade, then I am not sure why it would contain older versions. Also, if that was the case then wouldn’t I have old app versions after reverting to the old boot environment? But that did not happen either.

It’s unclear what time zone that date is in, and the “Boot Environments” UI does not show a time zone either… but if the UI shows local time, my 24.10.0.2 boot environment was created at 2024-12-15 14:58:00 UTC 5… so I am not sure where that backup timestamp is coming into play. It’s possible I tried reverting the upgrade and reediting the app configs once again to see if that would make a difference, before trying to upgrade again, but I don’t recall now.

It looks like another update is available for Syncthing. Perhaps I will update that and give the upgrade another try.

[sync-by-unito]

➤ ShannonC commented:

Ok, weird… here’s what happened:

1. Deleted the Electric Eel boot environment created by the previous upgrade

2. Updated Syncthing

3. Clicked “Edit” on each app, one by one.
   1. made no changes to any of the config
   2. clicked save (or update or whatever)
   3. each one re-deployed even though I didn’t make any changes

4. Upgraded to Electric Eel again

This time, Syncthing shows up in the list… but Emby still does not.

Contents of /var/log/app_migrations.log:

[2025/01/08 21:06:46] (DEBUG) app_migrations.migrate():250 - Migration details for 'system-update--2025-01-09_03:01:38' backup on 'isildur' pool
[2025/01/08 21:06:46] (DEBUG) app_migrations.migrate():253 - 'syncthing' app migrated successfully
[2025/01/08 21:06:46] (DEBUG) app_migrations.migrate():256 - 'emby' app failed to migrate successfully: 'Failed to migrate config: Failed to migrate config: Traceback (most recent call last):\n File "/mnt/.ix-apps/truenas_catalog/trains/stable/emby/1.2.11/migrations/migrate_from_kubernetes", line 49, in \n print(yaml.dump(migrate(yaml.safe_load(f.read()))))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/mnt/.ix-apps/truenas_catalog/trains/stable/emby/1.2.11/migrations/migrate_from_kubernetes", line 18, in migrate\n "additional_envs": config["embyConfig"].get("additionalEnvs", []),\n ~~~~~~^^^^^^^^^^^^^^\nKeyError: 'embyConfig'\n'
[2025/01/08 21:06:46] (DEBUG) app_migrations.migrate():256 - 'frigate' app failed to migrate successfully: 'App with same name is already installed'Certainly seems that something’s not quite right… Apparently it is capable of working correctly, but something is preventing it…

[sync-by-unito]

➤ Stavros Kois commented:

ShannonC
Can you jump into shell in dragonfish and run this? What’s the output?

kubectl get secrets --all-namespaces | grep "sh.helm.release