open /mnt/win_milvus/backup_20241223/binlogs/insert_log/446606056582023021/446606056582023215/452175458784407801/452175458784407801/104/452175458784407885: too many open files

[luliloveshe918]

I mounted a Windows shared directory on Linux at /mnt/win_milvus. When performing the backup, an error occurs. After changing the path to a local directory, the backup was successful. The configuration and error are shown in the image:

[huanghaoyuanhhy]

Thank you for reporting this issue!

Please check your current limit by running: ulimit -n

[luliloveshe918]

default value 1024

[huanghaoyuanhhy]

Thank you for the update!
A limit of 1024 is quite low for operations like this. Please increase it to 65535 temporarily by running ulimit -n 65535. Then, try running the backup again.

[luliloveshe918]

Thank you, I'll give it a try. Thanks a lot!

[luliloveshe918]

The issue still persists.

[2024/12/28 01:00:56.958 +08:00] [INFO] [core/backup_impl_create_backup.go:800] ["Fail to copy file after retry"] [collection_id=446606056582023021] [partition_id=452175458812479328] [segment_id=452175458836023781] [group_id=31] [error="All attempts results:\nattempt #1:open /mnt/win_milvus/backup_20241228/binlogs/insert_log/446606056582023021/452175458812479328/452175458836023781/452175458836023781/101/452175458836023783: too many open files\nattempt #2:open /mnt/win_milvus/backup_20241228/binlogs/insert_log/446606056582023021/452175458812479328/452175458836023781/452175458836023781/101/452175458836023783: too many open files\nattempt #3:context canceled\n"] [from=files/insert_log/446606056582023021/452175458812479328/452175458836023781/101/452175458836023783] [to=/mnt/win_milvus/backup_20241228/binlogs/insert_log/446606056582023021/452175458812479328/452175458836023781/452175458836023781/101/452175458836023783]
[2024/12/28 01:00:56.958 +08:00] [INFO] [core/backup_impl_create_backup.go:800] ["Fail to copy file after retry"] [collection_id=446606056582023021] [partition_id=452175458812479328] [segment_id=454052558199721633] [group_id=39] [error="All attempts results:\nattempt #1:open /mnt/win_milvus/backup_20241228/binlogs/insert_log/446606056582023021/452175458812479328/454052558199721633/454052558199721633/101/454052558199721635: too many open files\nattempt #2:open /mnt/win_milvus/backup_20241228/binlogs/insert_log/446606056582023021/452175458812479328/454052558199721633/454052558199721633/101/454052558199721635: too many open files\nattempt #3:context canceled\n"] [from=files/insert_log/446606056582023021/452175458812479328/454052558199721633/101/454052558199721635] [to=/mnt/win_milvus/backup_20241228/binlogs/insert_log/446606056582023021/452175458812479328/454052558199721633/454052558199721633/101/454052558199721635]
[2024/12/28 01:00:56.959 +08:00] [INFO] [core/backup_impl_create_backup.go:800] ["Fail to copy file after retry"] [collection_id=446606056582023021] [partition_id=452175458812479328] [segment_id=452175458836032977] [group_id=89] [error="All attempts results:\nattempt #1:open /mnt/win_milvus/backup_20241228/binlogs/insert_log/446606056582023021/452175458812479328/452175458836032977/452175458836032977/101/452175458836032993: too many open files\nattempt #2:open /mnt/win_milvus/backup_20241228/binlogs/insert_log/446606056582023021/452175458812479328/452175458836032977/452175458836032977/101/452175458836032993: too many open files\nattempt #3:open /mnt/win_milvus/backup_20241228/binlogs/insert_log/446606056582023021/452175458812479328/452175458836032977/452175458836032977/101/452175458836032993: too many open files\nattempt #4:open /mnt/win_milvus/backup_20241228/binlogs/insert_log/446606056582023021/452175458812479328/452175458836032977/452175458836032977/101/452175458836032993: too many open files\nattempt #5:open /mnt/win_milvus/backup_20241228/binlogs/insert_log/446606056582023021/452175458812479328/452175458836032977/452175458836032977/101/452175458836032993: too many open files\nattempt #6:context canceled\n"] [from=files/insert_log/446606056582023021/452175458812479328/452175458836032977/101/452175458836032993] [to=/mnt/win_milvus/backup_20241228/binlogs/insert_log/446606056582023021/452175458812479328/452175458836032977/452175458836032977/101/452175458836032993]
[2024/12/28 01:00:56.959 +08:00] [INFO] [core/backup_impl_create_backup.go:645] ["Resume Milvus GC response"] [response="404 page not found"] [address=http://localhost:9091]

[luliloveshe918]

添加的语句 ulimit -n 65535

[huanghaoyuanhhy]

Thank you for the update!

Please refer to this guide for detailed instructions on changing the open files limit in Linux:
https://stackoverflow.com/questions/34588/how-do-i-change-the-number-of-open-files-limit-in-linux