Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

the acceld binary crash wholly when converting #334

Open
hopegi opened this issue Jan 9, 2025 · 0 comments
Open

the acceld binary crash wholly when converting #334

hopegi opened this issue Jan 9, 2025 · 0 comments

Comments

@hopegi
Copy link

hopegi commented Jan 9, 2025

the acceld binary run for long time,it may crash randomly during a conversion
i saw the log is

time="2025-01-09T05:36:38.207924977Z" level=info msg="POST /api/v1/conversions 200 3.308053ms 624>5bytes 33.0.3.18" module=api
time="2025-01-09T05:36:38.208076119Z" level=info msg="pulling image 1830447625108312066.crharborcore.inner:80/ascend/kubectl-detection:v2" module=converter
time="2025-01-09T05:36:38.518224815Z" level=info msg="pulled image 1830447625108312066.crharborcore.inner:80/ascend/kubectl-detection:v2 , elapse 307.898398ms" module=converter
time="2025-01-09T05:36:38.518254209Z" level=info msg="converting image 1830447625108312066.crharborcore.inner:80/ascend/kubectl-detection:v2" module=converter

then i curl the health check api, there is not respone return
although @imeoer tell me to add this code in main func

	sigChan := make(chan os.Signal, 1)
	go func() {
		for range sigChan {
			profilePath := "/tmp/buildkitd-goroutine.prof"
			file, err := os.Create(profilePath)
			if err != nil {
				logrus.Errorf("create goroutine profile failed: %v", err)
				continue
			}
			if err := pprof.Lookup("goroutine").WriteTo(file, 1); err != nil {
				logrus.Errorf("write goroutine profile failed: %v", err)
				file.Close()
				continue
			}
			file.Close()
			logrus.Infof("writed goroutine profile to: %s", profilePath)
		}
	}()
	signal.Notify(sigChan, syscall.SIGQUIT)

i exec kill -SIGQUIT $pid , there is not anything log output in /tmp,it seem the process can not receive Signal. so i think it may be crash wholly.
when it crash,there is any dir like this nydus-converter-1261992607 in /tmp
exec top in container is

Tasks: 5 total,   1 running,   4 sleeping,   0 stopped,   0 zombie
  Mem:   128399M total,    17136M used,   111263M free,        1M buffers
 Swap:        0M total,        0M used,        0M free,    11458M cached
3200%cpu  608%user    0%nice   38%sys 2534%idle    0%iow    9%irq    9%sirq    2%host
    PID USER         PR  NI VIRT  RES  SHR S[%CPU] %MEM     TIME+ ARGS                                                                                                
     25 root         20   0 687M  94M  19M S 98.3   0.0 344:53.64 acceld --config /misc/config/config.nydus.yaml
   4888 root         20   0 8.5M 4.1M 3.5M R  0.0   0.0   0:00.00 top
   4865 root         20   0 4.2M 3.6M 3.1M S  0.0   0.0   0:00.02 bash
      7 root         20   0 1.8G  59M  29M S  0.0   0.0   0:13.38 containerd
      1 root         20   0 4.0M 2.9M 2.6M S  0.0   0.0   0:00.00 bash /entrypoint.sh /misc/config/config.nydus.yaml

@imeoer tell me exec cat /proc/$pid/task/*/stack,similarly, no useful information was found .

root [ /proc/25/task ]# cat */stack
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] hrtimer_nanosleep+0x9b/0x140
[<0>] __se_sys_nanosleep_time32+0xa1/0xd0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7
[<0>] futex_wait_queue_me+0xb6/0x110
[<0>] futex_wait+0xe3/0x1f0
[<0>] do_futex+0x124/0x1f0
[<0>] __se_sys_futex_time32+0x13c/0x1c0
[<0>] do_int80_syscall_32+0x4b/0x90
[<0>] entry_INT80_compat+0xa2/0xa7

version info

go version
go version go1.21.5 linux/amd64
acceld version
0.2.13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@hopegi