Add load testing scripts, profiler and data race fixes #1441

bkneis · 2024-12-03T10:41:44Z

This PR adds scripts to aid load testing and an optional golang profiler to build using a tag.

Additionally I built devpod with golang's race detector and did some testing with up and ssh and found two data races (trace below). One fix was to move the closing of the writer and the other was reusing an err var.

Data race for up

10:44:37 info ==================
10:44:37 info WARNING: DATA RACE
10:44:37 info Write at 0x00c000a00f10 by main goroutine:
10:44:37 info   internal/poll.(*FD).destroy()
10:44:37 info       /usr/lib/go/src/internal/poll/fd_unix.go:84 +0xd5
10:44:37 info   internal/poll.(*FD).decref()
10:44:37 info       /usr/lib/go/src/internal/poll/fd_mutex.go:213 +0x36
10:44:37 info   internal/poll.(*FD).Close()
10:44:37 info       /usr/lib/go/src/internal/poll/fd_unix.go:105 +0x7d
10:44:37 info   os.(*file).close()
10:44:37 info       /usr/lib/go/src/os/file_unix.go:333 +0xe4
10:44:37 info   os.(*File).Close()
10:44:37 debug done injecting
10:44:37 info       /usr/lib/go/src/os/file_posix.go:23 +0x45
10:44:37 info   github.com/loft-sh/devpod/pkg/inject.InjectAndExecute.deferwrap3()
10:44:37 info       /home/arthur/go/src/devpod/pkg/inject/inject.go:65 +0x17
10:44:37 info   runtime.deferreturn()
10:44:37 info       /usr/lib/go/src/runtime/panic.go:605 +0x5d
10:44:37 info   github.com/loft-sh/devpod/pkg/agent.InjectAgentAndExecute()
10:44:37 info       /home/arthur/go/src/devpod/pkg/agent/inject.go:111 +0x976
10:44:37 info   github.com/loft-sh/devpod/pkg/agent.InjectAgent()
10:44:37 info       /home/arthur/go/src/devpod/pkg/agent/inject.go:33 +0x1a4
10:44:37 info   github.com/loft-sh/devpod/pkg/devcontainer.(*runner).setupContainer()
10:44:37 info       /home/arthur/go/src/devpod/pkg/devcontainer/setup.go:36 +0x95
10:44:37 info   github.com/loft-sh/devpod/pkg/devcontainer.(*runner).runSingleContainer()
10:44:37 info       /home/arthur/go/src/devpod/pkg/devcontainer/single.go:127 +0xc4d
10:44:37 info   github.com/loft-sh/devpod/pkg/devcontainer.(*runner).Up()
10:44:37 info       /home/arthur/go/src/devpod/pkg/devcontainer/run.go:113 +0x533
10:44:37 info   github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).devPodUp()
10:44:37 info       /home/arthur/go/src/devpod/cmd/agent/workspace/up.go:128 +0x1f2
10:44:37 info   github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).up()
10:44:37 info       /home/arthur/go/src/devpod/cmd/agent/workspace/up.go:103 +0x93
10:44:37 info   github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).Run()
10:44:37 info       /home/arthur/go/src/devpod/cmd/agent/workspace/up.go:93 +0x517
10:44:37 info   github.com/loft-sh/devpod/cmd/agent/workspace.NewUpCmd.func1()
10:44:37 info       /home/arthur/go/src/devpod/cmd/agent/workspace/up.go:51 +0x3c
10:44:37 info   github.com/spf13/cobra.(*Command).execute()
10:44:37 info       /home/arthur/go/src/devpod/vendor/github.com/spf13/cobra/command.go:985 +0x10f3
10:44:37 debug Done InjectAgentAndExecute
10:44:37 info   github.com/spf13/cobra.(*Command).ExecuteC()
10:44:37 info       /home/arthur/go/src/devpod/vendor/github.com/spf13/cobra/command.go:1117 +0x655
10:44:37 info   github.com/spf13/cobra.(*Command).Execute()
10:44:37 info       /home/arthur/go/src/devpod/vendor/github.com/spf13/cobra/command.go:1041 +0x5a
10:44:37 info   github.com/loft-sh/devpod/cmd.Execute()
10:44:37 info       /home/arthur/go/src/devpod/cmd/root.go:88 +0x55
10:44:37 info   main.main()
10:44:37 info       /home/arthur/go/src/devpod/main.go:6 +0x1c
10:44:37 info   runtime.main()
10:44:37 info       /usr/lib/go/src/runtime/proc.go:272 +0x28a
10:44:37 info 
10:44:37 info Previous read at 0x00c000a00f10 by goroutine 178:
10:44:37 info   os.(*File).Fd()
10:44:37 info       /usr/lib/go/src/os/file_unix.go:94 +0x539
10:44:37 info   os.startProcess()
10:44:37 info       /usr/lib/go/src/os/exec_posix.go:51 +0x3ed
10:44:37 info   os.StartProcess()
10:44:37 info       /usr/lib/go/src/os/exec.go:319 +0x71
10:44:37 info   os/exec.(*Cmd).Start()
10:44:37 info       /usr/lib/go/src/os/exec/exec.go:709 +0xac4
10:44:37 info   os/exec.(*Cmd).Run()
10:44:37 info       /usr/lib/go/src/os/exec/exec.go:607 +0x26
10:44:37 info   github.com/loft-sh/devpod/pkg/docker.(*DockerHelper).RunWithDir()
10:44:37 info       /home/arthur/go/src/devpod/pkg/docker/helper.go:154 +0x1b8
10:44:37 info   github.com/loft-sh/devpod/pkg/docker.(*DockerHelper).Run()
10:44:37 info       /home/arthur/go/src/devpod/pkg/docker/helper.go:145 +0x54d
10:44:37 info   github.com/loft-sh/devpod/pkg/driver/docker.(*dockerDriver).CommandDevContainer()
10:44:37 info       /home/arthur/go/src/devpod/pkg/driver/docker/docker.go:90 +0x4b1
10:44:37 info   github.com/loft-sh/devpod/pkg/devcontainer.(*runner).setupContainer.func1()
10:44:37 info       /home/arthur/go/src/devpod/pkg/devcontainer/setup.go:37 +0x14f
10:44:37 info   github.com/loft-sh/devpod/pkg/inject.InjectAndExecute.func1()
10:44:37 info       /home/arthur/go/src/devpod/pkg/inject/inject.go:88 +0x18a
10:44:37 info 
10:44:37 info Goroutine 178 (running) created at:
10:44:37 info   github.com/loft-sh/devpod/pkg/inject.InjectAndExecute()
10:44:37 info       /home/arthur/go/src/devpod/pkg/inject/inject.go:83 +0x7b5
10:44:37 info   github.com/loft-sh/devpod/pkg/agent.InjectAgentAndExecute()
10:44:37 info       /home/arthur/go/src/devpod/pkg/agent/inject.go:111 +0x976
10:44:37 info   github.com/loft-sh/devpod/pkg/agent.InjectAgent()
10:44:37 info       /home/arthur/go/src/devpod/pkg/agent/inject.go:33 +0x1a4
10:44:37 info   github.com/loft-sh/devpod/pkg/devcontainer.(*runner).setupContainer()
10:44:37 info       /home/arthur/go/src/devpod/pkg/devcontainer/setup.go:36 +0x95
10:44:37 info   github.com/loft-sh/devpod/pkg/devcontainer.(*runner).runSingleContainer()
10:44:37 info       /home/arthur/go/src/devpod/pkg/devcontainer/single.go:127 +0xc4d
10:44:37 info   github.com/loft-sh/devpod/pkg/devcontainer.(*runner).Up()
10:44:37 info       /home/arthur/go/src/devpod/pkg/devcontainer/run.go:113 +0x533
10:44:37 info   github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).devPodUp()
10:44:37 info       /home/arthur/go/src/devpod/cmd/agent/workspace/up.go:128 +0x1f2
10:44:37 info   github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).up()
10:44:37 info       /home/arthur/go/src/devpod/cmd/agent/workspace/up.go:103 +0x93
10:44:37 info   github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).Run()
10:44:37 info       /home/arthur/go/src/devpod/cmd/agent/workspace/up.go:93 +0x517
10:44:37 info   github.com/loft-sh/devpod/cmd/agent/workspace.NewUpCmd.func1()
10:44:37 info       /home/arthur/go/src/devpod/cmd/agent/workspace/up.go:51 +0x3c
10:44:37 info   github.com/spf13/cobra.(*Command).execute()
10:44:37 info       /home/arthur/go/src/devpod/vendor/github.com/spf13/cobra/command.go:985 +0x10f3
10:44:37 info   github.com/spf13/cobra.(*Command).ExecuteC()
10:44:37 info       /home/arthur/go/src/devpod/vendor/github.com/spf13/cobra/command.go:1117 +0x655
10:44:37 info   github.com/spf13/cobra.(*Command).Execute()
10:44:37 info       /home/arthur/go/src/devpod/vendor/github.com/spf13/cobra/command.go:1041 +0x5a
10:44:37 info   github.com/loft-sh/devpod/cmd.Execute()
10:44:37 info       /home/arthur/go/src/devpod/cmd/root.go:88 +0x55
10:44:37 info   main.main()
10:44:37 info       /home/arthur/go/src/devpod/main.go:6 +0x1c
10:44:37 info   runtime.main()
10:44:37 info       /usr/lib/go/src/runtime/proc.go:272 +0x28a
10:44:37 info ==================

Data race for ssh

==================
WARNING: DATA RACE
Write at 0x00c000a9c6e0 by goroutine 147:
  github.com/loft-sh/devpod/pkg/tunnel.RunInContainer.func2()
      /home/arthur/go/src/devpod/pkg/tunnel/services.go:136 +0xf98
  k8s.io/client-go/util/retry.OnError.func1()
      /home/arthur/go/src/devpod/vendor/k8s.io/client-go/util/retry/util.go:51 +0x41
  k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection()
      /home/arthur/go/src/devpod/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:145 +0x53
  k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff()
      /home/arthur/go/src/devpod/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:461 +0x5e
  k8s.io/client-go/util/retry.OnError()
      /home/arthur/go/src/devpod/vendor/k8s.io/client-go/util/retry/util.go:50 +0x130
  github.com/loft-sh/devpod/pkg/tunnel.RunInContainer()
      /home/arthur/go/src/devpod/pkg/tunnel/services.go:59 +0x3eb
  github.com/loft-sh/devpod/cmd.(*SSHCmd).startServices()
      /home/arthur/go/src/devpod/cmd/ssh.go:484 +0x144
  github.com/loft-sh/devpod/cmd.(*SSHCmd).startTunnel.gowrap1()
      /home/arthur/go/src/devpod/cmd/ssh.go:402 +0x104

Previous write at 0x00c000a9c6e0 by goroutine 162:
  github.com/loft-sh/devpod/pkg/tunnel.RunInContainer.func2.1()
      /home/arthur/go/src/devpod/pkg/tunnel/services.go:95 +0x324

Goroutine 147 (running) created at:
  github.com/loft-sh/devpod/cmd.(*SSHCmd).startTunnel()
      /home/arthur/go/src/devpod/cmd/ssh.go:402 +0x3e4
  github.com/loft-sh/devpod/cmd.(*SSHCmd).jumpContainer.func1()
      /home/arthur/go/src/devpod/cmd/ssh.go:281 +0x144
  github.com/loft-sh/devpod/pkg/tunnel.(*ContainerHandler).runRunInContainer()
      /home/arthur/go/src/devpod/pkg/tunnel/container.go:230 +0x8c1
  github.com/loft-sh/devpod/pkg/tunnel.(*ContainerHandler).Run.func2()
      /home/arthur/go/src/devpod/pkg/tunnel/container.go:121 +0x4a9

Goroutine 162 (finished) created at:
  github.com/loft-sh/devpod/pkg/tunnel.RunInContainer.func2()
      /home/arthur/go/src/devpod/pkg/tunnel/services.go:90 +0x9d4
  k8s.io/client-go/util/retry.OnError.func1()
      /home/arthur/go/src/devpod/vendor/k8s.io/client-go/util/retry/util.go:51 +0x41
  k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection()
      /home/arthur/go/src/devpod/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:145 +0x53
  k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff()
      /home/arthur/go/src/devpod/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:461 +0x5e
  k8s.io/client-go/util/retry.OnError()
      /home/arthur/go/src/devpod/vendor/k8s.io/client-go/util/retry/util.go:50 +0x130
  github.com/loft-sh/devpod/pkg/tunnel.RunInContainer()
      /home/arthur/go/src/devpod/pkg/tunnel/services.go:59 +0x3eb
  github.com/loft-sh/devpod/cmd.(*SSHCmd).startServices()
      /home/arthur/go/src/devpod/cmd/ssh.go:484 +0x144
  github.com/loft-sh/devpod/cmd.(*SSHCmd).startTunnel.gowrap1()
      /home/arthur/go/src/devpod/cmd/ssh.go:402 +0x104

hack/dev_devpod.sh

hack/build-e2e-race.sh

pascalbreuninger

Forgot to submit review 🤦‍♂️

hack/dev_devpod.sh

hack/build-e2e-race.sh

pascalbreuninger

LGTM

bkneis added 4 commits December 3, 2024 10:40

Add load testing scripts

0e0df27

Fix race conditon

db19329

Add CI pipeline to use race detector

09a71e5

Build binaries using race detector for e2e race

7a0ab80

bkneis changed the title ~~Add load testing scripts~~ Add load testing scripts, profiler and data race fixes Dec 3, 2024

bkneis added 3 commits December 4, 2024 07:18

Allow stale to use more issues each run

e9325ce

Fix race e2e

1917b21

Remove race tests from CI

7050743

pascalbreuninger reviewed Dec 4, 2024

View reviewed changes

hack/dev_devpod.sh Outdated Show resolved Hide resolved

pascalbreuninger reviewed Dec 4, 2024

View reviewed changes

hack/build-e2e-race.sh Outdated Show resolved Hide resolved

Update dev scripts with race detector

6f16f21

bkneis requested review from a team and pascalbreuninger December 4, 2024 08:44

pascalbreuninger reviewed Dec 4, 2024

View reviewed changes

hack/dev_devpod.sh Outdated Show resolved Hide resolved

hack/build-e2e-race.sh Outdated Show resolved Hide resolved

PR comments

ead5205

bkneis requested a review from pascalbreuninger December 6, 2024 07:46

pascalbreuninger approved these changes Dec 9, 2024

View reviewed changes

pascalbreuninger merged commit a15412d into loft-sh:main Dec 9, 2024
23 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add load testing scripts, profiler and data race fixes #1441

Add load testing scripts, profiler and data race fixes #1441

bkneis commented Dec 3, 2024 •

edited

Loading

pascalbreuninger left a comment

pascalbreuninger left a comment

Add load testing scripts, profiler and data race fixes #1441

Add load testing scripts, profiler and data race fixes #1441

Conversation

bkneis commented Dec 3, 2024 • edited Loading

pascalbreuninger left a comment

Choose a reason for hiding this comment

pascalbreuninger left a comment

Choose a reason for hiding this comment

bkneis commented Dec 3, 2024 •

edited

Loading