Failure to launch Zed service

[philrz]

This issue seems to repro starting with Zui and Zui Insiders builds created some time starting around November 3, 2023.

I first reported the symptom in detail at #2866 (comment) while I was testing a Dev build based on that branch. However, I've since confirmed that recent Zui Insiders releases starting with 1.3.2-9 all exhibit the same symptom. I also went back and dug up the Dev Build from this Actions run based on commit acb5267 and confirmed it works fine as I remembered it, then just now made another Dev Build in another Actions run also based on commit acb5267 and found that one fails with the symptom. So the problem seems to follow "when the build was made" and not "our code that went into the build".

Conclusion: Based on all this, I have to suspect that maybe there's some dependency for which we're pointing at "the latest" and that dependency has advanced to something that breaks us. I have no idea what that might be though!

[philrz]

I've studied this a bit more and it seems the symptom varies based on the OS platform.

macOS

Since I happen to use macOS on the desktop, everything described in the opening text of this issue was first reproduced with macOS. Given this reference:

Cannot find module '/Applications/Zui.app/Contents/Resources/app.asar/node_modules/node-pipe/build/Release/pipe'

in #2866 (comment), then comparing the Actions run log from the working build with the failed one, the smoking gun does seem to reveal itself in the failed build's "Build Zui" step:

• cannot build optional native dependency  name=node-pipe
                                             version=0.1.1
                                             cause=exit status 1
                                             errorOut=gyp info it worked if it ends with ok
    gyp info using [email protected]
    gyp info using [email protected] | darwin | x64
    gyp info find Python using Python version 3.12.0 found at "/usr/local/bin/python3"
    gyp http GET https://electronjs.org/headers/v22.3.25/node-v22.3.25-headers.tar.gz
    gyp http 200 https://artifacts.electronjs.org/headers/v22.3.25/node-v22.3.25-headers.tar.gz?force_headers_dist=1
    gyp http GET https://electronjs.org/headers/v22.3.25/SHASUMS256.txt
    gyp http 200 https://artifacts.electronjs.org/headers/v22.3.25/SHASUMS256.txt?force_headers_dist=1
    gyp info spawn /usr/local/bin/python3
    gyp info spawn args [
    gyp info spawn args   '/Users/runner/work/zui/zui/node_modules/node-gyp/gyp/gyp_main.py',
    gyp info spawn args   'binding.gyp',
    gyp info spawn args   '-f',
    gyp info spawn args   'make',
    gyp info spawn args   '-I',
    gyp info spawn args   '/Users/runner/work/zui/zui/node_modules/node-pipe/build/config.gypi',
    gyp info spawn args   '-I',
    gyp info spawn args   '/Users/runner/work/zui/zui/node_modules/node-gyp/addon.gypi',
    gyp info spawn args   '-I',
    gyp info spawn args   '/Users/runner/.electron-gyp/22.3.25/include/node/common.gypi',
    gyp info spawn args   '-Dlibrary=shared_library',
    gyp info spawn args   '-Dvisibility=default',
    gyp info spawn args   '-Dnode_root_dir=/Users/runner/.electron-gyp/22.3.25',
    gyp info spawn args   '-Dnode_gyp_dir=/Users/runner/work/zui/zui/node_modules/node-gyp',
    gyp info spawn args   '-Dnode_lib_file=/Users/runner/.electron-gyp/22.3.25/<(target_arch)/node.lib',
    gyp info spawn args   '-Dmodule_root_dir=/Users/runner/work/zui/zui/node_modules/node-pipe',
    gyp info spawn args   '-Dnode_engine=v8',
    gyp info spawn args   '--depth=.',
    gyp info spawn args   '--no-parallel',
    gyp info spawn args   '--generator-output',
    gyp info spawn args   'build',
    gyp info spawn args   '-Goutput_dir=.'
    gyp info spawn args ]
    Traceback (most recent call last):
      File "/Users/runner/work/zui/zui/node_modules/node-gyp/gyp/gyp_main.py", line 42, in <module>
        import gyp  # noqa: E402
        ^^^^^^^^^^
      File "/Users/runner/work/zui/zui/node_modules/node-gyp/gyp/pylib/gyp/__init__.py", line 9, in <module>
        import gyp.input
      File "/Users/runner/work/zui/zui/node_modules/node-gyp/gyp/pylib/gyp/input.py", line 19, in <module>
        from distutils.version import StrictVersion
    ModuleNotFoundError: No module named 'distutils'
    gyp ERR! configure error 
    gyp ERR! stack Error: `gyp` failed with exit code: 1
    gyp ERR! stack     at ChildProcess.onCpExit (/Users/runner/work/zui/zui/node_modules/node-gyp/lib/configure.js:259:16)
    gyp ERR! stack     at ChildProcess.emit (node:events:3[90](https://github.com/brimdata/zui/actions/runs/6775178417/job/18413872960#step:4:93):28)
    gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:290:12)
    gyp ERR! System Darwin 21.6.0
    gyp ERR! command "/Users/runner/hostedtoolcache/node/16.10.0/x64/bin/node" "/Users/runner/work/zui/zui/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
    gyp ERR! cwd /Users/runner/work/zui/zui/node_modules/node-pipe
    gyp ERR! node -v v16.10.0
    gyp ERR! node-gyp -v v8.4.1
    gyp ERR! not ok 
    
                                             command=/private/var/folders/3s/vfzpb5r51gs6y328rmlgzm7c0000gn/T/xfs-52fcf281/yarn run install
                                             workingDir=/Users/runner/work/zui/zui/node_modules/node-pipe

Doing a web search for "ModuleNotFoundError: No module named 'distutils' node-gyp" uncovered nodejs/node-gyp#2869, where the tl;dr seems to be that node-gyp will fail if Python 3.12 is in use. Sure enough, grepping the raw log from the Actions run for the working build, we find:

gyp info find Python using Python version 3.11.5 found at "/usr/local/bin/python3"

whereas for the failed build:

gyp info find Python using Python version 3.12.0 found at "/usr/local/bin/python3"

That same thread points to a workaround that I've already tried out in a branch that seems effective. 👍

Linux

Thinking that explained everything, something made me think to try on other platforms. While it's true that the most recent Zui Insiders builds also don't work on Linux, in that case 1.3.2-9 actually works ok (that one failed on macOS) but 1.3.2-10 is where it starts failing. Looking at the log for building 1.3.2-10, it's still a problem at the node-pipe step, but looks different from what we saw on macOS.

• cannot build optional native dependency  name=node-pipe
                                             version=0.1.1
                                             cause=exit status 1
                                             errorOut=gyp info it worked if it ends with ok
    gyp info using [email protected]
    gyp info using [email protected] | linux | x64
    gyp info find Python using Python version 3.8.10 found at "/usr/bin/python3"
    gyp http GET https://electronjs.org/headers/v22.3.25/node-v22.3.25-headers.tar.gz
    gyp http 200 https://artifacts.electronjs.org/headers/v22.3.25/node-v22.3.25-headers.tar.gz?force_headers_dist=1
    gyp http GET https://electronjs.org/headers/v22.3.25/SHASUMS2[56](https://github.com/brimdata/zui/actions/runs/6758922143/job/18371044937#step:7:59).txt
    gyp WARN install got an error, rolling back install
    gyp ERR! configure error 
    gyp ERR! stack FetchError: request to https://www.electronjs.org/headers/v22.3.25/SHASUMS256.txt failed, reason: read ECONNRESET
    gyp ERR! stack     at ClientRequest.<anonymous> (/home/runner/work/zui/zui/node_modules/minipass-fetch/lib/index.js:110:14)
    gyp ERR! stack     at ClientRequest.emit (node:events:390:28)
    gyp ERR! stack     at TLSSocket.socketErrorListener (node:_http_client:447:9)
    gyp ERR! stack     at TLSSocket.emit (node:events:402:35)
    gyp ERR! stack     at emitErrorNT (node:internal/streams/destroy:1[57](https://github.com/brimdata/zui/actions/runs/6758922143/job/18371044937#step:7:60):8)
    gyp ERR! stack     at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    gyp ERR! stack     at processTicksAndRejections (node:internal/process/task_queues:83:21)
    gyp ERR! System Linux 5.15.0-1050-azure
    gyp ERR! command "/opt/hostedtoolcache/node/16.10.0/x[64](https://github.com/brimdata/zui/actions/runs/6758922143/job/18371044937#step:7:67)/bin/node" "/home/runner/work/zui/zui/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
    gyp ERR! cwd /home/runner/work/zui/zui/node_modules/node-pipe
    gyp ERR! node -v v16.10.0
    gyp ERR! node-gyp -v v8.4.1
    gyp ERR! not ok

Notice that it references an older Python version 3.8.10. Not sure what to make of that.

WIndows

This code:

zui/packages/zed-node/src/lake.ts

Lines 57 to 66 in 4186053

	// For unix systems, pass posix pipe read file descriptor into lake process.
	// In the event of Zui getting shutdown via `SIGKILL`, this will let lake
	// know that it has been orphaned and to shutdown.
	if (process.platform !== 'win32') {
	// eslint-disable-next-line @typescript-eslint/no-var-requires
	const { readfd } = require('node-pipe').pipeSync();
	opts.stdio.push(readfd);
	args.push(`-brimfd=${opts.stdio.length - 1}`);
	}

Makes it look like node-pipe is not used on Windows. Indeed, I was able to launch the latest Zui Insiders (1.3.2-11 at the moment) without a problem. Therefore I'm going to assume Windows is not our concern here.

[philrz]

After researching this issue I became concerned that Zui effectively has a hard dependency on node-pipe on macOS/Linux but these bad builds came from "green" runs on Actions. It turns out this is a known problem #1165 from 3+ years ago that's only just now burning us again. 😬 The good news is that @jameskerr is looking at fixing that now. 🤞 In the meantime, #2883 has a workaround that should address the No module named 'distutils' symptom that was causing bad builds on macOS.

[philrz]

The workaround in #2883 has been merged and builds are ok on macOS again (e.g., Zui Insiders 1.3.2-13). As for the other failure I showed above on Linux, in the notes in #2883 I mention how by the time I was done testing that change I also managed to see that same symptom on macOS as well. That led us to a team discussion about the wider exposure described in #1165 and we decided to move forward with replacing node-pipe with what's now tracked in #2884.

In conclusion, the primary issue originally tracked in this issue has now been addressed and the other problem spotted along the way is being tracked elsewhere, so I'm closing this one.