Multiple code locations in the same server

[ArtiNgqobe]

So I have a server that hosts 5 repositories/ code locations in one hardrive,

Everything has been running fine with just the daemon and webserver service until I started getting this error as I added more sensors, assets to the locations,

"dagster._core.errors.DagsterUserCodeUnreachableError: User code server request timed out due to taking longer than 60 seconds to complete"

Then I read about grpc servers, unfortunately because of resource restraints I had to set up the gprc servers as services for each location in the same server, this didn't seem to make a difference, I still get the same error every now and again,

For someone like me who is limited in resources how can I set up dagster to run efficiently for multiple code locations in the same server? What settings or configurations should I pay attention to?

[garethbrickman]

When specifically does this error happen? e.g. when reloading a code location, or when a sensor evaluation runs etc.

It may help to profile your code when running it locally using a tool such as py-spy. There is some additional advice in this GH Discussion.

[ArtiNgqobe]

It happens for regular runs and it normally looks like below, when reloading code location that works fine, I am not quite sure how to link py-spy and still reading on this,

dagster._core.errors.DagsterUserCodeUnreachableError: User code server request timed out due to taking longer than 60 seconds to complete.  File "C:\ProgramData\anaconda3\envs\dasgter\Lib\site-packages\dagster\_daemon\run_coordinator\queued_run_coordinator_daemon.py", line 389, in _dequeue_run    instance.run_launcher.launch_run(LaunchRunContext(dagster_run=run, workspace=workspace))  File "C:\ProgramData\anaconda3\envs\dasgter\Lib\site-packages\dagster\_core\launcher\default_run_launcher.py", line 119, in launch_run    DefaultRunLauncher.launch_run_from_grpc_client(  File "C:\ProgramData\anaconda3\envs\dasgter\Lib\site-packages\dagster\_core\launcher\default_run_launcher.py", line 79, in launch_run_from_grpc_client    grpc_client.start_run(  File "C:\ProgramData\anaconda3\envs\dasgter\Lib\site-packages\dagster\_grpc\client.py", line 657, in start_run    res = self._query(          ^^^^^^^^^^^^  File "C:\ProgramData\anaconda3\envs\dasgter\Lib\site-packages\dagster\_grpc\client.py", line 216, in _query    self._raise_grpc_exception(  File "C:\ProgramData\anaconda3\envs\dasgter\Lib\site-packages\dagster\_grpc\client.py", line 194, in _raise_grpc_exception    raise DagsterUserCodeUnreachableError(The above exception was caused by the following exception:grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:	status = StatusCode.DEADLINE_EXCEEDED	details = "Deadline Exceeded"	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2025-01-14T08:01:56.1279757+00:00"}">  File "C:\ProgramData\anaconda3\envs\dasgter\Lib\site-packages\dagster\_grpc\client.py", line 214, in _query    return self._get_response(method, request=request_type(**kwargs), timeout=timeout)           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "C:\ProgramData\anaconda3\envs\dasgter\Lib\site-packages\dagster\_grpc\client.py", line 174, in _get_response    return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "C:\ProgramData\anaconda3\envs\dasgter\Lib\site-packages\grpc\_channel.py", line 1181, in __call__    return _end_unary_response_blocking(state, call, False, None)           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "C:\ProgramData\anaconda3\envs\dasgter\Lib\site-packages\grpc\_channel.py", line 1006, in _end_unary_response_blocking    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[ArtiNgqobe]

Sensors evaluate and run the jobs and find that sometimes those jobs fail with the same error, no errors on the sensor itself,

see attached example, the evaluation succeeds just fine and triggers the run and the run fails with the above error, not sure how its linked

[garethbrickman]

We have this guide about handling sensor timeouts: #20021