Note about ssh connections in executors

[kessler-frost]

As of now, both, the SSHExecutor and the SlurmExecutor, use ssh connections to connect to the remote machines and wait for the result to be obtained before closing the connection and returning the result. They create a new connection for every new electron, which means if there are 20 long running parallelizable electrons then there have to be 20 open ssh connections. But as we've observed, there is usually a limit on the number of open ssh connections one machine can have (both, on the remote side and on the client side), which translates to a limit on the number of electrons one can execute at a time.

Since this limit is more or less a limitation of the system itself, the responsibility to handle it falls on the user of those executors. One can utilize the ct.wait functionality to batch their electron executions so that they don't exceed their number of open ssh connections. This, of course, come with the cost of reduced parallelizability.

Writers of ssh based executors can do little in this matter, but one course of action might be:

• Close the ssh connection once the job has been submitted
• Have a poll_time (user input) variable which means that a query will be made after every poll_time seconds to check whether the function has finished execution
• There will be a new connection made for every query and will be closed after the query command has returned
• Once a query returns with a successful result, close the connection, create a new connection to obtain the result then close that as well
• This should significantly reduce the number of open connections at a time and without compromising with concurrency of submitting jobs to remote machines

[santoshkumarradha]

Thanks for the points @kessler-frost,

@wjcunningham7 we might need a solution for this.