Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider using Forks for parallelism? #83

Closed
Shians opened this issue Nov 2, 2018 · 3 comments
Closed

Consider using Forks for parallelism? #83

Shians opened this issue Nov 2, 2018 · 3 comments

Comments

@Shians
Copy link
Contributor

Shians commented Nov 2, 2018

By default parallel::makeCluster() uses type = "PSOCK" for its parallelism. But changing the type to "FORK" is more efficient on non-Windows systems. I suggest wherever makeCluster() is used the code be changed to

if (.Platform$OS.type == "windows") {
    cl <- parallel::makeCluster(n_cores, type = "PSOCK", outfile = "")
} else {
    cl <- parallel::makeCluster(n_cores, type = "FORK", outfile = "")
}

This also significantly reduces memory usage on systems where forking is available.

With my quick testing of 300 cells with 5000 genes and ks = 1:10 I saw the PSOCK version taking ~90 seconds and the FORK version taking ~60 seconds.

The downside would be that forks do not parallelise across clusters.

@wikiselev
Copy link
Member

Hi, many thanks for your suggestions and very sorry for a late reply. Could you please create a pull request with the suggested changes and we will merge it to master?
Thanks again,
Vlad

@gorgitko
Copy link

@Shians See #104

@Shians
Copy link
Contributor Author

Shians commented Jun 30, 2021

Very nice! Glad to see such a clean revamp of the parallelism.

@Shians Shians closed this as completed Jun 30, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants