More up-to-date PyTorch?

[carlini]

Thanks for publishing this competition. It looks great.

Any chance we could get a requirements file with a more up-to-date version of PyTorch? We're currently on 1.13, and unfortunately version 1.8 doesn't play well with the CUDA/GPU configurations for the majority of the machines I have available to me.

I've spent a few hours fighting with things and if I figure out a configuration that works I'll close send a PR but if you happen to have one available it'd be much appreciated.

[s-zanella]

Thanks for your interest, Nicholas!

The reason why we require PyTorch 1.8 is that it is the latest version officially compatible with torchcsprng. There are open issues in pytorch/opacus#504 and pytorch/csprng#130 that address this.

That said, torchcsprng builds fine from source with PyTorch 1.13.0 (e.g., with pip install git+https://github.com/pytorch/csprng.git@7820f5ddf258cd06c2487caa53a6695daa0ee644). When running some limited tests, I didn't run into any issues and the behaviour seems consistent with what I get with PyTorch 1.8.1. However, this isn't the environment we used to create or test the competition, so I can't say whether it will work fine in all cases.

We need torchcsprng to train models with Opacus in secure mode, which uses a cryptographically secure PRNG for noise addition and batch sampling. This incidentally patches Gaussian noise sampling against some floating-point representation attacks (https://arxiv.org/abs/2107.10138). Presumably, you could train additional models with Opacus without setting secure mode which for most practical matters won't differ from the models that we provide. This won't require torchcsprng and will allow you to use a more recent version of PyTorch that plays well with your configuration.

Unfortunately, we also took a dependency on torchcsprng.create_mt19937_generator to recreate data splits from seeds. To overcome this, you could use torchcsprng just to recreate the data splits, store them, and then load them in a different environment without torchcsprng. (I just made importing torchcsprng conditional to allow one to do this.)

For loading and running the models in forward mode you won't need torchcsprng and I believe that there won't be any difference using a more up-to-date version of PyTorch, such as the latest stable 1.13.

[xehartnort]

To be more concise, according to [1], [2] and [3], the following combination of versions are the most up-to-date versions of pytorch, csprng, opacus for GPU that work flawlessly are:

• opacus==1.1.3
• torchcsprng==0.2.0+cu111
• torch==1.8.0+cu111

It took me some time to find the correct configuration: Could you had this to the repo, so that other people do not need to waste their time?

[1] csprng repo
[2] opacus releases
[3] pytorch

[s-zanella]

Thanks for digging into this @xehartnort!

For the record, these are the requirements we recommend:

• opacus==1.1.3
• torchcsprng==0.2.1
• torch==1.8.1

This is the configuration we used. We tested that it works well in a clean environment. What problem did you encounter when trying it?

Despite what the torchcsprng repository says, the latest version of torchcsprng available from PyPI (0.2.1) works fine with torch 1.8.1. Note that this is also the version Opacus maintainers recommended in the issue where you cross-posted your message.

We wouldn't want to prescribe a specific version of the CUDA toolkit or even the use of CUDA at all since GPU acceleration is not required. There are several versions that would work with torch 1.8.1 and many pre-compiled binaries available (https://anaconda.org/pytorch/pytorch/files?version=1.8.1).