Why VQVAE? How about VAE?

[LinB203]

In fact, we believe that both VQVAE and VAE can be used, but we have noticed that there is already a Video-VQVAE in the community that can be directly used for testing purposes.

Additionally, the combination of VQVAE and Diffusion works well for Latent-Diffusion. We have attempted to train a Video-VAE, but it was too resource-intensive. In the early stages of the Open-Sora Plan, the open-source Video-VQVAE can serve as a viable alternative.

[Edwardyan1112]

According to your estimates, how much computational resources and video datasets are required to train a Video VAE (not Video-VQVAE)? Approximately, how long would the training process take?

[LinB203]

We have tested that it can only batch size 2 on 80G. Input 16 frames x 256 x 256 and latent size 2 x 32 x 32.
We are not sure when it will converge because it is just too costly.

[lucasjinreal]

Does Sora used VAE not VQVAE?

[LinB203]

It's unknown, and at the moment we use Video-VQVAE as just a compromise.

[lucasjinreal]

Maybe VQVAE can handle it, stabledifusion3 also using VQVAE

[CuddleSabe]

Maybe just another ViT as encoder

[ttttt12345]

it is unreal bro,dont waste your time on this project.

[LinB203]

We are now able to support both vae and vqvae, please check our latest code.

[lucasjinreal]

how's the performance compare

[LinB203]

Refer to #93 for more details.

[hamza-m-farooqi]

looking at the intense resource requirements for Video-VAE and the practicality of Video-VQVAE, it seems like a smart compromise for now. eager to see how this compromise plays out 😊