Sorry, but I suspect your FID is fake

[johnzlj]

I used your provided pretrained model to create 50000 cifar images, and then I used the public TTUR project[1] to calculate the FID. Finally I get FID=66, Instead of the 5.33 you claimed in your paper.

Can you give me an explanation?

[1] TTUR: https://github.com/bioinf-jku/TTUR

[balthazar]

Dupe of #99

[Ranazzi]

Are you using the same sample size? bear in mind that FID is a biased metric

[mxochicale]

@Ranazzi, I would be interested to read more and have better understanding on why FID is a biased metric. Maybe you would like to add further references. Thanks

[Ranazzi]

@mxochicale You can find some content in arxiv.org/abs/1801.01401 and arxiv.org/abs/1911.07023. Feel free to contact me if you need some more content about GANs.

Regards.

[mxochicale]

Thanks @Ranazzi , I did use FID from my previous work in Ultrasound fetal brain imaging and looking into it using diffusion models but will check out shared papers and reach out if I have questions. It would also great that you provide alternatives methods to help @johnzlj and others on quantifying image synthesis.

[Ranazzi]

@mxochicale Some tips I recommend: besides FID, use KID (Kernel Inception Distance); use a sufficiently large sample to compute FID (In my personal experience, I always use at least 20k samples); when FID is used to compare different settings (architectures, losses, etc.) always use the same size during the analysis.

[johnzlj]

oh, sorry, I think maybe I made a mistake. Feel so sorry for interruptions. john ***@***.***  

…

------------------ Original ------------------ From: "NVlabs/stylegan2-ada" ***@***.***>; Date: Sat, Jan 28, 2023 12:41 PM ***@***.***>; ***@***.******@***.***>; Subject: Re: [NVlabs/stylegan2-ada] Sorry, but I suspect your FID is fake (#100) @mxochicale Some tips I recommend: besides FID, use KID (Kernel Inception Distance); use a sufficiently large sample to compute FID (In my personal experience, I always use at least 20k samples); when FID is used to compare different settings (architectures, losses, etc.) always use the same size during the analysis. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[ThanosM97]

Hi @johnzlj,

Did you manage to fix your mistake? If so, could you please tell me what was the problem because I am facing the same issue?

Thanks.