This repository has been archived by the owner on Mar 19, 2024. It is now read-only.
Incorrect accuracy when running benchmark on linear_image_classification on imagenet1K #545
Assignees
Comments
|
One other minor issue, what is the correct way to combine the configs from |
|
Hi @VicaYang, First of all, thank you for using VISSL :) So let me tackle first the question of how to combine training components. The way to do it is to use the override syntax of Hydra: The "+" syntax will deal with the merging. Now the classification accuracy issues: On the supervised ViT, I am not sure there is any issue. The train accuracy can sometimes be different from the test accuracy because the augmentations (especially the augmentations for ViT, because ViT lack a lot of biases that CNN have) can be pretty agressive to enforce regularisation. On the MoCo side there is clearly a big issue. I think the model is probably not correctly loaded, and so the accuracy you get is on linear evaluation on top of a random representation (hence the 1% top-1 accuracy). So we need to debug why. Could you please send me the logs that you got (the Inside of it, you can grep for Thank you, |
|
Updated: well, I checked the moco2/log.txt and found that the weight seems not correctly loaded. I am trying a different prefix to see if it works. =================== Thank @QuentinDuval for your help! I will try your config to see how it works. |
|
I spent 2 hours and still failed to load the weight from mocov3 correctly :( So I think |
|
The error above is related to #550 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Instructions To Reproduce the Issue:
I used the prebuilt vissl, created a new folder contains
tools/run_distributed_engines.pyand all files under folderconfigs, and created a new config to run the benchmark using vit-b16Check https://stackoverflow.com/help/minimal-reproducible-example for how to ask good questions.
Simplify the steps to reproduce the issue using suggestions from the above link, and provide them below:
git diff)and my
dataset_catalog.json(I am not sure should I usetestsplit orvalsplit for "val"; thevalfolder is created using this script2/metrics.jsonmoco2/metrics.jsonExpected behavior:
2/metrics.json, the top-1 accuracy on train and val should not differ so much (0.55 and 0.79 now)moco2/metrics.json, the top-1 accuracy should not be greater than 1(1.3 on train and 1.8 on test now)I also check the difference between the generated
train_config.yamlbetween my two folder, the only difference is thecheckpoint.dirandweight_initEnvironment:
Provide your environment information using the following command:
The text was updated successfully, but these errors were encountered: