Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update README #30

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Update README #30

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
85b8620
Update README.md
roymiles Nov 19, 2024
0f91d75
Update README.md
roymiles Nov 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 56 additions & 52 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,58 +107,60 @@
4. A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning. Yim, Junho et al. CVPR 2017
5. Like What You Like: Knowledge Distill via Neuron Selectivity Transfer. Huang, Zehao & Wang, Naiyan. 2017
6. Paraphrasing complex network: Network compression via factor transfer. Kim, Jangho et al. NeurIPS 2018
7. Knowledge transfer with jacobian matching. ICML 2018
8. Self-supervised knowledge distillation using singular value decomposition. Lee, Seung Hyun et al. ECCV 2018
9. Learning Deep Representations with Probabilistic Knowledge Transfer. Passalis et al. ECCV 2018
10. Variational Information Distillation for Knowledge Transfer. Ahn, Sungsoo et al. CVPR 2019
11. Knowledge Distillation via Instance Relationship Graph. Liu, Yufan et al. CVPR 2019
12. Knowledge Distillation via Route Constrained Optimization. Jin, Xiao et al. ICCV 2019
13. Similarity-Preserving Knowledge Distillation. Tung, Frederick, and Mori Greg. ICCV 2019
14. MEAL: Multi-Model Ensemble via Adversarial Learning. Shen,Zhiqiang, He,Zhankui, and Xue Xiangyang. AAAI 2019
15. A Comprehensive Overhaul of Feature Distillation. Heo, Byeongho et al. ICCV 2019 [[code]][2.15]
16. Feature-map-level Online Adversarial Knowledge Distillation. ICML 2020
17. Distilling Object Detectors with Fine-grained Feature Imitation. ICLR 2020
18. Knowledge Squeezed Adversarial Network Compression. Changyong, Shu et al. AAAI 2020
19. Stagewise Knowledge Distillation. Kulkarni, Akshay et al. arXiv: 1911.06786
20. Knowledge Distillation from Internal Representations. AAAI 2020
21. Knowledge Flow:Improve Upon Your Teachers. ICLR 2019
22. LIT: Learned Intermediate Representation Training for Model Compression. ICML 2019
23. Improving the Adversarial Robustness of Transfer Learning via Noisy Feature Distillation. Chin, Ting-wu et al. arXiv:2002.02998
24. Knapsack Pruning with Inner Distillation. Aflalo, Yonathan et al. arXiv:2002.08258
25. Residual Knowledge Distillation. Gao, Mengya et al. arXiv:2002.09168
26. Knowledge distillation via adaptive instance normalization. Yang, Jing et al. arXiv:2003.04289
27. Bert-of-Theseus: Compressing bert by progressive module replacing. Xu, Canwen et al. arXiv:2002.02925 [[code]][2.27]
28. Distilling Spikes: Knowledge Distillation in Spiking Neural Networks. arXiv:2005.00727
29. Generalized Bayesian Posterior Expectation Distillation for Deep Neural Networks. Meet et al. arXiv:2005.08110
30. Feature-map-level Online Adversarial Knowledge Distillation. Chung, Inseop et al. ICML 2020
31. Channel Distillation: Channel-Wise Attention for Knowledge Distillation. Zhou, Zaida et al. arXiv:2006.01683 [[code]][2.30]
32. Matching Guided Distillation. ECCV 2020 [[code]][2.31]
33. Differentiable Feature Aggregation Search for Knowledge Distillation. ECCV 2020
34. Interactive Knowledge Distillation. Fu, Shipeng et al. arXiv:2007.01476
35. Feature Normalized Knowledge Distillation for Image Classification. ECCV 2020 [[code]][2.34]
36. Layer-Level Knowledge Distillation for Deep Neural Networks. Li, Hao Ting et al. Applied Sciences, 2019
37. Knowledge Distillation with Feature Maps for Image Classification. Chen, Weichun et al. ACCV 2018
38. Efficient Kernel Transfer in Knowledge Distillation. Qian, Qi et al. arXiv:2009.14416
39. Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition. arXiv:2009.06902
40. Kernel Based Progressive Distillation for Adder Neural Networks. Xu, Yixing et al. NeurIPS 2020
41. Feature Distillation With Guided Adversarial Contrastive Learning. Bai, Tao et al. arXiv:2009.09922
42. Pay Attention to Features, Transfer Learn Faster CNNs. Wang, Kafeng et al. ICLR 2019
43. Multi-level Knowledge Distillation. Ding, Fei et al. arXiv:2012.00573
44. Cross-Layer Distillation with Semantic Calibration. Chen, Defang et al. AAAI 2021 [[code]][2.44]
45. Harmonized Dense Knowledge Distillation Training for Multi-­Exit Architectures. Wang, Xinglu & Li, Yingming. AAAI 2021
46. Robust Knowledge Transfer via Hybrid Forward on the Teacher-Student Model. Song, Liangchen et al. AAAI 2021
47. Show, Attend and Distill: Knowledge Distillation via Attention-­Based Feature Matching. Ji, Mingi et al. AAAI 2021 [[code]][2.47]
48. MINILMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers. Wang, Wenhui et al. arXiv:2012.15828
49. ALP-KD: Attention-Based Layer Projection for Knowledge Distillation. Peyman et al. AAAI 2021
50. In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation. Reyhan et al. arXiv:2103.00053
51. Fixing the Teacher-Student Knowledge Discrepancy in Distillation. Han, Jiangfan et al. arXiv:2103.16844
52. Student Network Learning via Evolutionary Knowledge Distillation. Zhang, Kangkai et al. arXiv:2103.13811
53. Distilling Knowledge via Knowledge Review. Chen, Pengguang et al. CVPR 2021
54. Knowledge Distillation By Sparse Representation Matching. Tran et al. arXiv:2103.17012
55. Task-Oriented Feature Distillation. Zhang et al. NeurIPS 2020 [[code]][2.55]
56. Adversarial Knowledge Transfer from Unlabeled Data. Gupta et al. ACM-MM 2020 [code](https://github.com/agupt013/akt)
57. Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferability. He et al. CVPR 2020
58. PDF-Distil: Including Prediction Disagreements in Feature-based Knowledge Distillation for Object Detection. Zhang et al. BMVC 2021 [code](https://github.com/ZHANGHeng19931123/MutualGuide)
7. VkD : Improving Knowledge Distillation using Orthogonal Projections. Roy Miles, Ismail Elezi, Jiankang Deng. CVPR 2024 [[code]][2.7]
8. Understanding the Role of the Projector in Knowledge Distillation. Roy Miles, Krystian Mikolajczyk. AAAI 2024 [[code]][2.8]
9. Knowledge transfer with jacobian matching. ICML 2018
10. Self-supervised knowledge distillation using singular value decomposition. Lee, Seung Hyun et al. ECCV 2018
11. Learning Deep Representations with Probabilistic Knowledge Transfer. Passalis et al. ECCV 2018
12. Variational Information Distillation for Knowledge Transfer. Ahn, Sungsoo et al. CVPR 2019
13. Knowledge Distillation via Instance Relationship Graph. Liu, Yufan et al. CVPR 2019
14. Knowledge Distillation via Route Constrained Optimization. Jin, Xiao et al. ICCV 2019
15. Similarity-Preserving Knowledge Distillation. Tung, Frederick, and Mori Greg. ICCV 2019
16. MEAL: Multi-Model Ensemble via Adversarial Learning. Shen,Zhiqiang, He,Zhankui, and Xue Xiangyang. AAAI 2019
17. A Comprehensive Overhaul of Feature Distillation. Heo, Byeongho et al. ICCV 2019 [[code]][2.15]
18. Feature-map-level Online Adversarial Knowledge Distillation. ICML 2020
19. Distilling Object Detectors with Fine-grained Feature Imitation. ICLR 2020
20. Knowledge Squeezed Adversarial Network Compression. Changyong, Shu et al. AAAI 2020
21. Stagewise Knowledge Distillation. Kulkarni, Akshay et al. arXiv: 1911.06786
22. Knowledge Distillation from Internal Representations. AAAI 2020
23. Knowledge Flow:Improve Upon Your Teachers. ICLR 2019
24. LIT: Learned Intermediate Representation Training for Model Compression. ICML 2019
25. Improving the Adversarial Robustness of Transfer Learning via Noisy Feature Distillation. Chin, Ting-wu et al. arXiv:2002.02998
26. Knapsack Pruning with Inner Distillation. Aflalo, Yonathan et al. arXiv:2002.08258
27. Residual Knowledge Distillation. Gao, Mengya et al. arXiv:2002.09168
28. Knowledge distillation via adaptive instance normalization. Yang, Jing et al. arXiv:2003.04289
29. Bert-of-Theseus: Compressing bert by progressive module replacing. Xu, Canwen et al. arXiv:2002.02925 [[code]][2.27]
30. Distilling Spikes: Knowledge Distillation in Spiking Neural Networks. arXiv:2005.00727
31. Generalized Bayesian Posterior Expectation Distillation for Deep Neural Networks. Meet et al. arXiv:2005.08110
32. Feature-map-level Online Adversarial Knowledge Distillation. Chung, Inseop et al. ICML 2020
33. Channel Distillation: Channel-Wise Attention for Knowledge Distillation. Zhou, Zaida et al. arXiv:2006.01683 [[code]][2.30]
34. Matching Guided Distillation. ECCV 2020 [[code]][2.31]
35. Differentiable Feature Aggregation Search for Knowledge Distillation. ECCV 2020
36. Interactive Knowledge Distillation. Fu, Shipeng et al. arXiv:2007.01476
37. Feature Normalized Knowledge Distillation for Image Classification. ECCV 2020 [[code]][2.34]
38. Layer-Level Knowledge Distillation for Deep Neural Networks. Li, Hao Ting et al. Applied Sciences, 2019
39. Knowledge Distillation with Feature Maps for Image Classification. Chen, Weichun et al. ACCV 2018
40. Efficient Kernel Transfer in Knowledge Distillation. Qian, Qi et al. arXiv:2009.14416
41. Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition. arXiv:2009.06902
42. Kernel Based Progressive Distillation for Adder Neural Networks. Xu, Yixing et al. NeurIPS 2020
43. Feature Distillation With Guided Adversarial Contrastive Learning. Bai, Tao et al. arXiv:2009.09922
44. Pay Attention to Features, Transfer Learn Faster CNNs. Wang, Kafeng et al. ICLR 2019
45. Multi-level Knowledge Distillation. Ding, Fei et al. arXiv:2012.00573
46. Cross-Layer Distillation with Semantic Calibration. Chen, Defang et al. AAAI 2021 [[code]][2.44]
47. Harmonized Dense Knowledge Distillation Training for Multi-­Exit Architectures. Wang, Xinglu & Li, Yingming. AAAI 2021
48. Robust Knowledge Transfer via Hybrid Forward on the Teacher-Student Model. Song, Liangchen et al. AAAI 2021
49. Show, Attend and Distill: Knowledge Distillation via Attention-­Based Feature Matching. Ji, Mingi et al. AAAI 2021 [[code]][2.47]
50. MINILMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers. Wang, Wenhui et al. arXiv:2012.15828
51. ALP-KD: Attention-Based Layer Projection for Knowledge Distillation. Peyman et al. AAAI 2021
52. In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation. Reyhan et al. arXiv:2103.00053
53. Fixing the Teacher-Student Knowledge Discrepancy in Distillation. Han, Jiangfan et al. arXiv:2103.16844
54. Student Network Learning via Evolutionary Knowledge Distillation. Zhang, Kangkai et al. arXiv:2103.13811
55. Distilling Knowledge via Knowledge Review. Chen, Pengguang et al. CVPR 2021
56. Knowledge Distillation By Sparse Representation Matching. Tran et al. arXiv:2103.17012
57. Task-Oriented Feature Distillation. Zhang et al. NeurIPS 2020 [[code]][2.55]
58. Adversarial Knowledge Transfer from Unlabeled Data. Gupta et al. ACM-MM 2020 [code](https://github.com/agupt013/akt)
59. Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferability. He et al. CVPR 2020
60. PDF-Distil: Including Prediction Disagreements in Feature-based Knowledge Distillation for Object Detection. Zhang et al. BMVC 2021 [code](https://github.com/ZHANGHeng19931123/MutualGuide)

### Graph-based

Expand Down Expand Up @@ -928,3 +930,5 @@ Contact: Yuang Liu (frankliu624![](https://res.cloudinary.com/flhonker/image/upl
[18.32]: http://zhiqiangshen.com/projects/LS_and_KD/index.html
[2.55]: https://github.com/ArchipLab-LinfengZhang/Task-Oriented-Feature-Distillation
[19.10]: https://github.com/roymiles/ITRD
[2.7]: https://github.com/roymiles/VkD
[2.8]: https://github.com/roymiles/Simple-Recipe-Distillation