Code to evaluate the downstream (linear classification) quality of linear representations learned by MatrixSSL [1] and Spectral Contrastive Learning [2] across various synthetic learning tasks such as the boolean hypercube task from Saunshi et al. (2022) [3]. This empirical work complements theoretical analyses on how the inductive biases of linear embedding functions affect representation quality.
[1] Zhang, Y., Tan, Z., Yang, J., Huang, W., & Yuan, Y. (2023). Matrix Information Theory for Self-Supervised Learning (Version 5). arXiv. https://doi.org/10.48550/ARXIV.2305.17326
[2] HaoChen, J. Z., Wei, C., Gaidon, A., & Ma, T. (2021). Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss (Version 7). arXiv. https://doi.org/10.48550/ARXIV.2106.04156
[3] Saunshi, N., Ash, J., Goel, S., Misra, D., Zhang, C., Arora, S., Kakade, S., & Krishnamurthy, A. (2022). Understanding Contrastive Learning Requires Incorporating Inductive Biases (Version 1). arXiv. https://doi.org/10.48550/ARXIV.2202.14037