This repository contains code that attempts to reproduce the results of the paper General-Purpose In-Context Learning by Meta-Learning Transformers in PyTorch. The most important pieces of code can be found in explore.ipynb.
In short, I couldn't get over the memorization to the point of phase transition in the model's generalization ability as was shown in Figure 4 of the paper. Given the computational burden of the number of tasks, I discontinued experiments but hope to get back to this later.
