Deleting StableEmbedding still occupies GPU RAM

[KMFODA]

System Info

Python 3.10.6
Ubuntu 22.04.2 LTS
transformers 4.39.3

Reproduction

import torch
import gc
import bitsandbytes

wte = bitsandbytes.nn.modules.StableEmbedding(50257, 1904)
wpe = bitsandbytes.nn.modules.StableEmbedding(50257, 1904)
wte.to("cuda")
wpe.to("cuda")

del wte
del wpe
gc.collect()
torch.cuda.empty_cache()

print(torch.cuda.memory_allocated() / 1024**3)
# 0.7148723602294922
print(torch.cuda.memory_reserved() / 1024**3)
# 0.716796875

Expected behavior

I've attached the following snippet of code which is a simplication for the actual problem I'm facing. I've built a version of GPT that has Stable Embeddings and my use case requires me to constantly be reloading new versions of that model. I find that this leads to a gradual increase in GPU RAM usage over time.

To investigate this I've tried initialising the StableEmbeddings components on their own as per the above code snippet and then deleting them. The issue is that when I delete the variable and then empty out the cuda cache the GPU memory is still occupied by it.