Int4 inference

[dmenig]

Is there any plan to release an int4 version ? Specifically, I'm interested in the video understanding part.

[xffxff]

Yes! You can check out https://github.com/mobiusml/hqq/blob/master/examples/hf/aria_multimodal.py, a HQQ 4-bit version. This implementation is about 4-6x faster and takes 3x less VRAM!

[AIWarper]

Yes! You can check out https://github.com/mobiusml/hqq/blob/master/examples/hf/aria_multimodal.py, a HQQ 4-bit version. This implementation is about 4-6x faster and takes 3x less VRAM!

I was looking into this as I need a local Video Vision solution - is it correct to assume the model was 12 shards at ~4gb a shard?

Also, you mentioned 3x less VRAM - what are the upper limits of the video vision you can achieve on 24gb or less?

EDIT: I tested locally with a 4090. Single image inference was fine but any video OOMed me no matter the length or resolution

[h123fire]

Error: accessing tensor output of CUDAGraphs that has been overwritten by a subsequent run. Stack trace,/transformers/models/llama/modeling_llama.py", line 1190, in forward