lava-v1.6: unsupported operand type(s) for //: 'int' and 'NoneType' #178

jrp2014 · 2025-01-10T21:24:50Z

These models used to work OK, but we now get:

mlx version: 0.22.0.dev20250110+1ce0c0fcb
mlx-vlm version: 0.1.10

vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
Running mlx-community/llava-v1.6-34b-8bit
Fetching 17 files: 100%|█████████████████████████████████████████████████████████████| 17/17 [00:00<00:00, 10828.12it/s]
Fetching 17 files: 100%|█████████████████████████████████████████████████████████████| 17/17 [00:00<00:00, 12417.83it/s]
==========
Image: ['/Users/jrp/Pictures/Processed/20250104-211707_DSC01899.jpg'] 

Prompt: <|im_start|>user
<image>
Provide a factual caption, description and comma-separated keywords or tags for this image so that it can be searched for easily.<|im_end|>
<|im_start|>assistant

Failed to generate output for model at mlx-community/llava-v1.6-34b-8bit: unsupported operand type(s) for //: 'int' and 'NoneType'
********************************************************************************

vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
Running mlx-community/llava-v1.6-mistral-7b-8bit
Fetching 12 files: 100%|█████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 11227.22it/s]
Fetching 12 files: 100%|█████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 26024.64it/s]
==========
Image: ['/Users/jrp/Pictures/Processed/20250104-211707_DSC01899.jpg'] 

Prompt: [INST] <image>
Provide a factual caption, description and comma-separated keywords or tags for this image so that it can be searched for easily. [/INST]
Failed to generate output for model at mlx-community/llava-v1.6-mistral-7b-8bit: unsupported operand type(s) for //: 'int' and 'NoneType'
********************************************************************************

The text was updated successfully, but these errors were encountered:

Blaizzy · 2025-01-11T21:46:30Z

I can't replicate this.

Could you provide the full traceback?

jrp2014 · 2025-01-11T21:57:24Z

Not sure what further traceback I can offer. In my case, the smoke test produces the same thing.

Blaizzy · 2025-01-11T22:23:27Z

There is a full traceback to that error that you are not printing in your tests. I need it to undestand where the error you are getting is located.

I ran the smoke test with that model and it passed on my side:

Blaizzy · 2025-01-11T22:25:46Z

mlx version: 0.22.0.dev20250110+1ce0c0fcb

Please note that you continue to use an unoffical release of mlx

I would recommend you uninstall it and install the official and try again.

pip uninstall mlx
pip install -U mlx

Outside of that we are running the same version of mlx-vlm so it should work.

Blaizzy · 2025-01-11T22:26:15Z

Let me know how if the error persists after you install the official release.

jrp2014 · 2025-01-11T22:33:42Z

It may well be an mlx issue. I expect that there is some way to print the full stack trace to identify the culprit.

python smoke_test.py --models-file llava.txt --image /Users/jrp/Pictures/Processed/20250104-211707_DSC01899.jpg
  0%|                                                                                             | 0/2 [00:00<?, ?it/s]╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Testing mlx-community/llava-v1.6-34b-8bit                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Loading model...
Fetching 17 files: 100%|█████████████████████████████████████████████████████████████| 17/17 [00:00<00:00, 10761.12it/s]
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
Fetching 17 files: 100%|█████████████████████████████████████████████████████████████| 17/17 [00:00<00:00, 14312.16it/s]
✓ Model loaded successfully                                                                      | 0/17 [00:00<?, ?it/s]


Testing vision-language generation...
==========
Image: ['/Users/jrp/Pictures/Processed/20250104-211707_DSC01899.jpg'] 

Prompt: <|im_start|>user
<image>
Describe this image.<|im_end|>
<|im_start|>assistant

✗ vision-language generation failed: unsupported operand type(s) for //: 'int' and 'NoneType'


Testing language-only generation...
==========
Image: None 

Prompt: <|im_start|>user
Hi, how are you?<|im_end|>
<|im_start|>assistant

Hello! I'm just a computer program, I don't have feelings or physical sensations, so I don't have a "how I am." But I'm here to help you with any questions or tasks you may have. How can I assist you today?
==========
Prompt: 15 tokens, 4.075 tokens-per-sec
Generation: 59 tokens, 11.798 tokens-per-sec
Peak memory: 36.997 GB
✓ language-only generation successful


Cleaning up...
✓ Cleanup complete

 50%|██████████████████████████████████████████▌                                          | 1/2 [00:20<00:20, 20.20s/it]╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Testing mlx-community/llava-v1.6-mistral-7b-8bit                                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Loading model...
Fetching 12 files: 100%|█████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 19448.09it/s]
Fetching 12 files: 100%|█████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 32896.50it/s]
✓ Model loaded successfully                                                                      | 0/12 [00:00<?, ?it/s]


Testing vision-language generation...
==========
Image: ['/Users/jrp/Pictures/Processed/20250104-211707_DSC01899.jpg'] 

Prompt: [INST] <image>
Describe this image. [/INST]
✗ vision-language generation failed: unsupported operand type(s) for //: 'int' and 'NoneType'


Testing language-only generation...
==========
Image: None 

Prompt: [INST] Hi, how are you? [/INST]
Hello! I'm just a computer program, so I don't have feelings or emotions. Is there something I can help you with? 
==========
Prompt: 14 tokens, 75.599 tokens-per-sec
Generation: 31 tokens, 60.094 tokens-per-sec
Peak memory: 8.080 GB
✓ language-only generation successful


Cleaning up...
✓ Cleanup complete

100%|█████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:23<00:00, 11.78s/it]


╭────────────────────────────────────────────────────── Results ───────────────────────────────────────────────────────╮
│ ✗ mlx-community/llava-v1.6-34b-8bit                                                                                  │
│ ✗ mlx-community/llava-v1.6-mistral-7b-8bit                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Some models tested failed to test


╭───────────────────────────────────────────────── System Information ─────────────────────────────────────────────────╮
│                                                                                                                      │
│ MAC OS:       v15.2                                                                                                  │
│ Python:       v3.12.7                                                                                                │
│ MLX:          v0.22.0                                                                                                │
│ MLX-VLM:      v0.1.10                                                                                                │
│ Transformers: v4.48.0                                                                                                │
│                                                                                                                      │
│ Hardware:                                                                                                            │
│ • Chip:       Apple M4 Max                                                                                           │
│ • RAM:        128.0 GB                                                                                               │
│ • CPU Cores:  16                                                                                                     │
│ • GPU Cores:  40                                                                                                     │
│                                                                                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Blaizzy · 2025-01-11T22:52:47Z

You will need to install mlx-vlm from source and change the test_smoke.py file. (I will make the necessary change to display full traceback on the next release)

Instead, you could:

run using mlx_vlm.generate in your terminal and get the full traceback.
or install mlx as I suggested earlier and see if it solves your issue.

jrp2014 · 2025-01-11T22:55:10Z

I have done a fuller traceback, and it seems that there issue is in transformers. I do seem to recall that there was a warning, running earlier versions.

python smoke_test.py --models-file llava.txt --image /Users/jrp/Pictures/Processed/20250104-211707_DSC01899.jpg
  0%|                                                                                             | 0/2 [00:00<?, ?it/s]╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Testing mlx-community/llava-v1.6-34b-8bit                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Loading model...
Fetching 17 files: 100%|█████████████████████████████████████████████████████████████| 17/17 [00:00<00:00, 11987.76it/s]
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
Fetching 17 files: 100%|█████████████████████████████████████████████████████████████| 17/17 [00:00<00:00, 10993.40it/s]
✓ Model loaded successfully                                                                      | 0/17 [00:00<?, ?it/s]


Testing vision-language generation...
==========
Image: ['/Users/jrp/Pictures/Processed/20250104-211707_DSC01899.jpg'] 

Prompt: <|im_start|>user
<image>
Describe this image.<|im_end|>
<|im_start|>assistant

✗ vision-language generation failed: unsupported operand type(s) for //: 'int' and 'NoneType'
Traceback (most recent call last):
  File "/Users/jrp/Documents/AI/mlx/scripts/vlm/smoke_test.py", line 110, in test_generation
    output = generate(**generate_args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/mlx/lib/python3.12/site-packages/mlx_vlm/utils.py", line 1101, in generate
    for response in stream_generate(model, processor, prompt, image, **kwargs):
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/mlx/lib/python3.12/site-packages/mlx_vlm/utils.py", line 1015, in stream_generate
    inputs = prepare_inputs(
             ^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/mlx/lib/python3.12/site-packages/mlx_vlm/utils.py", line 812, in prepare_inputs
    inputs = processor(
             ^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/mlx/lib/python3.12/site-packages/transformers/models/llava_next/processing_llava_next.py", line 162, in __call__
    num_image_tokens = self._get_number_of_features(orig_height, orig_width, height, width)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/mlx/lib/python3.12/site-packages/transformers/models/llava_next/processing_llava_next.py", line 181, in _get_number_of_features
    patches_height = height // self.patch_size
                     ~~~~~~~^^~~~~~~~~~~~~~~~~
TypeError: unsupported operand type(s) for //: 'int' and 'NoneType'



Testing language-only generation...
==========
Image: None 

Prompt: <|im_start|>user
Hi, how are you?<|im_end|>
<|im_start|>assistant

Hello! I'm an AI language model, so I don't have feelings or personal experience, but I'm always ready to assist you with any questions you may have. How can I help?
==========
Prompt: 15 tokens, 5.109 tokens-per-sec
Generation: 43 tokens, 11.431 tokens-per-sec
Peak memory: 36.997 GB
✓ language-only generation successful


Cleaning up...
✓ Cleanup complete

 50%|██████████████████████████████████████████▌                                          | 1/2 [00:15<00:15, 15.92s/it]╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Testing mlx-community/llava-v1.6-mistral-7b-8bit                                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Loading model...
Fetching 12 files: 100%|██████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 9843.86it/s]
Fetching 12 files: 100%|█████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 27654.75it/s]
✓ Model loaded successfully                                                                      | 0/12 [00:00<?, ?it/s]


Testing vision-language generation...
==========
Image: ['/Users/jrp/Pictures/Processed/20250104-211707_DSC01899.jpg'] 

Prompt: [INST] <image>
Describe this image. [/INST]
✗ vision-language generation failed: unsupported operand type(s) for //: 'int' and 'NoneType'
Traceback (most recent call last):
  File "/Users/jrp/Documents/AI/mlx/scripts/vlm/smoke_test.py", line 110, in test_generation
    output = generate(**generate_args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/mlx/lib/python3.12/site-packages/mlx_vlm/utils.py", line 1101, in generate
    for response in stream_generate(model, processor, prompt, image, **kwargs):
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/mlx/lib/python3.12/site-packages/mlx_vlm/utils.py", line 1015, in stream_generate
    inputs = prepare_inputs(
             ^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/mlx/lib/python3.12/site-packages/mlx_vlm/utils.py", line 812, in prepare_inputs
    inputs = processor(
             ^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/mlx/lib/python3.12/site-packages/transformers/models/llava_next/processing_llava_next.py", line 162, in __call__
    num_image_tokens = self._get_number_of_features(orig_height, orig_width, height, width)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniconda/base/envs/mlx/lib/python3.12/site-packages/transformers/models/llava_next/processing_llava_next.py", line 181, in _get_number_of_features
    patches_height = height // self.patch_size
                     ~~~~~~~^^~~~~~~~~~~~~~~~~
TypeError: unsupported operand type(s) for //: 'int' and 'NoneType'

Blaizzy · 2025-01-11T22:57:52Z

I see, you probably running the latest transformers.

Let me try something to see if it fixes it.

Blaizzy · 2025-01-11T23:00:41Z

Try again now :)

jrp2014 · 2025-01-11T23:57:44Z

No change., I'm afraid. Looking back at earlier runs, I see that there was a deprecation warning:

``
Expanding inputs for image tokens in LLaVa-NeXT should be done in processing. Please add patch_size and `vision_feature_select_strategy` to the model's processing config or set directly with `processor.patch_size = {{patch_size}}` and processor.vision_feature_select_strategy = {{vision_feature_select_strategy}}`. Using processors without these attributes in the config is deprecated and will throw an error in v4.50.

Blaizzy · 2025-01-12T00:35:58Z

I just fixed that

Download a fresh copy of the model weights :)

Blaizzy · 2025-01-12T00:37:02Z

https://huggingface.co/mlx-community/llava-v1.6-mistral-7b-8bit/commit/b8df5f329d95a7abe6429ed46093f9b84e8e6396

jrp2014 · 2025-01-12T21:30:31Z

Thanks. That seems to fix mlx-community/llava-v1.6-mistral-7b-8bit. Is the problem the same for mlx-community/llava-v1.6-34b-8bit?

Blaizzy · 2025-01-12T21:51:05Z

My pleasure!

I think so

I will update all models.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lava-v1.6: unsupported operand type(s) for //: 'int' and 'NoneType' #178

lava-v1.6: unsupported operand type(s) for //: 'int' and 'NoneType' #178

jrp2014 commented Jan 10, 2025

Blaizzy commented Jan 11, 2025

jrp2014 commented Jan 11, 2025 •

edited

Loading

Blaizzy commented Jan 11, 2025

Blaizzy commented Jan 11, 2025

Blaizzy commented Jan 11, 2025

jrp2014 commented Jan 11, 2025

Blaizzy commented Jan 11, 2025

jrp2014 commented Jan 11, 2025

Blaizzy commented Jan 11, 2025

Blaizzy commented Jan 11, 2025

jrp2014 commented Jan 11, 2025 •

edited

Loading

Blaizzy commented Jan 12, 2025 •

edited

Loading

Blaizzy commented Jan 12, 2025

jrp2014 commented Jan 12, 2025

Blaizzy commented Jan 12, 2025

lava-v1.6: unsupported operand type(s) for //: 'int' and 'NoneType' #178

lava-v1.6: unsupported operand type(s) for //: 'int' and 'NoneType' #178

Comments

jrp2014 commented Jan 10, 2025

Blaizzy commented Jan 11, 2025

jrp2014 commented Jan 11, 2025 • edited Loading

Blaizzy commented Jan 11, 2025

Blaizzy commented Jan 11, 2025

Blaizzy commented Jan 11, 2025

jrp2014 commented Jan 11, 2025

Blaizzy commented Jan 11, 2025

jrp2014 commented Jan 11, 2025

Blaizzy commented Jan 11, 2025

Blaizzy commented Jan 11, 2025

jrp2014 commented Jan 11, 2025 • edited Loading

Blaizzy commented Jan 12, 2025 • edited Loading

Blaizzy commented Jan 12, 2025

jrp2014 commented Jan 12, 2025

Blaizzy commented Jan 12, 2025

jrp2014 commented Jan 11, 2025 •

edited

Loading

jrp2014 commented Jan 11, 2025 •

edited

Loading

Blaizzy commented Jan 12, 2025 •

edited

Loading