VHELM setup

[smsarov]

I've followed all the steps for VHELM setup and got the error:

(crfm-helm) smsarov@MacBook-Air-Alexander diploma % helm-run --run-entries mmmu:subject=Accounting,model=openai/gpt-4o-mini-2024-07-18 --suite my-vhelm-suite --max-eval-instances 10

main {
Reading tokenizer configs from /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/helm/config/tokenizer_configs.yaml...
Reading model deployments from /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/helm/config/model_deployments.yaml...
} [0.717s]
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.10/bin/helm-run", line 8, in
sys.exit(main())
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/helm/common/hierarchical_logger.py", line 104, in wrapper
return fn(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/helm/benchmark/run.py", line 332, in main
run_specs = run_entries_to_run_specs(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/helm/benchmark/run.py", line 43, in run_entries_to_run_specs
for run_spec in construct_run_specs(parse_object_spec(entry.description)):
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/helm/benchmark/run_spec_factory.py", line 64, in construct_run_specs
run_specs: List[RunSpec] = [run_spec_function(**args)]
TypeError: get_mmmu_spec() missing 1 required positional argument: 'question_type'

I tried running quick start for usual helm and it worked as expected

[ImKeTT]

Thanks for your interest in using the framework! There are two question types in MMMU: open and multiple-choice, you have to pass question_type=multiple-choice when testing on the task.
You can refer to the run_entries_vhelm.conf for other args in vhelm tasks.

helm/src/helm/benchmark/presentation/run_entries_vhelm.conf

Line 82 in ee10b8f

{description: "mmmu:subject=Accounting,question_type=multiple-choice,model=vlm", priority: 1}