LoRA Adapters for vLLM & support for s3, gs, oss for pulling adapters and models (to cache) from buckets

[nstogner]

• Add .spec.adapters
• Support requesting adapters using the pattern: {"model": "<model>_<adapter>", ... }
• Load LoRA adapters into running vLLM containers
• Support updating LoRA adapters without needing to restart vLLM
• Rewrite .model to use adapter in chat request body when proxying to vLLM
• Add adapters to model list
• Add support for s3://, gs://, oss:// urls (for adapters and cache loading)
• Add new cloud credentials to support new urls
• Update docs
• Update Model validation

NOTE:

• Was unable to test oss:// urls... Had issues opening acct.

FOLLOWUP:

• Need to add adapter e2e tests (have not found a small enough model with adapters for use in kind cluster)
• Need to update chart values.yaml to include GH-actions-built image for model loader after merge!!!

Fixes #132, #303

[alpe]

Nice drawing. This is very helpful! 🌻
I am not super familiar with LoRa adapters but they can have significant size from what I saw. Caching seems a good idea. For the non-cache scenario, I would suggest to have no-cache or container-managed profile so that it does not look like the default to skip this.
With on-demand LoRa, the disk size may become a problem at some point. This is off scope but purge job or retention time can be things that need to be configured at some point in the profile.

[samos123]

Can you show an example that has the url field? I'm assuming the url field must be used to specify the base model?

[nstogner]

@samos123 I currently have all examples in the diagrams

[samos123]

That's where I looked but none of them have the base model URL set?

[nstogner]

Model .spec.url would be the same as normal.

[nstogner]

Note, it looks like vLLM supports loading adapters from huggingface: vllm-project/vllm#6234

[nstogner]

Note, vLLM has an endpoint to support dynamic loading/unloading of adapters: vllm-project/vllm#6566

[samos123]

does vLLM support directly loading this adapter from HF or is it a hard requirement to download the lora adapter first?

[nstogner]

vLLM can load it from HF but not S3

[nstogner]

See: https://github.com/vllm-project/vllm/blob/d1c6799b8870e513bf4f2305cbf6cda9fc3d773b/vllm/lora/utils.py#L178

[samos123]

Not sure whether this should work or not. So sharing what I did and the error message I get:

git checkout $THIS_PR
helm install kubeai ./charts/kubeai

k get pods
NAME                         READY   STATUS             RESTARTS     AGE
kubeai-6bf98d5b77-txmpb      0/1     CrashLoopBackOff   1 (2s ago)   5s
openwebui-69ffb7dbb4-xcvlj   0/1     Running            0            5s

k logs -f kubeai-6bf98d5b77-txmpb 
2024-11-24T06:37:29Z    INFO    manager run finished
2024-11-24T06:37:29Z    ERROR   manager failed to run command   {"error": "invalid config: Key: 'System.ModelLoaders' Error:Field validation for 'ModelLoaders' failed on the 'required' tag"}
main.main
        /workspace/cmd/main.go:50
runtime.main
        /usr/local/go/src/runtime/proc.go:271

I was trying to play around with the branch and see how the helm validation worked.

[samos123]

I didn't finish all. Leaving partial feedback, will continue more tomorrow.

[nstogner]

Not sure whether this should work or not. So sharing what I did and the error message I get:

I was trying to play around with the branch and see how the helm validation worked.

Here is what I used to make sure that the most recent image was running:

gcloud container clusters create-auto cluster-1 \
    --location=us-central1

skaffold run -f ./skaffold.yaml --profile kubeai-only-gke --default-repo us-central1-docker.pkg.dev/substratus-dev

[alpe]

nice work!