Consider supporting a reranker model?

[qiankunli]

No description provided.

[samos123]

Do you have a specific
reranker API, model and engine in mind?

OpenAI doesn't provide a reranker API. I am not too familiar with reranking use case. So please provide as much context and details as possible.

[qiankunli]

Do you have a specific reranker API, model and engine in mind?

OpenAI doesn't provide a reranker API. I am not too familiar with reranking use case. So please provide as much context and details as possible.

The reranker model is an important part of RAG (Retrieval-Augmented Generation), and we hope to operate a reranker model (for example, BAAI/bge-reranker-v2-m3) and provide a rerank API.

You can refer to the rerank API provided by Cohere: Rerank API.
@samos123

[samos123]

Edit: The docs call out that Infinity adheres to Cohere API, which is great. I think maybe that's our shortest route to support reranker models.

@michaelfeil what's the API that Infinity provides? https://github.com/michaelfeil/infinity?tab=readme-ov-file#reranking

I'm thinking maybe we can re-use Infinity support for adding support for reranker models.

[michaelfeil]

Reranker API’s don’t have a common ground in their spec. E.g. LiteLLM and I followed Cohere.

[samos123]

Seems Cohere is also supported in other solutions: https://docs.continue.dev/customize/model-types/reranking

I think supporting a Reranker API would be great for fully local private code completion with continue.dev. We support all endpoints (chat completions, code completion, embeddings) except Reranker.

[qiankunli]

Seems Cohere is also supported in other solutions: https://docs.continue.dev/customize/model-types/reranking

I think supporting a Reranker API would be great for fully local private code completion with continue.dev. We support all endpoints (chat completions, code completion, embeddings) except Reranker.

I'm hopeful that this plan will be put into action.

[samos123]

It's definitely on the roadmap and I think I have all the info I would need. Right now our focus is on adding support for PVC support and Prefix Cache aware routing. Afterwards we should be able to tackle this.