How does this Semantic Router Works?

[shrijayan]

Is this Architecture correct?

[shrijayan]

Is this using the technique of Dynamic Few-shot prompting?

[andreped]

Hello, @shrijayan :] Let me try to explain.

Semantic-Router does not use an LLM for routing. No prompt involved either.

You first construct a set of utterances defining the different routes. The utterances you create embeddings for and tag which belong to which route. Then when a new user question is given to the route, it embeds the user question and compared it to the different routes. Based on some algorithm, it classifies the question to one of the routes. The core concept is to route the user question based on how semantically similar a user question is to one of the given routes, defined by the corpus of utterances. Hence the name Semantic-Router.

The algorithm varies depending on which Route Layer implementation you use. The simplest is to route using RouteLayer which uses simple k nearest neighbour (kNN) for the final classification. The user question is embedded, and compared against the other embedded utterances in the embedded spaces for instance using cosine distance. Then based on the k closest utterances, it classifies the user question to one of the routes.

The actual voting of how to weight these k closest values can be adjusted, including how many (k in topK) and which aggregation method. For older releases it used to be possible to do something like this:

from semantic_router.encoders import AzureOpenAIEncoder, TfidfEncoder
from semantic_router.hybrid_layer import HybridRouteLayer

model = HybridRouteLayer(
    encoder=AzureOpenAIEncoder(...),
    sparse_encoder=TfidfEncoder(),
    routes=routes,
    alpha=0.3,  # default = 0.3
    top_k=3,  # default = 5
    aggregation: str = "mean",  # default = "sum"
)

Here using the HybridRouteLayer instead of the basic one, where the Hybrid uses both embeddings (semantic search) and keyword search.

---

EDIT: Note that the actual route methods available may have changed, introducing an LLM for routing, but the regular methods you have likely tested only use basic embedding models, which do not involve any LLMs or prompting or anything like that. Just simple embedding computation using a retrained embedding model and basic math :]

[shrijayan]

Thank you so much! This was so clear and straight to the point.

[shrijayan]

This how it works I guess