Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add first two copilot integration tests #523

Merged
merged 4 commits into from
Jan 9, 2025
Merged

Conversation

jhrozek
Copy link
Contributor

@jhrozek jhrozek commented Jan 9, 2025

  • Add invokehttp to dummy data - We'd like to write unit tests that would ideally work equally well with the released docker container as well as for local development. Let's add a well-known malicious library to the data set for testing.

  • Add data types for pipeline routing, add route for OpenAI - We just had a simple if-else for pipeline routing. Instead, let's add a data type that includes the paths and optionally a target_url. We can use that to add a routing for OpenAI which we'll use for testing the Copilot provider. In the future we can make the routings pluggable.

  • When processing forwarded HTTP request, don't process until we have received the whole body - It might happen that the proxied request arrives in two chunks - first just the headers and then the body. In that case we would have sent just the headers with an empty body through the pipeline which might trigger errors like "400 Client Error", then the body would arrive in the next chunk. This patch changes the logic to keep the whole body in the buffer until it is complete and can be processed by the pipeline and sent off to the server.

  • Add integration tests for the copilot provider -

Since the copilot provider is a proxy, we add a "requester" module that
depending on the provider makes a request either using raw python
requests like earlier or by setting a proxy and using a CA cert file.

To be able to add more tests, we also add more kinds of checks, in
addition to the existing one which makes sure the reply is like the
expected one using cosine distance, we also add checks that make sure
the LLM reply contains or doesn't contain a string.

We use those to add a test that ensures that the copilot provider chat
works and that the copilot chat refuses to generate code snippet with a
malicious package.

To be able to run a subset of tests, we also add the ability to select a
subset of tests based on a provider (codegate_providers) or the test name (codegate_test_names)

These serve as the base for further integration tests.

To run them, call:

CODEGATE_PROVIDERS=copilot \
CA_CERT_FILE=/Users/you/devel/codegate/codegate_volume/certs/ca.crt \
ENV_COPILOT_KEY=your-openapi-key \
python tests/integration/integration_tests.py

Related: #402
Fixes: #525
Fixes: #517

We'd like to write unit tests that would ideally work equally well with
the released docker container as well as for local development. Let's
add a well-known malicious library to the data set for testing.
We just had a simple if-else for pipeline routing. Instead, let's add a
data type that includes the paths and optionally a `target_url`. We can
use that to add a routing for OpenAI which we'll use for testing the
Copilot provider.

In the future we can make the routings pluggable.
…eceived the whole body

It might happen that the proxied request arrives in two chunks - first
just the headers and then the body. In that case we would have sent just
the headers with an empty body through the pipeline which might trigger
errors like "400 Client Error", then the body would arrive in the next
chunk.

This patch changes the logic to keep the whole body in the buffer until
it is complete and can be processed by the pipeline and sent off to the
server.

Fixes: stacklok#517
Since the copilot provider is a proxy, we add a "requester" module that
depending on the provider makes a request either using raw python
requests like earlier or by setting a proxy and using a CA cert file.

To be able to add more tests, we also add more kinds of checks, in
addition to the existing one which makes sure the reply is like the
expected one using cosine distance, we also add checks that make sure
the LLM reply contains or doesn't contain a string.

We use those to add a test that ensures that the copilot provider chat
works and that the copilot chat refuses to generate code snippet with a
malicious package.

To be able to run a subset of tests, we also add the ability to select a
subset of tests based on a provider (`codegate_providers`) or the test name (`codegate_test_names`)

These serve as the base for further integration tests.

To run them, call:
```
CODEGATE_PROVIDERS=copilot \
CA_CERT_FILE=/Users/you/devel/codegate/codegate_volume/certs/ca.crt \
ENV_COPILOT_KEY=your-openapi-key \
python tests/integration/integration_tests.py
```

Related: stacklok#402
@ptelang
Copy link
Contributor

ptelang commented Jan 9, 2025

Looks great! Thanks for refactoring the test framework code.

@jhrozek jhrozek merged commit ff4a3a7 into stacklok:main Jan 9, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants