Add first two copilot integration tests #523
Merged
+436
−62
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add invokehttp to dummy data - We'd like to write unit tests that would ideally work equally well with the released docker container as well as for local development. Let's add a well-known malicious library to the data set for testing.
Add data types for pipeline routing, add route for OpenAI - We just had a simple if-else for pipeline routing. Instead, let's add a data type that includes the paths and optionally a
target_url
. We can use that to add a routing for OpenAI which we'll use for testing the Copilot provider. In the future we can make the routings pluggable.When processing forwarded HTTP request, don't process until we have received the whole body - It might happen that the proxied request arrives in two chunks - first just the headers and then the body. In that case we would have sent just the headers with an empty body through the pipeline which might trigger errors like "400 Client Error", then the body would arrive in the next chunk. This patch changes the logic to keep the whole body in the buffer until it is complete and can be processed by the pipeline and sent off to the server.
Add integration tests for the copilot provider -
Since the copilot provider is a proxy, we add a "requester" module that
depending on the provider makes a request either using raw python
requests like earlier or by setting a proxy and using a CA cert file.
To be able to add more tests, we also add more kinds of checks, in
addition to the existing one which makes sure the reply is like the
expected one using cosine distance, we also add checks that make sure
the LLM reply contains or doesn't contain a string.
We use those to add a test that ensures that the copilot provider chat
works and that the copilot chat refuses to generate code snippet with a
malicious package.
To be able to run a subset of tests, we also add the ability to select a
subset of tests based on a provider (
codegate_providers
) or the test name (codegate_test_names
)These serve as the base for further integration tests.
To run them, call:
Related: #402
Fixes: #525
Fixes: #517