-
Notifications
You must be signed in to change notification settings - Fork 70
LCORE-1223: Add e2e tests for rlsapi v1 /infer endpoint #1067
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
major
wants to merge
1
commit into
lightspeed-core:main
Choose a base branch
from
major:LCORE-1223-rlsapi-v1-e2e-tests
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
| @Authorized | ||
| Feature: rlsapi v1 /infer endpoint API tests | ||
|
|
||
| Background: | ||
| Given The service is started locally | ||
| And REST API service prefix is /v1 | ||
|
|
||
| Scenario: Basic inference with minimal request (question only) | ||
| Given The system is in default state | ||
| And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva | ||
| When I use "infer" to ask question with authorization header | ||
| """ | ||
| {"question": "How do I list files in Linux?"} | ||
| """ | ||
| Then The status code of the response is 200 | ||
| And The rlsapi response should have valid structure | ||
|
|
||
| Scenario: Inference with full context (systeminfo populated) | ||
| Given The system is in default state | ||
| And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva | ||
| When I use "infer" to ask question with authorization header | ||
| """ | ||
| {"question": "How do I configure SELinux?", "context": {"systeminfo": {"os": "RHEL", "version": "9.3", "arch": "x86_64"}}} | ||
| """ | ||
| Then The status code of the response is 200 | ||
| And The rlsapi response should have valid structure | ||
|
|
||
| Scenario: Request without authorization returns 401 | ||
| Given The system is in default state | ||
| When I use "infer" to ask question | ||
| """ | ||
| {"question": "How do I list files?"} | ||
| """ | ||
| Then The status code of the response is 401 | ||
| And The body of the response is the following | ||
| """ | ||
| { | ||
| "detail": { | ||
| "response": "Missing or invalid credentials provided by client", | ||
| "cause": "No Authorization header found" | ||
| } | ||
| } | ||
| """ | ||
|
|
||
| Scenario: Request with empty bearer token returns 401 | ||
| Given The system is in default state | ||
| And I set the Authorization header to Bearer | ||
| When I use "infer" to ask question with authorization header | ||
| """ | ||
| {"question": "How do I list files?"} | ||
| """ | ||
| Then The status code of the response is 401 | ||
| And The body of the response contains No token found in Authorization header | ||
|
|
||
| Scenario: Empty/whitespace question returns 422 | ||
| Given The system is in default state | ||
| And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva | ||
| When I use "infer" to ask question with authorization header | ||
| """ | ||
| {"question": " "} | ||
| """ | ||
| Then The status code of the response is 422 | ||
| And The body of the response contains Question cannot be empty | ||
|
|
||
| Scenario: Response contains valid structure (data.text, data.request_id) | ||
| Given The system is in default state | ||
| And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva | ||
| When I use "infer" to ask question with authorization header | ||
| """ | ||
| {"question": "What is RHEL?"} | ||
| """ | ||
| Then The status code of the response is 200 | ||
| And The rlsapi response should have valid structure | ||
|
|
||
| Scenario: Multiple requests generate unique request_ids | ||
| Given The system is in default state | ||
| And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva | ||
| When I use "infer" to ask question with authorization header | ||
| """ | ||
| {"question": "First question"} | ||
| """ | ||
| Then The status code of the response is 200 | ||
| And I store the rlsapi request_id | ||
| When I use "infer" to ask question with authorization header | ||
| """ | ||
| {"question": "Second question"} | ||
| """ | ||
| Then The status code of the response is 200 | ||
| And The rlsapi request_id should be different from the stored one | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| @RBAC | ||
| Feature: rlsapi v1 /infer endpoint error response tests | ||
|
|
||
| Tests for error conditions on the rlsapi v1 /infer endpoint including | ||
| authorization failures (403) and service unavailability (503). | ||
|
|
||
| Background: | ||
| Given The service is started locally | ||
| And REST API service prefix is /v1 | ||
|
|
||
| # ============================================ | ||
| # Authorization - 403 Forbidden | ||
| # ============================================ | ||
|
|
||
| Scenario: User without rlsapi_v1_infer permission returns 403 | ||
| Given The system is in default state | ||
| And I authenticate as "viewer" user | ||
| When I use "infer" to ask question with authorization header | ||
| """ | ||
| {"question": "How do I list files?"} | ||
| """ | ||
| Then The status code of the response is 403 | ||
| And The body of the response contains does not have permission | ||
|
|
||
| Scenario: User with rlsapi_v1_infer permission can access endpoint | ||
| Given The system is in default state | ||
| And I authenticate as "user" user | ||
| When I use "infer" to ask question with authorization header | ||
| """ | ||
| {"question": "How do I list files?"} | ||
| """ | ||
| Then The status code of the response is 200 | ||
| And The rlsapi response should have valid structure | ||
|
|
||
| # ============================================ | ||
| # Service Unavailable - 503 | ||
| # ============================================ | ||
|
|
||
| @skip-in-library-mode | ||
| Scenario: Returns 503 when llama-stack connection is broken | ||
| Given The system is in default state | ||
| And I authenticate as "user" user | ||
| And The llama-stack connection is disrupted | ||
| When I use "infer" to ask question with authorization header | ||
| """ | ||
| {"question": "How do I list files?"} | ||
| """ | ||
| Then The status code of the response is 503 | ||
| And The body of the response contains Llama Stack |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| """rlsapi v1 endpoint test steps.""" | ||
|
|
||
| from behave import then, step # pyright: ignore[reportAttributeAccessIssue] | ||
| from behave.runner import Context | ||
|
|
||
|
|
||
| @then("The rlsapi response should have valid structure") | ||
| def check_rlsapi_response_structure(context: Context) -> None: | ||
| """Check that rlsapi v1 response has valid structure. | ||
|
|
||
| Validates that the response contains: | ||
| - data.text (non-empty string) | ||
| - data.request_id (non-empty string) | ||
| """ | ||
| assert context.response is not None, "Request needs to be performed first" | ||
| response_json = context.response.json() | ||
|
|
||
| assert "data" in response_json, "Response missing 'data' field" | ||
| data = response_json["data"] | ||
|
|
||
| assert "text" in data, "Response data missing 'text' field" | ||
| assert isinstance(data["text"], str), "data.text must be a string" | ||
| assert len(data["text"]) > 0, "data.text must not be empty" | ||
|
|
||
| assert "request_id" in data, "Response data missing 'request_id' field" | ||
| assert isinstance(data["request_id"], str), "data.request_id must be a string" | ||
| assert len(data["request_id"]) > 0, "data.request_id must not be empty" | ||
|
|
||
|
|
||
| @step("I store the rlsapi request_id") | ||
| def store_rlsapi_request_id(context: Context) -> None: | ||
| """Store the request_id from rlsapi response for later comparison.""" | ||
| assert context.response is not None, "Request needs to be performed first" | ||
| response_json = context.response.json() | ||
|
|
||
| assert "data" in response_json, "Response missing 'data' field" | ||
| assert "request_id" in response_json["data"], "Response data missing 'request_id'" | ||
| assert isinstance( | ||
| response_json["data"]["request_id"], str | ||
| ), "data.request_id must be a string" | ||
| assert ( | ||
| len(response_json["data"]["request_id"]) > 0 | ||
| ), "data.request_id must not be empty" | ||
|
|
||
| context.stored_request_id = response_json["data"]["request_id"] | ||
|
|
||
|
|
||
| @then("The rlsapi request_id should be different from the stored one") | ||
| def check_rlsapi_request_id_different(context: Context) -> None: | ||
| """Verify that the current request_id differs from the stored one.""" | ||
| assert context.response is not None, "Request needs to be performed first" | ||
| assert hasattr(context, "stored_request_id"), "No request_id was stored previously" | ||
|
|
||
| response_json = context.response.json() | ||
| assert "data" in response_json, "Response missing 'data' field" | ||
| assert "request_id" in response_json["data"], "Response data missing 'request_id'" | ||
|
|
||
| current_request_id = response_json["data"]["request_id"] | ||
| assert isinstance(current_request_id, str), "data.request_id must be a string" | ||
| assert len(current_request_id) > 0, "data.request_id must not be empty" | ||
| stored_request_id = context.stored_request_id | ||
|
|
||
| assert ( | ||
| current_request_id != stored_request_id | ||
| ), f"request_id should be unique, but got same value: {current_request_id}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am missing here the cases for 403 and 429 status codes,
you can acheive 403 when using different bearer token.
For 429, the quota is configurable in configuration file.
Also, add a test for 503, with broken llama-stack connection
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@radofuchs At the moment, we don't have 429 responses because we don't currently limit via a quota (it's not in our API spec). Should I do the 403 and 503 errors?