feat: enable remote pilot logging system [MISSING AUTH] #269
Conversation
|
@chrisburr I think we need a pilot logging router, even in its basic form. One of the tests fails because it is missing. |
|
@chrisburr What is a reason of test failures here ? The demo runs fine locally, can be contacted with a client etc. The test fails with |
|
@chrisburr @chaen This type of error occurs (in the demo) where charts don't list a relevant DB Both local and CI tests reach the same point - server installation failure. |
|
@chaen I fixed the database problem, now integration tests are passing, however I'm getting gubbins errors in unit tests. Is it an internal gubbins problem ? |
martynia
force-pushed
the
main
branch
2 times, most recently
from
3b0ea4c to
6eb1ebd
Compare
| # here, users with privileged properties will see logs from all VOs. Is it what we want ? | ||
| search_params = [{"parameter": "PilotID", "operator": "eq", "value": pilot_id}] | ||
| if _non_privileged(user_info): | ||
| search_params.append( | ||
| {"parameter": "VO", "operator": "eq", "value": user_info.vo} | ||
| ) | ||
| result = await db.search( | ||
| ["Message"], | ||
| search_params, | ||
| [{"parameter": "LineNumber", "direction": "asc"}], | ||
| ) | ||
| if not result: | ||
| return [{"Message": f"No logs for pilot ID = {pilot_id}"}] |
There was a problem hiding this comment.
Similarly, this bloc of code should be a method of diracx-dbs/src/diracx/db/sql/pilots/pilot_agents.py PilotAgentsDB
diracx-routers/src/diracx/routers/pilot_logging/remote_logger.py
Outdated
Show resolved
Hide resolved
| def _non_privileged(user_info: AuthorizedUserInfo): | ||
| return ( | ||
| SERVICE_ADMINISTRATOR not in user_info.properties | ||
| and OPERATOR not in user_info.properties | ||
| ) |
There was a problem hiding this comment.
I feel like this should be moved in the access_policies.py function? May be by adding an action type? Or by adding some logic like in:
diracx/diracx-routers/src/diracx/routers/jobs/access_policies.py
Lines 79 to 82 in 320c554
There was a problem hiding this comment.
I had a logical problem what to do here. I would like to search only once (and delete by query for a delete operation w/o prior searching). This is a reason why I removed the DB argument from a policy call. The problem is more visible in the delete case:
- I could search in the policy for a given client VO and if pilot IDs are not from the VO throw an exception (not sure what to do if some IDs are and some are not from the VO the client has - for a bulk PilotID search)
- then delete by query (which searches again)
Or could I use the AgentsDB to match the VO ? Still an extra DB query, this time and SQL one.
I opted for a restricted silent delete and and a restricted search by adding the VO, but outside the policy.
|
|
||
| @router.delete("/logs") | ||
| async def delete( | ||
| pilot_id: int, |
There was a problem hiding this comment.
We also tend to make bulk requests to the db to improve the performances of the system. I would take a list of pilot_ids as inputs instead of a single pilot_id.
Then you can return sucess and failures as it is done for the jobs for instance:
diracx/diracx-routers/src/diracx/routers/jobs/status.py
Lines 35 to 62 in 320c554
|
|
||
| @router.get("/logs") | ||
| async def get_logs( | ||
| pilot_id: int, |
There was a problem hiding this comment.
Similarly,
| pilot_id: int, | |
| pilot_ids: List[int], |
| if _non_privileged(user_info): | ||
| search_params.append(non_privil_params) | ||
| await db.delete(search_params) | ||
| message = f"Logs for for pilots with submission data >='{data.min}' successfully deleted" |
There was a problem hiding this comment.
| message = f"Logs for for pilots with submission data >='{data.min}' successfully deleted" | |
| message = f"Logs for pilots with submission data >='{data.min}' successfully deleted" |
| logger.warning(f"Retrieving logs for pilot ID '{pilot_id}'") | ||
| user_info = await check_permissions(action=ActionType.QUERY) | ||
|
|
||
| # here, users with privileged properties will see logs from all VOs. Is it what we want ? |
martynia
force-pushed
the
main
branch
2 times, most recently
from
378ee3b to
4ee88a1
Compare
chaen
changed the title
|
Waiting for the pilot authentication to be ready |
Enable remote logging system with open search. This PR only shows a router in its preliminary form.
The router defines following operations:
@router.post("/")- bulk insert log record sent by a pilot. The method returns apilotID, so it can use it next time to insert messaged w/o consultingPilotAgentsDBto get PilotID fromPilotStamp.@router.get("/logs")- to retrieve logs for a given pilot_idPolicies allow pilots to store VO-aware logs. Normal users can only retrieve log for their own VOs. diracAdmin can retrieve logs for any VO.