Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API: improve database queries #287

Merged
merged 2 commits into from
Dec 12, 2024
Merged

API: improve database queries #287

merged 2 commits into from
Dec 12, 2024

Conversation

jmaupetit
Copy link
Member

Purpose

The request user is injected in every API endpoint. Not prefetching related objects such as groups and operational units generates a huge number of unexpected queries and slows down API response time.

Proposal

  • prefetch user-related groups and operational units
  • improve bulk endpoints permissions checking

Decrease the number of database queries by prefetching groups and
related operational units linked to the request user.
Refactored a bit dynamic data linking to points of charge and
permissions using sets.
@jmaupetit jmaupetit self-assigned this Dec 12, 2024
@jmaupetit jmaupetit added improvement API Item related to the API service labels Dec 12, 2024
@jmaupetit
Copy link
Member Author

This is work is related to #179

Copy link

Current benchmark

Type Name Request Count Failure Count Median Response Time Average Response Time Min Response Time Max Response Time Average Content Size Requests/s Failures/s 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% git timestamp version
GET /auth/whoami 29 0 680.0 763.4085177241378 14.024212000009584 2229.70069500002 160.0 0.9994515846752628 0.0 680 950 1100 1200 1600 2200 2200 2200 2200 2200 2200 f85ecea 2024-12-12 16:35:55.849929+00:00 0.16.0
POST /dynamique/session/ 21 0 480.0 778.5714072857145 12.520840999997064 2830.3657170000065 45.0 0.72374080269588 0.0 480 850 1000 1300 2200 2500 2800 2800 2800 2800 2800 f85ecea 2024-12-12 16:35:55.849929+00:00 0.16.0
POST /dynamique/session/bulk 31 0 790.0 834.4799234838657 39.88664299998845 2165.2583300000288 411.0 1.0683792801701086 0.0 790 1200 1400 1400 1600 2000 2200 2200 2200 2200 2200 f85ecea 2024-12-12 16:35:55.849929+00:00 0.16.0
GET /dynamique/status/ 30 0 400.0 753.8158720999994 15.38997299999778 2545.4519589999904 285.0 1.0339154324226858 0.0 480 960 1400 1600 1900 2000 2500 2500 2500 2500 2500 f85ecea 2024-12-12 16:35:55.849929+00:00 0.16.0
POST /dynamique/status/ 33 0 920.0 1008.6386725454528 15.98396800000046 2552.004898000007 45.0 1.1373069756649543 0.0 920 1300 1600 2000 2100 2300 2600 2600 2600 2600 2600 f85ecea 2024-12-12 16:35:55.849929+00:00 0.16.0
GET /dynamique/status/FRALLEGO002006P3 34 0 380.0 654.8573842058871 15.61049899999034 2178.0332320000184 276.7058823529412 1.1717708234123771 0.0 500 790 1100 1300 1600 2000 2200 2200 2200 2200 2200 f85ecea 2024-12-12 16:35:55.849929+00:00 0.16.0
GET /dynamique/status/FRALLEGO002006P3/history 28 0 600.0 1035.97130821429 15.77455899999336 2798.890265000012 52236.71428571428 0.96498773692784 0.0 770 1400 1800 2000 2500 2500 2800 2800 2800 2800 2800 f85ecea 2024-12-12 16:35:55.849929+00:00 0.16.0
POST /dynamique/status/bulk 30 0 580.0 870.5320791333293 25.30209100001457 2831.265218999988 411.0 1.0339154324226858 0.0 580 870 1400 1600 2100 2700 2800 2800 2800 2800 2800 f85ecea 2024-12-12 16:35:55.849929+00:00 0.16.0
POST /statique/ 19 7 920.0 1063.773145842107 47.17572399999881 2520.259102000012 59.36842105263158 0.6548131072010343 0.24124693423196 920 1400 1600 1700 2300 2500 2500 2500 2500 2500 2500 f85ecea 2024-12-12 16:35:55.849929+00:00 0.16.0
GET /statique/?limit=10 28 0 750.0 943.4246961785744 29.283504999995102 2552.1679849999828 13826.32142857143 0.96498773692784 0.0 760 1300 1600 2000 2300 2500 2600 2600 2600 2600 2600 f85ecea 2024-12-12 16:35:55.849929+00:00 0.16.0
GET /statique/?limit=100 30 0 740.0 1028.414871333327 149.8223019999898 2690.0347929999957 129006.36666666668 1.0339154324226858 0.0 860 1100 1500 2000 2300 2600 2700 2700 2700 2700 2700 f85ecea 2024-12-12 16:35:55.849929+00:00 0.16.0
PUT /statique/FR073EP7QU5233 30 0 570.0 867.1347487000056 85.14588400001344 2523.4783590000234 1371.0 1.0339154324226858 0.0 570 850 1400 1600 2200 2300 2500 2500 2500 2500 2500 f85ecea 2024-12-12 16:35:55.849929+00:00 0.16.0
GET /statique/FRALLEGO002006P3 31 0 510.0 889.2607643870931 26.457209000000148 2570.6753289999824 1202.0 1.0683792801701086 0.0 510 1200 1500 1500 2100 2600 2600 2600 2600 2600 2600 f85ecea 2024-12-12 16:35:55.849929+00:00 0.16.0
POST /statique/bulk 28 0 1200.0 1333.774997785717 395.3573950000191 2533.7914240000146 1129.178571428571 0.96498773692784 0.0 1400 1400 1900 2000 2300 2300 2500 2500 2500 2500 2500 f85ecea 2024-12-12 16:35:55.849929+00:00 0.16.0
Aggregated 402 7 750.0 910.5782092487556 12.520840999997064 2831.265218999988 14629.830845771145 13.854466794463988 0.24124693423196 760 1100 1400 1600 2100 2300 2600 2700 2800 2800 2800 f85ecea 2024-12-12 16:35:55.849929+00:00 0.16.0

Comparison with the latest previous benchmark

A lower (negative) value means the current version performs better than the previous one.

Type Name Request Count Failure Count Median Response Time Average Response Time Min Response Time Max Response Time Average Content Size Requests/s
GET /auth/whoami
POST /dynamique/session/
POST /dynamique/session/bulk
GET /dynamique/status/
POST /dynamique/status/
GET /dynamique/status/FRALLEGO002006P3
GET /dynamique/status/FRALLEGO002006P3/history
POST /dynamique/status/bulk
POST /statique/
GET /statique/?limit=10
GET /statique/?limit=100
PUT /statique/FR073EP7QU5233
GET /statique/FRALLEGO002006P3
POST /statique/bulk
Aggregated

@jmaupetit jmaupetit merged commit 1322d06 into main Dec 12, 2024
15 checks passed
@jmaupetit jmaupetit deleted the improve-db-queries branch December 12, 2024 19:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API Item related to the API service improvement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants