[RFE] [PERF] Revamp quads-web methods to use async views or SqlAlchemy directly.

[sadsfae]

Problem

quads-web does a lot of heavy lifting to render the dynamic /assignments page. This calls a few methods to render parts of the page every time it's loaded. Lazy loading helps a lot to reduce the load time but it can still be faster.

Additionally, navigating from the main QUADS wiki to /available can also be intermittently slow as it may need to wait until all page elements are rendered by Flask. In general /available rendering can also just be slower than we want.

This is mostly due to having a large amount of hosts, +800 or so, and isn't noticeable in our STAGE environment of just a dozen or so systems.

Potential Solutions

1) Using async views with Flask

Since Flask 2.0+ async views are supported, this let's us create independent event loops inside our render methods using async views. Let's experiment with and implement this into quads-web so that all API endpoints contacted via quads-server are rendered much faster.

(Great blog post on implementing async views with Flask)
https://testdriven.io/blog/flask-async/

(Good corollary blog post about async design pitfalls with event loops)
https://dev.to/sethmlarson/the-problem-with-flask-async-views-and-async-globals-pl

Design Considerations

quads-web renders are already broken up until multiple methods e.g. summary, totals, and per-cloud rendering so we may not need to worry too much about anything here.

We may however want to use gunicorn with Uvicorn workers so we are using a proper ASGI instead of a WSGI as denoted in the last blog post.

gunicorn example:app -w 4 -k uvicorn.workers.UvicornWorker

This would require changing some package dependencies as well to one of these if not both:

python3-uvicorn+standard.noarch : Metapackage for python3-uvicorn: standard extras
python3-uvicorn.noarch : The lightning-fast ASGI server

2) Using Native SQLAlchemy calls to generate content

Another option here which may be better is simply changing the content generation methods to use native SQLAlchemy calls directly against the database.

This is done a few places already with great success like

quads/src/quads/server/dao/cloud.py

Line 74 in 31cbab4

def get_free_clouds() -> List[Cloud]:

free_clouds = (
            db.session.query(Cloud)
            .outerjoin(Assignment)
            .outerjoin(Schedule)
            .filter(~Cloud.id.in_(active_or_future_schedules))
            .filter(Cloud.name != Config["spare_pool_name"])
            .order_by(Cloud.name)
            .all()
        )
        return free_clouds

More detail on a follow-up comment.

[sadsfae]

After some design discussion with @kambiz-aghaiepour another approach here is to take all the more expensive and locking-prone methods that generate the web content and just rewrite them to use the SQLAlchemy ORM.

This would have a significant speedup and perhaps less re-architecture of quads-web so far as changing the Gunicorn listener type to uvicorn etc.

We wouldn't forsee ever running quads-web on a host that also isn't running the QUADS database, though the same API calls used now would still be available, calling native SQL to generate the same content into jinja should provide much faster return and rendering under load and with a large fleet of systems to query on each page reload.