feat: instrument served apps with Prometheus metrics

[squat]

This commit adds basic Prometheus instrumentation of the HTTP server of
all applications served with serve=True. This lets us conveniently
monitor how models are performing.

Signed-off-by: Lucas Servén Marín [email protected]

[isidentical]

@squat i'd recommend manually testing it out with a basic python server and trying fal fn run t.py test_server to see the exact error, integration tests just show the server couldn't start and i assume its because of the loopy stuff but there might be others as well

[efiop]

LGTM, but at least 1 test is failing with 503 (could be unrelated, but restart didn't help)

[kakkoyun]

/subscribe

[squat]

re-posting a conversation I had with @efiop on diagnosing the mysterious test failures we would get when we tried to import the PrometheusMiddleware in the _build_app func:

the problem was that re generate the metadata during app registration (not in the isolate agent). this means that the func to build the open api schema is run locally, which means that the middleware packages have to be installed locally as well as in the agent. So when we imported from starlette_exporter import PrometheusMiddleware this would raise an exception. HOWEVER we catch and ignore all exceptions raised during metadata generation and leave an empty metadata

fal/projects/fal/src/fal/api.py

Lines 397 to 409 in 47be563

	if metadata is None:
	metadata = {}
	# TODO: let the user send more metadata than just openapi
	if isinstance(func, ServeWrapper):
	# Assigning in a separate property leaving a place for the user
	# to add more metadata in the future
	try:
	metadata["openapi"] = func.openapi()
	except Exception as e:
	print(
	f"[warning] Failed to generate OpenAPI metadata for function: {e}"
	)

, which is why we would get a 200 but with an empty payload and no error

thanks @chamini2 for the pointers that lead to this fix!

[efiop]

Have you considered running them in separate threads? Also interesting if there are implications to running two servers in the same event loop (basically same thread?).

[squat]

we run our two servers in the same event loop for all of our isolate cloud instances. We could run them in different threads, but that doesn't really fix this issue commented out here at all. We have to fix signal handling in the isolate agent to signal the thread in this fal app using a threading event, then this fal app can decide what to do with that event, e.g. set an asyncio event for async servers somehow call stop on the uvicorn workers. In short, threading vs async doesn't fix this graceful shutdown at all. I prefer to use async for our multiple servers. Some of our internal fal isolate servers have >3 servers running in the same event loop: grpc, http, metrics, etc. Eventually I'd like to define our async-friendly uvicorn server class here in fal-ai/fal and import it in our internal tooling.

[efiop]

Oh, didn't know that we have an additional issue with the agent. Not a blocker for me, we can fix this when we need to.