Benchmark Ideas

[ankush]

"nanobenchmarks"

• frappe._dict
• flt, cint, round
• Document methods
• other similar things

Unit Microbenchmarks

Read only benchmarks

Note: all of these are supposed to run on "fresh" install.

• ORM
  • get_doc - single document, multiple
  • get_meta - without frappe.local cache. i.e. first call-in request.
• Redis - {single, multiple, with and without local cache}
  • get_value, set_value, exists
  • hget, hset
  • make_key
• Database {single, multiple, mixed, all without local cache}
  • get_value
    • different filter compositions - dict, list, various operators, fields
  • get_single_value
  • commit / rollback (empty)
  • set_value - simple k:v
  • delete - simple by name
  • table metadata caching
  • sql queries (use as_dict as default)
    • as_dict, as_list
    • Many rows retrieval >1000 documents
• QB / DBQuery
  • Top kinds of arguments - just test query generation, not execution.
• Web pages - PathResolver
• frappe.publish_realtime
• frappe.enqueue
• scheduler - empty scheduler that just makes scheduling decisions.
• permission checks:
  • has_perm
  • has_only_select
• Web requests overhead - Guest and authed
• background jobs overhead

Read/Write benchmarks

• db.set_value - different compositions
• document.save - single, multiple
• document.insert - single, multiple
• document.submit
• document.db_set

E2E minibenchmarks for top requests

• /api/method/ping - guest and auth
• /api/method/frappe.desk.form.save.savedocs - insert, update, submit
• /api/method/frappe.desk.reportview.get - multiple filter combinations
• /api/method/frappe.desk.query_report.run - just overheads
• /api/method/frappe.desk.notifications.get_open_count
• /api/method/frappe.desk.reportview.get_count
• /api/method/frappe.desk.form.load.getdoc
• /api/method/run_doc_method - some read only method to do "server side reactivity"
• /app - desk render
• /api/method/frappe.desk.form.load.getdoctype - metadata bundle (should actually be cached.
• /api/method/frappe.realtime.can_subscribe_doc
• /api/method/frappe.utils.print_format.download_pdf
• /api/method/frappe.www.printview.get_html_and_style
• /login - login page render
• /printview
• /api/method/frappe.client.validate_link
• /api/method/frappe.client.get_value
• /api/method/frappe.realtime.get_user_info
• /api/method/frappe.desk.reportview.get_list
• /api/method/frappe.desk.doctype.notification_log.notification_log.get_notification_logs
• /api/resource/{DT}

Trace replays (Open loop)

• "Markov mix" of top requests ^
• Synthetic trace that represents "typical" cloud user.
• Production trace of Frappe's internal sites

[ankush]

I'll start with microbenchmarks (easier to write) using pyperf. I considered https://codspeed.io/ but I don't like their measurement methodology and don't trust that they'll stay "free" for long.

So roughly:

• all kinds of benchmarks can live in this repository.
• They'll be executed manually on local setup + remote setup to start with. Eventually, we can have a self-hosted GHA runner on the bare metal server to do "continuous benchmarks".
• Data can be sent to a site running insights for dashboards.

[kunhimohamed]

Is there any plan to replace Python?
heard that other languages have shorter execution times than Python.

[ankush]

Nope. There is no plan to change architecture or runtime at all.

I might write some rust extensions to speed up frequent parts if it makes sense though. (e.g. MySQL connector's parser is written in python and is quite slow)