Blog post: Overview of core plugins

docs/news/posts/2024-12-13.md

@@ -0,0 +1,107 @@
+---
+blogpost: true
+category: Blog
+tags: plugins
+author: Julian Geiger
+date: 2024-12-13
+---
+
+# Non research-domain specific plugins to extend core functionality
+
+Dear users, we're happy to announce that we have recently crossed the 100-plugin mark of registered plugin packages on the [AiiDA plugin registry](https://aiidateam.github.io/aiida-registry/).
+Among those are classics like [`aiida-quantumespresso`](https://github.com/aiidateam/aiida-quantumespresso/) and [`aiida-vasp`](https://github.com/aiidateam/aiida-vasp/), to drive the electronic structure codes we all know and love.
+However, did you know that there are various non research-domain specific plugins that AiiDA developers have created over the years, which can extend AiiDA's core functionality?
+
+In today's post, we would therefore like to give you a short introduction to each of these plugins, and show you when you can incorporate them into your work, to make sure you'll get the most out of your AiiDA journey![^1]
+
+## Simplifying code execution and workflow creation
+
+### [`aiida-workgraph`](https://github.com/aiidateam/aiida-workgraph/)
+
+We've been making quite a bit of fuzz about this one in recent weeks already, as we're targeting a first, stable release for early next year (fingers crossed :crossed_fingers:).
+The `aiida-workgraph` provides a new, alternative way of construction workflows, in addition to the classical `WorkChain`.
+It comes with all the typical AiiDA features, like provenance tracking, checkpointing, caching, and remote execution, but provides a simpler syntax to construct workflows.
+This can be achieved by adding nodes (or `Tasks`, as we call them) and links, in equivalence to how one would typically build-up a graph.
+In this way, `aiida-workgraph` makes it simpler to quickly chain existing AiiDA components (`CalcJobs`, `WorkChains`, or other `WorkGraphs`) together, and even allows visualizing the flow of data via the different processes before the workflow is executed (through an interactive web GUI).
+As we're currently stabilizing the API, any feedback is valuable and helps us to polish the tool as best as we can until the release!
+You can read more about it on its [RTD page](https://aiida-workgraph.readthedocs.io/en/latest/), and find example `WorkGraphs` in [this repository](https://github.com/superstar54/workgraph-collections).
+
+### [`aiida-shell`](https://github.com/sphuber/aiida-shell/)
+
+Probably we don't have to talk too much about this one, as we covered it in depth in one of our recent [blog posts](https://aiida.net/news/posts/2024-11-01-aiida-shell.html).
+In a nutshell, `aiida-shell` allows you to run any executable (e.g., a shell script, python script, compiled binary, etc.) without the need for writing an entire plugin interface for it.
+Just plug it into the `launch_shell_job` function, and you're ready to rumble! If you eventually _do_ want to have a more sophisticated interface for your code or script, and you can't find one on the [AiiDA plugin registry](https://aiidateam.github.io/aiida-registry/), [this later section](#aiida-plugin-cutter) of the post might be interesting for you.
+
+## Enhancing high-throughput
+
+### [`aiida-submission-controller`](https://github.com/aiidateam/aiida-submission-controller/)
+
+Have you ever wondered how a "hero run" like the one recently conducted on the new CSCS Alps infrastructure (read more about it [here](https://nccr-marvel.ch/highlights/AiiDA-hero-run-Alps)) can be orchestrated?
+
+### [`aiida-hyperqueue`](https://github.com/aiidateam/aiida-hyperqueue/)
+
+If you've ever run simulations on a high-performance computing (HPC) infrastructure, you're likely familiar with scheduling systems such as SLURM, SGE and PBSPro (for all of which AiiDA provides an interface, btw :wink:).
+These are all mature projects that work well, but now imagine you find yourself in the situation (maybe you've already found yourself in it) that you have to run many, say, a couple thousand, small jobs.
+Utilizing one node for each job, these might finish quickly, but you have to queue for every single one of them.
+What the [HyperQueue scheduler](https://github.com/It4innovations/hyperqueue) allows you to do instead, is to submit a whole array of such jobs (say, 128 if you're working with a 128-core node), where each job runs on sub-node resources (1 CPU core per job in the given example).
+While, the jobs then take a bit longer (maybe a couple hours rather than a couple minutes), if they are small enough, that is probably still acceptable.
+However, as the whole array is submitted as one job through the scheduling system provided by the HPC, you only have to queue once, rather than 128 times!
+So if you ever need to run a lot of shorter simulations, make sure to give it a shot![^2]
+
+## AiiDA setup
+
+### [`aiida-project`](https://github.com/aiidateam/aiida-project/)
+
+As we outlined in [another one of our recent blog posts](https://aiida.net/news/posts/2024-09-20-simpler-installation.html), since v2.0, we have significantly simplified the creation of an AiiDA profile.
+Still, if you find yourself juggling many different profiles on your machine, each with its own configuration, Python environment, and directory of setup files (e.g., for `Computer` and `Code` instances), it can become easy to mess things up.[^3]
+To help out with this, `aiida-project` encapsulates and automates the steps to create and switch between different projects using AiiDA, basically functioning as an _AiiDA project manager_.
+It creates the necessary directories and Python environments for you (via `pip` or `conda`) in standardized locations, and enables easy switching between projects.
+Installation is simple via `pipx`, and `uv` support is [coming soon](https://github.com/aiidateam/aiida-project/pull/28)!
+
+### [`aiida-code-registry`](https://github.com/aiidateam/aiida-code-registry/)
+
+In the previous paragraph, we mentioned storing setup files for `Computer` and `Code` instances in specific directories.
+Let's elaborate on this further:
+If you have interactively created such entities via `verdi computer setup` and `verdi code create` by filling in the prompts you are being asked on the command line (which is the route we usually mention in the tutorials), it might be worthwhile to export their configurations from the AiiDA database into YAML files.[^4]
+From these files, new instances can then be created by passing them to the above two commands via the `--config` flag.
+This allows you to re-use the configurations, make and track changes, and follow a Configuration-as-Code (CaC) approach.
+To facilitate this process, we maintain the `aiida-code-registry`[^5] as a collection of such configuration files, which can thus serve as a great way to share existing and start setting up new resources! If you don't find the national HPC your are using in the repository, and your configuration is somewhat transferable, feel free to make a PR — you might help out others!
+
+## Miscellaneous
+
+
+### [`aiida-plugin-cutter`](https://github.com/aiidateam/aiida-plugin-cutter/)
+abc
+
+### [`aiida-pythonjob`](https://github.com/aiidateam/aiida-pythonjob/)
+
+### [`aiida-firecrest`](https://github.com/aiidateam/aiida-firecrest/)
+
+Initially part of th
+
+***
+
+As most of the plugins discussed in this post have been in use by many developers over the years, we now feel comfortable to promote them to a larger audience.
+Therefore, we added an [overview of the plugins](https://aiida.readthedocs.io/projects/aiida-core/en/latest/reference/core_plugins.html) to the RTD page of AiiDA, which is also linked to from a new panel of the [landing page](https://aiida.readthedocs.io/projects/aiida-core/en/latest/).
+
+***
+
+
+[^1]:
+    Please note that, while core developers have been using these plugins for a while to enhance their own work with AiiDA, many of them are still in a development, pre-stable release phase.
+    Therefore, use them at your own risk, and please do provide us with your feedback so we can stabilize and improve them.
+    As always, issues and PRs more than welcome!
+
+[^2]:
+    Bonus points because the tool is written in Rust! :crab:
+
+[^3]:
+    Forgetting to set the `$AIIDA_PATH` environment variable when switching profiles — we've all been there.
+
+[^4]:
+    Which can be easily achieved via `verdi computer export <computer pk|label|uuid>` and `verdi code export <code pk|label|uuid>`.
+
+[^5]:
+    As written in the `README` of the [`aiida-code-registry`](https://github.com/aiidateam/aiida-code-registry),migration is eventually planned to the [`aiida-resource-registry`](https://github.com aiidateam/aiida-resource-registry), which contains configuration files for [AiiDAlab](https://github.com/aiidalab/).
+    Currently, the migration is still hindered, as we use `jinja2` fields for AiiDAlab, which are not yet supported for the creation of `Computer` and `Code` instances in `aiida-core` (see open issue [here](https://github.com/aiidateam/aiida-core/issues/4680)).
+    For now, it is good to keep an eyeopen on both repositories, but we are planning to fix this soon!