Adding docs for datapoints, cleaning up experiment client and routes

docs/source/pages/_autosummary/wei.core.notifications.rst

@@ -0,0 +1,30 @@
+wei.core.notifications
+======================
+
+.. automodule:: wei.core.notifications
+
+
+
+
+
+
+
+   .. rubric:: Functions
+
+   .. autosummary::
+
+      send_email
+      send_failed_step_notification
+
+
+
+
+
+
+
+
+
+
+
+
+

docs/source/pages/_autosummary/wei.core.rst

@@ -1,4 +1,4 @@
-wei.core
+wei.core
 ========
 
 .. automodule:: wei.core
@@ -34,6 +34,7 @@ wei.core
    wei.core.location
    wei.core.loggers
    wei.core.module
+   wei.core.notifications
    wei.core.scheduler
    wei.core.state_manager
    wei.core.step

docs/source/pages/_autosummary/wei.routers.data_routes.rst

@@ -0,0 +1,30 @@
+wei.routers.data\_routes
+========================
+
+.. automodule:: wei.routers.data_routes
+
+
+
+
+
+
+
+   .. rubric:: Functions
+
+   .. autosummary::
+
+      get_datapoint
+      get_datapoint_info
+
+
+
+
+
+
+
+
+
+
+
+
+

docs/source/pages/_autosummary/wei.routers.experiment_routes.rst

@@ -13,6 +13,7 @@ wei.routers.experiment\_routes
 
    .. autosummary::
 
+      event_return
       get_all_experiments
       get_campaign
       get_experiment_endpoint

docs/source/pages/_autosummary/wei.routers.rst

@@ -1,4 +1,4 @@
-wei.routers
+wei.routers
 ===========
 
 .. automodule:: wei.routers
@@ -28,6 +28,7 @@ wei.routers
    :recursive:
 
    wei.routers.admin_routes
+   wei.routers.data_routes
    wei.routers.event_routes
    wei.routers.experiment_routes
    wei.routers.location_routes

docs/source/pages/_autosummary/wei.types.datapoint_types.rst

@@ -0,0 +1,23 @@
+wei.types.datapoint\_types
+==========================
+
+.. automodule:: wei.types.datapoint_types
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

docs/source/pages/_autosummary/wei.types.exceptions.rst

@@ -0,0 +1,30 @@
+wei.types.exceptions
+====================
+
+.. automodule:: wei.types.exceptions
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+   .. rubric:: Exceptions
+
+   .. autosummary::
+
+      WorkflowCanceledException
+      WorkflowFailedException
+
+
+
+
+

docs/source/pages/_autosummary/wei.types.rst

@@ -1,4 +1,4 @@
-wei.types
+wei.types
 =========
 
 .. automodule:: wei.types
@@ -28,7 +28,9 @@ wei.types
    :recursive:
 
    wei.types.base_types
+   wei.types.datapoint_types
    wei.types.event_types
+   wei.types.exceptions
    wei.types.experiment_types
    wei.types.interface_types
    wei.types.module_types

docs/source/pages/_autosummary/wei.utils.rst

@@ -1,4 +1,4 @@
-wei.utils
+wei.utils
 =========
 
 .. automodule:: wei.utils
@@ -18,7 +18,10 @@ wei.utils
       initialize_state
       json_to_csv
       parse_args
+      pretty_type_repr
       string_to_bool
+      threaded_daemon
+      threaded_task
 
 
 

docs/source/pages/concepts/datapoints.rst

@@ -0,0 +1,56 @@
+
+===========
+Data Points
+===========
+
+A data point is a single measurement or observation taken during the course of an experiment. Data points are the fundamental building blocks of experimental data, and are used to generate the results and conclusions of an experiment.
+
+WEI currently supports two broad categories of data points:
+
+- ``data_value``: A single JSON serializable data point, such as a number, string, or dictionary. This is useful for scalar data points, such as single sensor readings (temperature, pressure, etc.) or for data that lends itself well to a dictionary format (e.g. a set of key-value pairs).
+- ``local_file``: A file that is stored locally on the WEI server. This is useful for larger data points, such as images, videos, or other large files that are generated as part of an experiment. The file is stored on the server and can be accessed later by the user. Any file type can be stored as a local file, but the user should be aware of the storage limitations of the WEI server.
+
+Data points are typically generated by a Workflow Run, and are associated with a specific Experiment. They can be logged to the Experiment by the WEI server (such as when returned by a module as part of a Workflow Run) or by the user, and are stored in WEI's database and/or local storage for later retrieval and analysis.
+
+Data Labels
+===========
+
+In addition to a unique ID, all data points have a ``label`` field, which is a human-readable string that describes the data point. This label is used to identify the data point in the WEI database and in the results of an experiment. The label should be descriptive and informative, and should help the user understand the context and meaning of the data point.
+
+There are two ways a data point can come to be labeled:
+
+- By the module, as part of the action that generates the data point. In this case, the module should provide a label as part of the data point, which will be used by WEI to identify the data point.
+- By the experimenter, as part of defining a data point in a workflow. In this case, the experimenter should provide a label for the data point in the workflow file, which will be used by WEI to identify the data point.
+
+For instance, in the following example Workflow definition, the data point generated by the ``take_picture`` action is labeled "experiment_result":
+
+.. code-block:: yaml
+
+    - name: Take Picture
+        module: camera
+        action: take_picture
+        args:
+            file_name: "experiment_result.jpg"
+        data_labels:
+          image: "experiment_result"
+
+If this step were part of a Workflow Run, the data point generated by the ``take_picture`` action would be labeled "experiment_result". This label would be used to identify the data point in the WEI database and in the results of the experiment. If the user does not provide a label, the default will be used for each data point returned by a module (in this case, "image")
+
+Retrieving Data Points
+======================
+
+Data points can always be retrieved by their ID using the WEI REST API (`/data/{datapoint_id}`) or using the :func:`wei.experiment_client.ExperimentClient.get_datapoint_value` and :func:`wei.experiment_client.ExperimentClient.save_datapoint_value` python methods. You can find the datapoint_id using the information returned in the Workflow Run object; the ids for each datapoint are included in the `result.data` field for the step.
+
+For convenience, you can also use the :func:`wei.types.workflow_types.WorkflowRun.get_datapoint_id_by_label` and :func:`wei.types.workflow_types.WorkflowRun.get_all_datapoint_ids_by_label` methods to retrieve data points by their label.
+
+For instance, to retrieve the data point labeled "experiment_result" from the example above, you could use the following code:
+
+.. code-block:: python
+
+
+    datapoint_id = wf_run.get_datapoint_id_by_label("experiment_result")
+
+    # Retrieve the data point using the ExperimentClient
+    value = experiment_client.get_datapoint_value(datapoint_id)
+
+    print(value)

docs/source/pages/concepts/index.rst

@@ -9,3 +9,4 @@ Core Concepts
     workcell
     workflow
     experiment
+    datapoints

docs/source/pages/concepts/module.rst

@@ -6,10 +6,10 @@ A WEI Module is a combination of software and a (typically) physical device (e.g
 
 Modules are designed to be:
 
-- **Independent**: each module can act without relying on any other module.
-- **Self-contained**: everything needed to integrate a single device is contained within that module
-- **Composable**: multiple modules can be easily combined to create Workcells.
-- **Portable and Reusable**: so they can be used in different workcells, or repeatedly in the same workcell to control multiple instances of a device.
+- *Independent*: each module can act without relying on any other module.
+- *Self-contained*: everything needed to integrate a single device is contained within that module
+- *Composable*: multiple modules can be easily combined to create Workcells.
+- *Portable and Reusable*: so they can be used in different workcells, or repeatedly in the same workcell to control multiple instances of a device.
 
 .. figure:: /assets/module_logic.png
 
@@ -29,15 +29,18 @@ The Action method is called by WEI to trigger a specified Action on the device o
 
 The Action method should accept the following parameters:
 
-- `action_handle`: The name of the specific action to be performed.
-- `action_vars`: A json-serializable dictionary of variables specific to the action in question. The keys and values of the dictionary correspond to the names and values of the variables, respectively.
+- ``name``: The name of the specific action to be performed.
+- ``args``: A json-serializable dictionary of variables specific to the action in question. The keys and values of the dictionary correspond to the names and values of the variables, respectively.
 
 State
 -----
 
-The State method is called by WEI to query the current state of the device or instrument controlled by the Module. It should return a dictionary of the current state of the device or instrument.
+The State method is called by WEI to query the current state of the device or instrument controlled by the Module. It should return a dictionary of the current state of the module that conforms to :class:`wei.types.module_types.ModuleState`.
 
-The status returned by this method should conform to the :class:`wei.types.ModuleStatus` enum.
+It should, at the very least, return the following keys:
+
+- ``status``: the status of the module, must be included in the :class:`wei.types.ModuleStatus` enum.
+- ``error``: a string describing any error that has occurred, or an empty string if no error has occurred.
 
 About
 -----

docs/source/pages/concepts/workcell.rst

@@ -11,7 +11,7 @@ Modules
 
 Modules are the individual components that make up a Workcell. These can be anything from a robot arm to a camera to a PCR machine. Each Module has a number of properties, including a name, a model, an interface, and a configuration.
 
-The name of a module in the Workcell is also used in the Workflow file to specify which module should take an action for a given step.
+The name of a module in the Workcell is also used in Workflow definitions to specify which module should take an action for a given step.
 
 Locations
 ==========

docs/source/pages/concepts/workflow.rst

@@ -7,12 +7,12 @@ WEI Workflows define a sequence of steps that can be executed in a :doc:`./workc
 The Workflow File
 ==================
 
-Workflows can be defined declaratively using a YAML file, which should conform with the :class:`wei.types.Workflow`.
+Workflows can be defined declaratively using a YAML file, which should conform with the :class:`wei.types.workflow_types.Workflow`.
 
 In general, a workflow file consists of 3 parts:
 
 - Metadata: defines information about the workflow as a whole
-- Modules: lists the modules used in the workflow
+- Modules (Optional): lists the modules used in the workflow
 - Flowdef: a sequence of steps to execute
 
 As an example, consider the following Workflow file:

docs/source/pages/deployment_guide/bare_metal.rst

@@ -7,7 +7,7 @@ This guide provides instructions for deploying a bare-metal WEI workcell. It ass
 Installing the WEI Python Package
 =================================
 
-Note on platforms: WEI is primarily developed for and tested on Linux. While we have tried to write it to be platform-agnostic, we do not target or test natively on Windows or Mac. If you are using Windows, we recommend using the `Windows Subsystem for Linux (WSL) <https://learn.microsoft.com/en-us/windows/wsl/install>`_.
+Note on platforms: WEI is primarily developed for and tested on Linux. While we have tried to write it to be platform-agnostic, we do not target or test natively on Windows or Mac. If you are using Windows, one options is using the `Windows Subsystem for Linux (WSL) <https://learn.microsoft.com/en-us/windows/wsl/install>`_.
 
 Installing from PyPi
 ---------------------
@@ -41,6 +41,12 @@ The WEI engine is responsible for the scheduling and execution of workflows and
 
 This will start the engine and load the workcell configuration from the specified YAML file. The engine will begin scheduling and executing workflows and other operations as they are queued via the WEI Server.
 
+Alternatively, you can start the engine automatically as a spin-off thread from the server, via the ``autostart_engine`` configuration option in the workcell YAML file or command line arguments, e.g.:
+
+``python -m wei.server --workcell <path/to/workcell.yaml> --autostart_engine True``
+
+This will start the engine automatically when the server starts, and will become the default behavior in a future release.
+
 Running Redis
 =============
 
@@ -53,4 +59,4 @@ There are two methods for configuring the WEI Engine and Server that work in tan
 
 The only required argument for the WEI Engine and Server is the path to the workcell YAML file.
 
-For a complete breakdown of configuration options, see :class:`wei.types.WorkcellConfig`.
+For a complete breakdown of configuration options, see :class:`wei.types.workcell_types.WorkcellConfig`.

docs/source/pages/deployment_guide/docker.rst

@@ -74,7 +74,6 @@ Starting a workcell that has already been dockerized is as simple as the followi
 
     - ``USER_ID``: The user ID of the user running in the container. This is used to ensure that files created by the container are owned by the correct user on the host system. You can find your user ID by running ``id -u`` in a terminal on Linux/WSL/Mac.
     - ``GROUP_ID``: The group ID of the user running in the container. This is used to ensure that files created by the container are owned by the correct group on the host system. You can find your group ID by running ``id -g`` in a terminal on Linux/WSL/Mac.
-    - ``PROJECT_DIR``: The absolute path to the directory containing the workcell, usually the top level directory of the repository.
 
 #. Run ``docker compose up`` in the directory containing the ``compose.yaml`` file (usually the top of the repository, or in ``/docker``). In some cases, there are multiple compose files for a single workcell, such as a compose file for the WEI core services and one or more compose files for modules. In these cases, the individual compose files might have names like ``workcell_name.compose.yml`` or ``hostname.compose.yml``, and you will need to run ``docker compose -f /path/to/file.yaml up`` to specify the compose file to use.
 #. Upon running this command, you should see a log stream from the containers that are starting up. If you see any errors/container exits, you may need to troubleshoot them.