Documentation rewrite - phase one

README.md

@@ -0,0 +1,85 @@
+## Table of Contents
+- [About](#about)
+- [How It Works](#how-it-works)
+- [Getting started](#getting-started)
+    - [Requirements](#requirements)
+    - [Installation](#installation)
+    - [Test config files](#test-config-files)
+    - [Testing Quick Starts](#installation-and-quick-start)
+
+
+## About
+Zathras is a test automation framework for use with bare metal, cloud, and virtualized systems. Tests are defined by wrappers - external, self-contained scripts that enable easy extension of Zathras's functionality. 
+
+Currently, Zathras offers the following test wrappers:
+
+| Test Name                                          | Description                                                                         | Wrapper URL                                                |
+|----------------------------------------------------|-------------------------------------------------------------------------------------|------------------------------------------------------------|
+| CoreMark                                           | Test processor core functionality                                                   | https://github.com/redhat-performance/coremark-wrapper     |
+| CoreMark Pro                                       | EEMBC's advanced processor benchmark.                                               | https://github.com/redhat-performance/coremark_pro-wrapper |
+| Flexible I/O tester                                | Test the Linux I/O subsystem and schedulers                                         | https://github.com/redhat-performance/fio-wrapper          |
+| HPL                                                | Freely available implementation of the High Performance Computing Linpack Benchmark | https://github.com/redhat-performance/autohpl-wrapper      |
+| HammerDB                                           | Database benchmarking                                                               | https://github.com/redhat-performance/hammerdb-wrapper     |
+| IOzone                                             | File system benchmarking                                                            | https://github.com/redhat-performance/iozone-wrapper       |
+| Linpack                                            | Licensed version of Linpack benchmark (see also: HPL)                               | https://github.com/redhat-performance/linpack-wrapper      |
+| NUMA STREAM                                        | Extension of STREAM benchmark to measure NUMA effects                               | https://github.com/redhat-performance/numa_streams-wrapper |
+| Passmark                                           | CPU benchmark                                                                       | https://github.com/redhat-performance/passmark-wrapper     |
+| pig                                                | Processor scheduler efficiency test                                                 | https://github.com/redhat-performance/pig_wrapper          |
+| Python Performance Benchmark Suite (pyperformance) | Performance testing for Python using real-world benchmarks                          | https://github.com/redhat-performance/pyperf-wrapper       |
+| SPEC CPU 2017                                      | Suite of benchmarks for compute-intensive performance                               | https://github.com/redhat-performance/speccpu2017-wrapper  |
+| SPEC JBB2005                                       | Evaluate the performance of server side Java                                        | https://github.com/redhat-performance/specjbb-wrapper      |
+| STREAM                                             | Benchmark for sustained memory bandwidth                                            | https://github.com/redhat-performance/streams-wrapper      |
+| Uperf                                              | Network performance testing                                                         | https://github.com/redhat-performance/uperf-wrapper        |
+
+## How It Works
+1. Zathras is installed to a controller system.
+2. Zathras runs using either a configuration file or by supplying parameters in your command-line interface (CLI) (see the [CLI reference](docs/command_line_reference.md) for more info).  
+3. Test(s) run on the system under test (SUT).
+4. When tests finish, Zathras copies the results back to the controller system.
+5. Once results have been retrieved, Zathras will attempt to cleanup the testing environment (such as by tearing down instances in the cloud or by removing directories and files on bare metal). 
+
+The user configures the Zathras installation with information about the system under test (SUT) and what test(s) to run.
+
+## Getting Started
+
+### Requirements
+To use Zathras, you need: 
+1. A system that will coordinate the execution of tests, which we will call the *control node* or simply the *controller*.
+2. One or more systems where tests will be run. This can be another physical system (aka bare metal), an instance on a public cloud like Amazon Web Services (AWS), or even a virtual machine (VM) either on the controller or another host system.
+
+The following packages will need to be installed on the controller to run Zathras. If you use the included installer script (next section), they will be installed for you.
+- ansible-core
+- git
+- jq
+- python
+- python3-pip
+- terraform
+- yq, version 2.10.0 (newer versions currently break Zathras)
+- wget
+
+### Installation
+Installing Zathras is easy! First, clone this repository to your controller system.
+
+    git clone https://github.com/redhat-performance/zathras.git
+
+Installation can then be completed by running the install script located within the repo's bin/ directory:
+
+    ./install.sh
+
+
+Note: for most installations the install script should be run as a user, not root.
+
+### Test config files
+Test config files, stored in the config/ directory of the Zathras installation, are the connective tissue between Zathras and the test wrappers. Currently this Zathras repo does not contain any defined test config files. Before you can use this tool, you will need to write your own test configs using the included example.yml, or add the configs from an external source if one is available to you.
+
+The process of creating test configuration files is further explained [here](https://github.com/redhat-performance/zathras/blob/documentation-rewrite/docs/test_config_files.md).
+
+### Testing Quickstarts
+In general, testing with Zathras follows a similar set of steps across modalities: configure the tests to run, then run burden via the command line. 
+
+
+However, there are some aspects that are particular to the testing environment being used. Please refer to the following documentation that matches your use case:
+
+- [Testing on bare metal](https://github.com/redhat-performance/zathras/blob/documentation-rewrite/docs/test_config_files.md)
+- [Testing on cloud](https://github.com/redhat-performance/zathras/blob/documentation-rewrite/docs/test_config_files.md)
+- [Testing on virtualization](https://github.com/redhat-performance/zathras/blob/documentation-rewrite/docs/test_config_files.md)

docs/command_line_reference.md

@@ -0,0 +1,178 @@
+# Command line interface (CLI) reference
+
+## Usage
+
+    ./bin/burden <options> [<args>...]
+
+## Options
+
+### --archive \<dir>/\<results>
+Sets the location where the archive information will be saved.
+
+### --child
+Tells burden it is a child of another burden process and not to perform the initial setup work.
+
+### --create_attempts
+Number of times we attempt to create an instance to get the desginated cpu type.
+
+### --git_timeout
+Number of seconds to timeout on git requests. Default is 60.
+
+### --system_type \<vendor>
+aws, azure, gcp or local.
+
+### --host_config \<config options>
+Specification of the system and configuration. Possible values are "cloud" and "local".
+
+If the --system_type option is local, then this is simply the system name to run on, and it will pull the config value from the file <hostname>.conf in the local_configs directory. For more information on local_configs, please see (TODO: add link). 
+
+If the --system_type option is cloud, then the following fields may be specified in the config file:
+- instance type: The cloud instance (ie i3en.xlarge). Inclusion of \[region=\<value>&zone=\<value>] is optional.
+- region: The region where the cloud is created. The default option is the user's default region.
+- zone: The zone within the region the cloud will be created in. Randomly chosen by default.
+
+### --ignore_repo_errors
+If present we will ignore repo errors, default is to abort the run when a repo error occurs.
+
+### --individ_vars
+Contains various burden settings. Takes precedence over the scenario file, but is overridden by the command line. Default is config zathras_specific_vals_def.
+
+### --java_version
+Java version to install, java-8, java-11.
+
+### --kit_upload_directory
+Full path to directory uploading to. If not present, Zathras will locate the filesystem with the most space on it and use that location.
+
+### --max_systems \<n>
+Maximum number of burden subinstances that will be created from the parent. Each subinstance is a cloud or local system. 3 is the default.
+
+### --no_clean_up
+Do not cleanup at the end of the test.
+
+### --no_packages
+Do not install any packages, default is no.
+
+### --no_pbench_install
+Do not install pbench. The default is 0 (install pbench).
+
+### --no_spot_recover
+Do not recover from a spot system going away.
+
+### --package_name \<name>
+Use this set of packages to override the default in the test config file instead of the default. Default format package name \<os>_pkg, new name \<os>_pkg_\<ver>.
+
+### --persistent_log
+Enable persistent logging.
+
+### --preflight_check
+Performs various checks on the scenario file, and Zathras and then exits.
+
+### --results_prefix \<prefix>
+Run directory prefix
+
+### --retry_failed_tests \<0/1>
+Indicates to retry any detected failed tests if set to 1 (1 is the default).
+
+### --scenario \<scenario definition file>
+Reads in a scenario and then runs it (if used, host configs are designated in the file). 
+
+If the scenario name starts with https: or git:, then the scenario file is retrieved from a git repo. 
+
+If the line in the scenario file starts with #, then that line is a comment. 
+
+If the line starts with a %, it indicates to replace the string.
+
+Format to replace a string % \<current string>=\<new string>
+
+### --scenario_vars <file>
+File that contains the variables for the scenario file, default is config/zathras_scenario_vars_def.
+
+### --selinux_level
+enforcing/permissive/disabled. The setting to have in /etc/selinux/config file. Note: Zathras does not support enforcing in Ubuntu at this time.
+
+### --selinux_state 
+disabled/enabled. If disabled is selected, selinux will be disabled via grubby (Amazon and RHEL). For Ubuntu,
+enabled will install the require packages, update the config file and reboot.
+
+### --ssh_key_file
+Designates the ssh key file we are to use.
+
+### --show_os_versions
+Given the cloud type and OS vendor, show the available operating system versions.
+
+### --show_tests
+List the available test as defined in config/test_defs.yml
+
+### --test_def_file \<file>
+Test definition file to use.
+
+### --test_def_dir \<dir>
+Test definition directory. Default is \<execution dir>/config. If https: or git: is at the start of the location, then we will pull from a git repo.
+
+### --test_override \<options>
+Overrides the given options for a specific test in the scenario file.
+
+### --tests \<test>
+testname, you may use "test1,test2" to execute test1 and test2.
+
+### --test_iter \<iterations>
+How many iterations of the test to run (includes linpack).
+For cloud instances, this will terminate the cloud image and start a new image for each iteration.
+
+### --test_versions \<test>,\<test>....
+Shows the versions of the test the are available and brief description of the version. This only applies to git repos
+
+### --test_version_check
+Checks to see if we are running the latest versions of the tests and exits out when done. Default is no.
+
+### --tf_list
+List active systems created via tf (terraform?).
+
+### --tf_terminate_list
+Delete the designated terraform systems.
+
+### --tf_terminate_all
+Go into each terraform directory and attempt to remove the terraform instance.
+
+### --tuned_profiles \<comma separated list of tuned profiles>
+Note: only for RHEL. Designates the tuned profiles to use. if the system type is a cloud environment, then each tuned profile is a distinct cloud instance.
+
+### --tuned_reboot \<>
+Note: only for RHEL. If value is 1 we will reboot the system each time a new tuned profile is set.
+
+### --verbose
+Verbose usage message.
+
+### --upload_rpms ,....
+Comma separated list of rpms (full path) to upload and install.
+
+### --update_target
+Image to update.
+
+Note: only 1 update image can be used, makes no difference if designate a different one for each system in the scenario file, the first one will be used.
+
+### --use_latest_version
+Will update the templates so we are using the latest versions of the test (git repos only).
+
+### -h --usage
+Condensed usage information.
+
+## Cloud-only options
+
+## --cloud_os_id \<os id>
+Image of the OS to install (example aws aminumber)
+For multiple architectures, this is allowed"
+x86:ami-0fbec8a0a2beb6a71,arm64:ami-0cfa90ca3ebfc506e"
+Burden will select the right ami for the designated host."
+
+### --create_only
+Only do the VM creation and OS install action.
+
+### --os_vendor \<os vendor>
+Currently rhel, ubuntu, amazon.
+
+### --terminate_cloud
+If 1, terminate the cloud instance, if 0 leave the cloud image running. Default is to terminate.
+
+### --use_spot
+Uses spot pricing based on the contents of config/spot_price.cfg. Default is not to use spot_pricing

docs/scenario_files.md

@@ -0,0 +1,117 @@
+# Scenario files
+Using a scenario file to configure tests offers several advantages over setting the parameters via the command line. Scenario files provide more flexibility in defining tests, greater ease of modifying tests, improved test reproducibility, and enable the creation of more complex test runs. Once a scenario file is created, that scenario can also easily be exported for future use.
+
+## Scenario file format
+
+    global:
+        <global options>
+    systems:
+        system1:
+            <system1 options>
+
+The outline above can also easily be extended to incorporate multiple systems within a single scenario, like so:
+
+    global:
+        <global options>
+    systems:
+        system1:
+            <system1 options>
+        system2:
+            <system2 options>
+
+## Scenario file options
+The options in the parameters file maps directly back to the CLI options, minus the dash.
+
+
+## Using scenario files
+
+Scenario files are placed within your Zathras working directory, and can be given any name. For example, given a scenario file `my_scenario`, the following command would begin a Zathras run using that scenario file:
+
+    ./bin/burden --scenario my_scenario
+
+## Examples
+
+### Scenario file for bare metal
+#### Example 1 
+This scenario would run the STREAM benchmark on a local bare metal system named "test_sys".
+
+    global:
+        result_prefix: just_a_test
+        system_type: local
+    systems:
+        system1:
+            tests: streams
+            host_config: "test_sys"
+
+### Scenario files for cloud
+#### Example 1 
+Create an AWS system, list system information, then delete the instance.
+
+    global:
+        ssh_key_file: replace_your_ssh_key
+        terminate_cloud: 0
+        os_vendor: rhel
+        results_prefix: create_only
+        os_vendor: rhel
+        system_type: aws
+    systems:
+        system1:
+            cloud_os_id: ami-078cbc4c2d057c244
+            host_config: "m6a.xlarge"
+
+#### Example 2 
+This scenario would run a test on public cloud using AWS with two hosts, "m5.xlarge" and "m5.12xlarge".
+
+    global:
+        ssh_key_file: /home/test_user/permissions/aws_region_2_ssh_key
+        terminate_cloud: 1
+        cloud_os_id: aminumber
+        os_vendor: rhel
+        results_prefix: linpack
+        system_type: aws
+      systems:
+        system1:
+          java_version: java-8
+          tests: linpack
+          system_type: aws
+          host_config: "m5.xlarge"
+        system2:
+          java_version: java-8
+          tests: linpack
+          system_type: aws
+          host_config: "m5.4xlarge"
+
+#### Example 3 
+Run fio on AWS m5.xlarge (2 disks) and m5.12xlarge (4 disks); disks are 6TB, and type is gp2:
+
+    global:
+        ssh_key_file: /home/test_user/permissions/aws_region_2_ssh_key
+        terminate_cloud: 1
+        cloud_os_id: ami-0fdea47967124a409
+        os_vendor: rhel
+        results_prefix: just_a_test
+        system_type: aws
+    systems:
+        system1:
+        tests: fio
+            host_config: "m5.xlarge:Disks;number=2;size=6000;type=gp2,m12.xlarge:Disks;number=4;size=6000;type=gp2
+
+#### Example 4 
+Run fio on AWS m5.xlarge with 2 disks, and then run uperf on AWS m5.xlarge, 1 network. Note the SYS_BARRIER that indicates we will pause at that point, wait for all tests to finish, and then start the next batch. The m5.xlarge created for fio will be terminated at the end of the fio run and a new m5.xlarge for the uperf run will be created:
+
+    global:
+        ssh_key_file: /home/test_user/permissions/aws_region_2_ssh_key
+        terminate_cloud: 1
+        cloud_os_id: ami-0fdea47967124a409
+        os_vendor: rhel
+        results_prefix: just_a_test
+        system_type: aws
+    systems:
+        system1:
+            tests: fio
+            host_config: "m5.xlarge:Disks;number=2;size=6000;type=gp2"
+        system2:
+            host_config: "SYS_BARRIER"
+        system3:
+            tests: uperf
+            host_config: "m5.xlarge:Networks;number=1"