From 57fb3991995de76449e88f6698a829d5d02afa03 Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Mon, 29 Jul 2024 14:38:26 -0400 Subject: [PATCH 01/27] Mark old README as backup --- README.bak | 228 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 228 insertions(+) create mode 100644 README.bak diff --git a/README.bak b/README.bak new file mode 100644 index 0000000..527d4e9 --- /dev/null +++ b/README.bak @@ -0,0 +1,228 @@ +# +# Copyright (C) 2021 David Valin dvalin@redhat.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +# +# Zathras is an automation framework built in 5 layers. It is designed to be easily +# extendable to other cloud types, works on bare metal systems, and have new tests easily +# added to it. +# +General Requirements + 1) Have an account on targeted cloud provider + 2) Ansible installed on the system + 3) ssh files in config setup properly. + +Description +Zathras is an automation framework built in 5 layers. It is designed to be easily +extendable to other cloud types, works on bare metal systems, and have new tests easily +added to it. + +Name: The name Zathras is from the name of a character in the science fiction series, +Babylon 5. Zathras did not say much, but was instrumental in keeping The Great Machine +on Epsilon III running. One of the few things he did say was "Zathras is used to being beast +of burden to other peoples needs...very sad life...probably have very sad death...but, +at least there is symmetry!". For many reasons, that seems fitting for this automation +tool, thus the name. + +Layer 1: CLI +Usage: ./bin/burden --verbose +Version: 3.0 +General options + --archive /: location to save the archive information to + --child: tells burden it is a child of another burden process and not to + --dir_index: designates an index value to add to the run directory, default is 0 + --force_upload: upload all packages and tools to the cloud instance. + --git_timeout: Number of seconds to timeout on git requests. Default is 60 + --system_type : aws, azure or local + --host_config : Specification of the system and configuration + If the --system_type option is local, then this is simply the system name + to run on, and it will pull the config value from the file .conf + in local_configs + local_configs format: + server_ips: , + client_ips: , + storage: /dev/nvme2n1,/dev/nvme1n1 + + If the --system_type option is a cloud environment, then the following can + be specified + config_file format: + Fields definition: + instance_type: The cloud instance size (ie i3en.xlarge). + [region=&zone=] is totally optional + region: the region the cloud is created in, defaults to + whatever the user's default region is + zone: The zone in the region the cloud is to be created in, if not specified + will randomly pick one + number_networks: number of internal networks to create + sysctl_settings: files in sysctl_setting to use. Each file + sets a set of tunables, separator is + + number_of_disks: How many disks to create and attach + disk_size: How large is the disk in gigabytes + disk_type: Type of disk to be created + or + :&& + Fields definition: + &zone=: The cloud instance name (ie i3en.xlarge). + includes region and zone requests, both are optional. + + number: How many disks to create and attach + size: How large is the disk in gigabytes + type: Type of disk creating + + number_networks: number of internal networks to create + + sysctl_settings: files in sysctl_setting to use. Each file + sets a set of tunables, separator is + + System config file Examples + Example 1: Designate 2 systems, no config options + m5.xlarge,m5.4xlarge + Example 2: Designate m5.24xlarge, 8 gp2 disks of 1200 Gig + m5.24xlarge:Disks;number=8;size=1200;type=gp2 + Example 3: Designate m5.24xlarge with 4 networks + m5.24xlarge:Networks;number=4 + Example 4: Designate m5.24xlarge with sys tunings udp_fix and none + m5.24xlarge:Sysctl_settings=none+udp_fix + Example 5: Designate m5.xlarge to be created in us-east-1 and zone b + "m5.xlarge[region=us-east-1&zone=b]" + --java_version: java version to install, java-8, java-11 + --max_systems : Maximum number of systems to be running at once. 3 is the default + --no_yq_checks: skips checking for proper yq installation + --persistent_log: enable persistent logging, 0/1 + --results_prefix : Run directory prefix + --scenario : Reads in a scenario and then runs it + (if used, host configs are designated in the file). If the scenario name starts with https: or git: + then we are retrieving the scenario from a git repo. If the line in the scenario file starts with # + , then that line is a comment. If the line starts with a %, it indicates to replace the string. + Format to replace a string % = + --selinux_state: disabled/enabled + --ssh_key_file: Designates the ssh file we are to use. + --show_os_versions: given the cloud type, and OS vendor, show the available os versions + --show_tests: list the available test as defined in config/test_defs.yml + --test_def_file : test definition file to use. + --test_def_dir : test definition directory. Default is /config. If + https: or git: is at the start of the location, then we will pull from a git repo. + --test_override : Overrides the given options for a specific test in the scenario file + Example: + global: + ssh_key_file: /home/test_user/permissions/aws_region_2_ssh_key + terminate_cloud: 1 + cloud_os_id: aminumber + os_vendor: rhel + results_prefix: linpack + system_type: aws + systems: + system1: + java_version: java-8 + tests: linpack + system_type: aws + host_config: "m5.xlarge" + system2: + java_version: java-8 + tests: linpack + system_type: aws + host_config: "m5.4xlarge" + To override java_version for system1: + --test_override "system1:java_version=java-11" + --tests : testname, you may use "test1,test2" to execute test1 and test2. + Note if the option is present, we will ignore all other options passed in. + --test_iter : how many iterations of the test to run (includes linpack). + For cloud instances, this will terminate the cloud image and start + a new one for each iteration + --test_version ,....: Shows the versions of the test the are available + and brief description of the version. This only applies to git repos + --test_version_check: Checks to see if we are running the latest versions of the tests and + exits out when done. Default is no + --tuned_profiles , only for RHEL + --verbose: Verbose usage message + --update_target: Image to update + Note: only 1 update image can be used, makes no difference + if designate a different one for each system in the + scenario file, the first one will be used + -h --usage: condensed usage information +Cloud options only + --cloud_os_id : Image of the OS to install (example aws aminumber) + --create_only: Only do the VM creation and OS installaction. + --os_vendor : currently rhel, ubuntu, amazon + --terminate_cloud: If 1, terminate the cloud instance, if 0 leave the cloud image running. + Default is to terminate + --use_spot: uses spot pricing based on the contents of config/spot_price.cfg. Default is not + to use spot_pricing + +Layer 2: Cloud Creation + This layer creates the required cloud instances, and attaches any required devices. Note, + if doing network testing, another system of the same instance type is created and connected + via a private network. + +Layer 3: Test automation wrapper + This layer is a script that is wrapped around an existing test. Purpose of this wrapper is + to look at the system and configuration and determine what the test options should be. Each + of these automation scripts have their own set of parameters, which are expected to have a default + value that is either determined by the system configuration or simply a hard coded value. + This layer is also expected to handle the results if layer 4 does not. + +Layer 4: test + This later is the actual test running. It can be anything that can be automated. The test + are not part of the kit, but are currently stored in 1 of 2 places on perf1.perf.lab.eng.bos.redhat.com:/pub, either in + +Layer 5: Cleanup + This layer tears down the cloud instance, and any associated resources. + +RSA files +inventory: start of the ansible inventory file, simply localhost to start with + +Need following packages +dnf install python2-boto.noarch +dnf install python2-boto3.noarch +dnf install python2-botocore.noarch +dnf install python3-boto.noarch +dnf install dialog.x86_64 + +Using scenario_vars +To run the general scenario regression + +./burden --scenario regression_scenario --system_type aws --scenario_vars definition_vars + +That will run the regression scenario in the AWS environment. + +Order of option selection + +There are 3 places that the options to burden may be placed in. The following is the order in +which those options are acquired. + +1) command line options to burden. These override any options designated in the scenario vars + file, or the scenario file +2) scenario vars file (--scenario_vars), these are overridden by the cli, but override the options + designated in the scenario file. +3) scenario file (--scenario), these are overriden by the options designated either via the cli or + the scenario vars file. + +Execution example + +aws_example: + +global: + ssh_key_file: /home/user/permissions/aws_region_2_ssh_key + terminate_cloud: 1 + cloud_os_id: ami-myaminumbe + os_vendor: rhel + results_prefix: linpack + system_type: aws +systems: + system1: + tests: linpack + host_config: "m5.xlarge" + + +# ./burden --scenario aws_sample --test_def_dir https://git.com/user/zathras_config From 6846db5beb8d33ae8ba0cacb1e7a7861651604c5 Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Mon, 29 Jul 2024 14:40:31 -0400 Subject: [PATCH 02/27] Begin new README markdown file --- README.md | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..6b2d5ef --- /dev/null +++ b/README.md @@ -0,0 +1,2 @@ +## Table of Contents +- [About] From 4db2718849905bcc667f3669fdabf9f115e0a310 Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Mon, 29 Jul 2024 14:41:41 -0400 Subject: [PATCH 03/27] Remove original README file --- README | 228 --------------------------------------------------------- 1 file changed, 228 deletions(-) delete mode 100644 README diff --git a/README b/README deleted file mode 100644 index 527d4e9..0000000 --- a/README +++ /dev/null @@ -1,228 +0,0 @@ -# -# Copyright (C) 2021 David Valin dvalin@redhat.com -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU General Public License -# as published by the Free Software Foundation; either version 2 -# of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program; if not, write to the Free Software -# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -# -# Zathras is an automation framework built in 5 layers. It is designed to be easily -# extendable to other cloud types, works on bare metal systems, and have new tests easily -# added to it. -# -General Requirements - 1) Have an account on targeted cloud provider - 2) Ansible installed on the system - 3) ssh files in config setup properly. - -Description -Zathras is an automation framework built in 5 layers. It is designed to be easily -extendable to other cloud types, works on bare metal systems, and have new tests easily -added to it. - -Name: The name Zathras is from the name of a character in the science fiction series, -Babylon 5. Zathras did not say much, but was instrumental in keeping The Great Machine -on Epsilon III running. One of the few things he did say was "Zathras is used to being beast -of burden to other peoples needs...very sad life...probably have very sad death...but, -at least there is symmetry!". For many reasons, that seems fitting for this automation -tool, thus the name. - -Layer 1: CLI -Usage: ./bin/burden --verbose -Version: 3.0 -General options - --archive /: location to save the archive information to - --child: tells burden it is a child of another burden process and not to - --dir_index: designates an index value to add to the run directory, default is 0 - --force_upload: upload all packages and tools to the cloud instance. - --git_timeout: Number of seconds to timeout on git requests. Default is 60 - --system_type : aws, azure or local - --host_config : Specification of the system and configuration - If the --system_type option is local, then this is simply the system name - to run on, and it will pull the config value from the file .conf - in local_configs - local_configs format: - server_ips: , - client_ips: , - storage: /dev/nvme2n1,/dev/nvme1n1 - - If the --system_type option is a cloud environment, then the following can - be specified - config_file format: - Fields definition: - instance_type: The cloud instance size (ie i3en.xlarge). - [region=&zone=] is totally optional - region: the region the cloud is created in, defaults to - whatever the user's default region is - zone: The zone in the region the cloud is to be created in, if not specified - will randomly pick one - number_networks: number of internal networks to create - sysctl_settings: files in sysctl_setting to use. Each file - sets a set of tunables, separator is + - number_of_disks: How many disks to create and attach - disk_size: How large is the disk in gigabytes - disk_type: Type of disk to be created - or - :&& - Fields definition: - &zone=: The cloud instance name (ie i3en.xlarge). - includes region and zone requests, both are optional. - - number: How many disks to create and attach - size: How large is the disk in gigabytes - type: Type of disk creating - - number_networks: number of internal networks to create - - sysctl_settings: files in sysctl_setting to use. Each file - sets a set of tunables, separator is + - System config file Examples - Example 1: Designate 2 systems, no config options - m5.xlarge,m5.4xlarge - Example 2: Designate m5.24xlarge, 8 gp2 disks of 1200 Gig - m5.24xlarge:Disks;number=8;size=1200;type=gp2 - Example 3: Designate m5.24xlarge with 4 networks - m5.24xlarge:Networks;number=4 - Example 4: Designate m5.24xlarge with sys tunings udp_fix and none - m5.24xlarge:Sysctl_settings=none+udp_fix - Example 5: Designate m5.xlarge to be created in us-east-1 and zone b - "m5.xlarge[region=us-east-1&zone=b]" - --java_version: java version to install, java-8, java-11 - --max_systems : Maximum number of systems to be running at once. 3 is the default - --no_yq_checks: skips checking for proper yq installation - --persistent_log: enable persistent logging, 0/1 - --results_prefix : Run directory prefix - --scenario : Reads in a scenario and then runs it - (if used, host configs are designated in the file). If the scenario name starts with https: or git: - then we are retrieving the scenario from a git repo. If the line in the scenario file starts with # - , then that line is a comment. If the line starts with a %, it indicates to replace the string. - Format to replace a string % = - --selinux_state: disabled/enabled - --ssh_key_file: Designates the ssh file we are to use. - --show_os_versions: given the cloud type, and OS vendor, show the available os versions - --show_tests: list the available test as defined in config/test_defs.yml - --test_def_file : test definition file to use. - --test_def_dir : test definition directory. Default is /config. If - https: or git: is at the start of the location, then we will pull from a git repo. - --test_override : Overrides the given options for a specific test in the scenario file - Example: - global: - ssh_key_file: /home/test_user/permissions/aws_region_2_ssh_key - terminate_cloud: 1 - cloud_os_id: aminumber - os_vendor: rhel - results_prefix: linpack - system_type: aws - systems: - system1: - java_version: java-8 - tests: linpack - system_type: aws - host_config: "m5.xlarge" - system2: - java_version: java-8 - tests: linpack - system_type: aws - host_config: "m5.4xlarge" - To override java_version for system1: - --test_override "system1:java_version=java-11" - --tests : testname, you may use "test1,test2" to execute test1 and test2. - Note if the option is present, we will ignore all other options passed in. - --test_iter : how many iterations of the test to run (includes linpack). - For cloud instances, this will terminate the cloud image and start - a new one for each iteration - --test_version ,....: Shows the versions of the test the are available - and brief description of the version. This only applies to git repos - --test_version_check: Checks to see if we are running the latest versions of the tests and - exits out when done. Default is no - --tuned_profiles , only for RHEL - --verbose: Verbose usage message - --update_target: Image to update - Note: only 1 update image can be used, makes no difference - if designate a different one for each system in the - scenario file, the first one will be used - -h --usage: condensed usage information -Cloud options only - --cloud_os_id : Image of the OS to install (example aws aminumber) - --create_only: Only do the VM creation and OS installaction. - --os_vendor : currently rhel, ubuntu, amazon - --terminate_cloud: If 1, terminate the cloud instance, if 0 leave the cloud image running. - Default is to terminate - --use_spot: uses spot pricing based on the contents of config/spot_price.cfg. Default is not - to use spot_pricing - -Layer 2: Cloud Creation - This layer creates the required cloud instances, and attaches any required devices. Note, - if doing network testing, another system of the same instance type is created and connected - via a private network. - -Layer 3: Test automation wrapper - This layer is a script that is wrapped around an existing test. Purpose of this wrapper is - to look at the system and configuration and determine what the test options should be. Each - of these automation scripts have their own set of parameters, which are expected to have a default - value that is either determined by the system configuration or simply a hard coded value. - This layer is also expected to handle the results if layer 4 does not. - -Layer 4: test - This later is the actual test running. It can be anything that can be automated. The test - are not part of the kit, but are currently stored in 1 of 2 places on perf1.perf.lab.eng.bos.redhat.com:/pub, either in - -Layer 5: Cleanup - This layer tears down the cloud instance, and any associated resources. - -RSA files -inventory: start of the ansible inventory file, simply localhost to start with - -Need following packages -dnf install python2-boto.noarch -dnf install python2-boto3.noarch -dnf install python2-botocore.noarch -dnf install python3-boto.noarch -dnf install dialog.x86_64 - -Using scenario_vars -To run the general scenario regression - -./burden --scenario regression_scenario --system_type aws --scenario_vars definition_vars - -That will run the regression scenario in the AWS environment. - -Order of option selection - -There are 3 places that the options to burden may be placed in. The following is the order in -which those options are acquired. - -1) command line options to burden. These override any options designated in the scenario vars - file, or the scenario file -2) scenario vars file (--scenario_vars), these are overridden by the cli, but override the options - designated in the scenario file. -3) scenario file (--scenario), these are overriden by the options designated either via the cli or - the scenario vars file. - -Execution example - -aws_example: - -global: - ssh_key_file: /home/user/permissions/aws_region_2_ssh_key - terminate_cloud: 1 - cloud_os_id: ami-myaminumbe - os_vendor: rhel - results_prefix: linpack - system_type: aws -systems: - system1: - tests: linpack - host_config: "m5.xlarge" - - -# ./burden --scenario aws_sample --test_def_dir https://git.com/user/zathras_config From ea54de54b04c7383cebd553d6283850ac5f49b16 Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Tue, 30 Jul 2024 13:53:36 -0400 Subject: [PATCH 04/27] Create testing quickstart doc --- docs/testing_quickstart.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 docs/testing_quickstart.md diff --git a/docs/testing_quickstart.md b/docs/testing_quickstart.md new file mode 100644 index 0000000..cdcbc5d --- /dev/null +++ b/docs/testing_quickstart.md @@ -0,0 +1,30 @@ +# Testing with Zathras - Quickstart + +## Testing on bare metal + + +## Testing on cloud + +### Testing example using AWS: + +We could define a scenario file, "aws_example", as follows: + + global: + ssh_key_file: /home/user/permissions/aws_region_2_ssh_key + terminate_cloud: 1 + cloud_os_id: ami-myaminumbe + os_vendor: rhel + results_prefix: linpack + system_type: aws + systems: + system1: + tests: linpack + host_config: "m5.xlarge" + + +We could then begin running tests by entering in the command line: + + ./burden --scenario aws_sample + + +## Testing in virtualization From 82adbbc49d959be6b7df42442e27dcbe37dce22d Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Tue, 30 Jul 2024 13:54:11 -0400 Subject: [PATCH 05/27] Create doc explaining scenario files --- docs/scenario_files.md | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 docs/scenario_files.md diff --git a/docs/scenario_files.md b/docs/scenario_files.md new file mode 100644 index 0000000..7e932a3 --- /dev/null +++ b/docs/scenario_files.md @@ -0,0 +1,2 @@ +## Testing with scenario files +Using a scenario file to configure tests offers several advantages over setting the parameters via the command line. \ No newline at end of file From a51d3992bf898a315079885d3042a27797dd344e Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Tue, 30 Jul 2024 13:55:12 -0400 Subject: [PATCH 06/27] Update draft of primary repo README --- README.md | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 57 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 6b2d5ef..c1736ca 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,58 @@ ## Table of Contents -- [About] +- [About](#about) +- [How It Works](#how-it-works) +- [Installation and Quick Start](#installation-and-quick-start) +- [License](#license) + +## About +Zathras is a test automation framework for use with bare metal, cloud, and virtualized systems. Tests are defined by wrappers - external, self-contained scripts that enable easy extension of Zathras's functionality. + +Currently, Zathras offers the following test wrappers: + +| Test Name | Description | Wrapper URL | | | +|----------------------------------------------------|-------------------------------------------------------------------------------------|------------------------------------------------------------|---|---| +| CoreMark | Test processor core functionality | https://github.com/redhat-performance/coremark-wrapper | | | +| CoreMark Pro | EEMBC's advanced processor benchmark. | https://github.com/redhat-performance/coremark_pro-wrapper | | | +| Flexible I/O tester | Test the Linux I/O subsystem and schedulers | https://github.com/redhat-performance/fio-wrapper | | | +| HPL | Freely available implementation of the High Performance Computing Linpack Benchmark | https://github.com/redhat-performance/autohpl-wrapper | | | +| HammerDB | Database benchmarking | https://github.com/redhat-performance/hammerdb-wrapper | | | +| IOzone | File system benchmarking | https://github.com/redhat-performance/iozone-wrapper | | | +| Linpack | Licensed version of Linpack benchmark (see also: HPL) | https://github.com/redhat-performance/linpack-wrapper | | | +| NUMA STREAM | Extension of STREAM benchmark to measure NUMA effects | https://github.com/redhat-performance/numa_streams-wrapper | | | +| Passmark | CPU benchmark | https://github.com/redhat-performance/passmark-wrapper | | | +| pig | Processor scheduler efficiency test | https://github.com/redhat-performance/pig_wrapper | | | +| Python Performance Benchmark Suite (pyperformance) | Performance testing for Python using real-world benchmarks | https://github.com/redhat-performance/pyperf-wrapper | | | +| SPEC CPU 2017 | Suite of benchmarks for compute-intensive performance | https://github.com/redhat-performance/speccpu2017-wrapper | | | +| SPEC JBB2005 | Evaluate the performance of server side Java | https://github.com/redhat-performance/specjbb-wrapper | | | +| STREAM | Benchmark for sustained memory bandwidth | https://github.com/redhat-performance/streams-wrapper | | | +| Uperf | Network performance testing | https://github.com/redhat-performance/uperf-wrapper | | | + +## How It Works +1. Zathras is installed to a controller system. +2. Zathras runs using either a configuration file or by supplying parameters in your command-line interface (CLI). +3. Test(s) run on the system under test (SUT). +4. When tests finish, Zathras copies the results back to the controller system. +5. Once results have been retrieved, Zathras will attempt to cleanup the testing environment (such as by tearing down instances in the cloud or by removing directories and files on bare metal). + +The user configures the Zathras installation with information about the system under test (SUT) and what test(s) to run. + +## Getting Started + +### Requirements +- git + +### Installation +First, clone this repository to your controller system. + +### Testing Quickstarts +In general, testing with Zathras follows a similar set of steps across modalities: configure the tests to run, then run burden via the command line. + + + +However, there are some aspects that are particular to the testing environment being used. + +- Testing on bare metal +- Testing on cloud +- Testing on virtualization + +## License From d52621505dc502e7d1e7169a762c2dc66803ca10 Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Tue, 30 Jul 2024 14:14:11 -0400 Subject: [PATCH 07/27] Remove unused columns from wrappers table --- README.md | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index c1736ca..3824f47 100644 --- a/README.md +++ b/README.md @@ -9,23 +9,23 @@ Zathras is a test automation framework for use with bare metal, cloud, and virtu Currently, Zathras offers the following test wrappers: -| Test Name | Description | Wrapper URL | | | -|----------------------------------------------------|-------------------------------------------------------------------------------------|------------------------------------------------------------|---|---| -| CoreMark | Test processor core functionality | https://github.com/redhat-performance/coremark-wrapper | | | -| CoreMark Pro | EEMBC's advanced processor benchmark. | https://github.com/redhat-performance/coremark_pro-wrapper | | | -| Flexible I/O tester | Test the Linux I/O subsystem and schedulers | https://github.com/redhat-performance/fio-wrapper | | | -| HPL | Freely available implementation of the High Performance Computing Linpack Benchmark | https://github.com/redhat-performance/autohpl-wrapper | | | -| HammerDB | Database benchmarking | https://github.com/redhat-performance/hammerdb-wrapper | | | -| IOzone | File system benchmarking | https://github.com/redhat-performance/iozone-wrapper | | | -| Linpack | Licensed version of Linpack benchmark (see also: HPL) | https://github.com/redhat-performance/linpack-wrapper | | | -| NUMA STREAM | Extension of STREAM benchmark to measure NUMA effects | https://github.com/redhat-performance/numa_streams-wrapper | | | -| Passmark | CPU benchmark | https://github.com/redhat-performance/passmark-wrapper | | | -| pig | Processor scheduler efficiency test | https://github.com/redhat-performance/pig_wrapper | | | -| Python Performance Benchmark Suite (pyperformance) | Performance testing for Python using real-world benchmarks | https://github.com/redhat-performance/pyperf-wrapper | | | -| SPEC CPU 2017 | Suite of benchmarks for compute-intensive performance | https://github.com/redhat-performance/speccpu2017-wrapper | | | -| SPEC JBB2005 | Evaluate the performance of server side Java | https://github.com/redhat-performance/specjbb-wrapper | | | -| STREAM | Benchmark for sustained memory bandwidth | https://github.com/redhat-performance/streams-wrapper | | | -| Uperf | Network performance testing | https://github.com/redhat-performance/uperf-wrapper | | | +| Test Name | Description | Wrapper URL | +|----------------------------------------------------|-------------------------------------------------------------------------------------|------------------------------------------------------------| +| CoreMark | Test processor core functionality | https://github.com/redhat-performance/coremark-wrapper | +| CoreMark Pro | EEMBC's advanced processor benchmark. | https://github.com/redhat-performance/coremark_pro-wrapper | +| Flexible I/O tester | Test the Linux I/O subsystem and schedulers | https://github.com/redhat-performance/fio-wrapper | +| HPL | Freely available implementation of the High Performance Computing Linpack Benchmark | https://github.com/redhat-performance/autohpl-wrapper | +| HammerDB | Database benchmarking | https://github.com/redhat-performance/hammerdb-wrapper | +| IOzone | File system benchmarking | https://github.com/redhat-performance/iozone-wrapper | +| Linpack | Licensed version of Linpack benchmark (see also: HPL) | https://github.com/redhat-performance/linpack-wrapper | +| NUMA STREAM | Extension of STREAM benchmark to measure NUMA effects | https://github.com/redhat-performance/numa_streams-wrapper | +| Passmark | CPU benchmark | https://github.com/redhat-performance/passmark-wrapper | +| pig | Processor scheduler efficiency test | https://github.com/redhat-performance/pig_wrapper | +| Python Performance Benchmark Suite (pyperformance) | Performance testing for Python using real-world benchmarks | https://github.com/redhat-performance/pyperf-wrapper | +| SPEC CPU 2017 | Suite of benchmarks for compute-intensive performance | https://github.com/redhat-performance/speccpu2017-wrapper | +| SPEC JBB2005 | Evaluate the performance of server side Java | https://github.com/redhat-performance/specjbb-wrapper | +| STREAM | Benchmark for sustained memory bandwidth | https://github.com/redhat-performance/streams-wrapper | +| Uperf | Network performance testing | https://github.com/redhat-performance/uperf-wrapper | ## How It Works 1. Zathras is installed to a controller system. From d789126dd91aa428769d797ea747255dab871565 Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Tue, 30 Jul 2024 14:31:55 -0400 Subject: [PATCH 08/27] Add installation instructions --- README.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/README.md b/README.md index 3824f47..9f7c071 100644 --- a/README.md +++ b/README.md @@ -44,6 +44,13 @@ The user configures the Zathras installation with information about the system u ### Installation First, clone this repository to your controller system. +Installation can then be completed by running the install script located within the repo's bin/ directory: + + ./install.sh + + +Note: for most installations the install script should be run as a user, not root. + ### Testing Quickstarts In general, testing with Zathras follows a similar set of steps across modalities: configure the tests to run, then run burden via the command line. From cb5f408b3d8a09bb72a474539a111d83a2a4e656 Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Tue, 30 Jul 2024 14:32:18 -0400 Subject: [PATCH 09/27] Create cli reference doc --- docs/command_line_reference.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/command_line_reference.md diff --git a/docs/command_line_reference.md b/docs/command_line_reference.md new file mode 100644 index 0000000..28b6a2d --- /dev/null +++ b/docs/command_line_reference.md @@ -0,0 +1 @@ +# Command line interface (CLI) reference From 92bbd3d91eb78c9f109e22d35496f56ab0096287 Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Tue, 30 Jul 2024 17:11:09 -0400 Subject: [PATCH 10/27] Create initial version of CLI reference doc --- docs/command_line_reference.md | 177 +++++++++++++++++++++++++++++++++ 1 file changed, 177 insertions(+) diff --git a/docs/command_line_reference.md b/docs/command_line_reference.md index 28b6a2d..8badc9a 100644 --- a/docs/command_line_reference.md +++ b/docs/command_line_reference.md @@ -1 +1,178 @@ # Command line interface (CLI) reference + +## Usage + + ./bin/burden [...] + +## Options + +### --archive \/\ +Sets the location where the archive information will be saved. + +### --child +Tells burden it is a child of another burden process and not to perform the initial setup work. + +### --create_attempts +Number of times we attempt to create an instance to get the desginated cpu type. + +### --git_timeout +Number of seconds to timeout on git requests. Default is 60. + +### --system_type \ +aws, azure, gcp or local. + +### --host_config \ +Specification of the system and configuration. Possible values are "cloud" and "local". + +If the --system_type option is local, then this is simply the system name to run on, and it will pull the config value from the file .conf in the local_configs directory. For more information on local_configs, please see (TODO: add link). + +If the --system_type option is cloud, then the following fields may be specified in the config file: +- instance type: The cloud instance (ie i3en.xlarge). Inclusion of \[region=\&zone=\] is optional. +- region: The region where the cloud is created. The default option is the user's default region. +- zone: The zone within the region the cloud will be created in. Randomly chosen by default. + +### --ignore_repo_errors +If present we will ignore repo errors, default is to abort the run when a repo error occurs. + +### --individ_vars +Contains various burden settings. Takes precedence over the scenario file, but is overridden by the command line. Default is config zathras_specific_vals_def. + +### --java_version +Java version to install, java-8, java-11. + +### --kit_upload_directory +Full path to directory uploading to. If not present, Zathras will locate the filesystem with the most space on it and use that location. + +### --max_systems \ +Maximum number of burden subinstances that will be created from the parent. Each subinstance is a cloud or local system. 3 is the default. + +### --no_clean_up +Do not cleanup at the end of the test. + +### --no_packages +Do not install any packages, default is no. + +### --no_pbench_install +Do not install pbench. The default is 0 (install pbench). + +### --no_spot_recover +Do not recover from a spot system going away. + +### --package_name \ +Use this set of packages to override the default in the test config file instead of the default. Default format package name \_pkg, new name \_pkg_\. + +### --persistent_log +Enable persistent logging. + +### --preflight_check +Performs various checks on the scenario file, and Zathras and then exits. + +### --results_prefix \ +Run directory prefix + +### --retry_failed_tests \<0/1> +Indicates to retry any detected failed tests if set to 1 (1 is the default). + +### --scenario \ +Reads in a scenario and then runs it (if used, host configs are designated in the file). + +If the scenario name starts with https: or git:, then the scenario file is retrieved from a git repo. + +If the line in the scenario file starts with #, then that line is a comment. + +If the line starts with a %, it indicates to replace the string. + +Format to replace a string % \=\ + +### --scenario_vars +File that contains the variables for the scenario file, default is config/zathras_scenario_vars_def. + +### --selinux_level +enforcing/permissive/disabled. The setting to have in /etc/selinux/config file. Note: Zathras does not support enforcing in Ubuntu at this time. + +### --selinux_state +disabled/enabled. If disabled is selected, selinux will be disabled via grubby (Amazon and RHEL). For Ubuntu, +enabled will install the require packages, update the config file and reboot. + +### --ssh_key_file +Designates the ssh key file we are to use. + +### --show_os_versions +Given the cloud type and OS vendor, show the available operating system versions. + +### --show_tests +List the available test as defined in config/test_defs.yml + +### --test_def_file \ +Test definition file to use. + +### --test_def_dir \ +Test definition directory. Default is \/config. If https: or git: is at the start of the location, then we will pull from a git repo. + +### --test_override \ +Overrides the given options for a specific test in the scenario file. + +### --tests \ +testname, you may use "test1,test2" to execute test1 and test2. + +### --test_iter \ +How many iterations of the test to run (includes linpack). +For cloud instances, this will terminate the cloud image and start a new image for each iteration. + +### --test_versions \,\.... +Shows the versions of the test the are available and brief description of the version. This only applies to git repos + +### --test_version_check +Checks to see if we are running the latest versions of the tests and exits out when done. Default is no. + +### --tf_list +List active systems created via tf (terraform?). + +### --tf_terminate_list +Delete the designated terraform systems. + +### --tf_terminate_all +Go into each terraform directory and attempt to remove the terraform instance. + +### --tuned_profiles \ +Note: only for RHEL. Designates the tuned profiles to use. if the system type is a cloud environment, then each tuned profile is a distinct cloud instance. + +### --tuned_reboot \<> +Note: only for RHEL. If value is 1 we will reboot the system each time a new tuned profile is set. + +### --verbose +Verbose usage message. + +### --upload_rpms ,.... +Comma separated list of rpms (full path) to upload and install. + +### --update_target +Image to update. + +Note: only 1 update image can be used, makes no difference if designate a different one for each system in the scenario file, the first one will be used. + +### --use_latest_version +Will update the templates so we are using the latest versions of the test (git repos only). + +### -h --usage +Condensed usage information. + +## Cloud-only options + +## --cloud_os_id \ +Image of the OS to install (example aws aminumber) +For multiple architectures, this is allowed" +x86:ami-0fbec8a0a2beb6a71,arm64:ami-0cfa90ca3ebfc506e" +Burden will select the right ami for the designated host." + +### --create_only +Only do the VM creation and OS install action. + +### --os_vendor \ +Currently rhel, ubuntu, amazon. + +### --terminate_cloud +If 1, terminate the cloud instance, if 0 leave the cloud image running. Default is to terminate. + +### --use_spot +Uses spot pricing based on the contents of config/spot_price.cfg. Default is not to use spot_pricing \ No newline at end of file From f9501cd1b1721a17eaa39e8ca49d1424bd312d89 Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Tue, 30 Jul 2024 17:11:32 -0400 Subject: [PATCH 11/27] Expand Getting Started section --- README.md | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 9f7c071..2079c86 100644 --- a/README.md +++ b/README.md @@ -39,10 +39,22 @@ The user configures the Zathras installation with information about the system u ## Getting Started ### Requirements -- git +To use Zathras, you need: +1. A system that will coordinate the execution of tests, which we will call the *control node* or simply the *controller*. +2. One or more systems where tests will be run. This can be another physical system (aka bare metal), an instance on a public cloud like Amazon Web Services (AWS), or even a virtual machine (VM) either on the controller or another host system. + +The following packages will need to be installed on the controller to run Zathras. If you use the included installer script (next section), they will be installed for you. +- ansible-core +- git +- jq +- python +- python3-pip +- terraform +- yq, version 2.10.0 (newer versions currently break Zathras) +- wget ### Installation -First, clone this repository to your controller system. +Installing Zathras is easy! First, clone this repository to your controller system. Installation can then be completed by running the install script located within the repo's bin/ directory: @@ -51,6 +63,9 @@ Installation can then be completed by running the install script located within Note: for most installations the install script should be run as a user, not root. +### Test config files + + ### Testing Quickstarts In general, testing with Zathras follows a similar set of steps across modalities: configure the tests to run, then run burden via the command line. From b57abf06b62efc4fcae6e3fbbd8911481b93d771 Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Tue, 30 Jul 2024 19:59:09 -0400 Subject: [PATCH 12/27] Update README getting started section --- README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 2079c86..2ae3b0d 100644 --- a/README.md +++ b/README.md @@ -56,6 +56,8 @@ The following packages will need to be installed on the controller to run Zathra ### Installation Installing Zathras is easy! First, clone this repository to your controller system. + git clone https://github.com/redhat-performance/zathras.git + Installation can then be completed by running the install script located within the repo's bin/ directory: ./install.sh @@ -64,14 +66,13 @@ Installation can then be completed by running the install script located within Note: for most installations the install script should be run as a user, not root. ### Test config files - +Test config files, stored in the config/ directory of the Zathras installation, are the connective tissue between Zathras and the test wrappers. Currently this Zathras repo does not contain any defined test config files. Before you can use this tool, you will need to write your own test configs using the included example.yml, or add the configs from an external source if one is available to you. ### Testing Quickstarts In general, testing with Zathras follows a similar set of steps across modalities: configure the tests to run, then run burden via the command line. - -However, there are some aspects that are particular to the testing environment being used. +However, there are some aspects that are particular to the testing environment being used. Please refer to the following documentation that matches your use case: - Testing on bare metal - Testing on cloud From 6dd0e6c79094bb4247b033e493dc462d23e28b07 Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Tue, 30 Jul 2024 20:03:34 -0400 Subject: [PATCH 13/27] Update table of contents --- README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 2ae3b0d..301ee4d 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,11 @@ ## Table of Contents - [About](#about) - [How It Works](#how-it-works) -- [Installation and Quick Start](#installation-and-quick-start) +- [Getting started](#getting-started) + - [Requirements](#requirements) + - [Installation](#installation) + - [Test config files](#test-config-files) + - [Testing Quick Start](#installation-and-quick-start) - [License](#license) ## About From 0175f4170b82d6471435cf29a87c3ff908ae092c Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Tue, 30 Jul 2024 20:15:11 -0400 Subject: [PATCH 14/27] Update README.md Add link to CLI reference doc --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 301ee4d..81b8e63 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ Currently, Zathras offers the following test wrappers: ## How It Works 1. Zathras is installed to a controller system. -2. Zathras runs using either a configuration file or by supplying parameters in your command-line interface (CLI). +2. Zathras runs using either a configuration file or by supplying parameters in your command-line interface (CLI) (see the [CLI reference](docs/command_line_reference.md) for more info). 3. Test(s) run on the system under test (SUT). 4. When tests finish, Zathras copies the results back to the controller system. 5. Once results have been retrieved, Zathras will attempt to cleanup the testing environment (such as by tearing down instances in the cloud or by removing directories and files on bare metal). From 1b5298b00e96399761df0606725efd2c7249ceac Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Thu, 1 Aug 2024 13:39:38 -0400 Subject: [PATCH 15/27] Add outline of baremetal and virt testing process --- docs/testing_quickstart.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/testing_quickstart.md b/docs/testing_quickstart.md index cdcbc5d..b6c6a9b 100644 --- a/docs/testing_quickstart.md +++ b/docs/testing_quickstart.md @@ -1,6 +1,10 @@ # Testing with Zathras - Quickstart ## Testing on bare metal +- Define a scenario file +- ssh-copy-id from the SUT to the controller +- Add a local_config for the SUT +- Add SUT to the controller's known hosts ## Testing on cloud @@ -28,3 +32,10 @@ We could then begin running tests by entering in the command line: ## Testing in virtualization +If the virtual machine is hosted on the controller, then the process follows the same steps as testing on bare metal. +- Define a scenario file +- ssh-copy-id from the SUT to the controller +- Add a local_config for the SUT +- Add SUT to the controller's known hosts + +If the virtual machine is hosted on an external system, then you will need to create a network bridge so that Zathras can reach the SUT. From 9a7bb44a1439f00d9b2085e987c4bae141f08f74 Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Thu, 1 Aug 2024 14:11:07 -0400 Subject: [PATCH 16/27] Exapnd scenario files doc --- docs/scenario_files.md | 97 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 95 insertions(+), 2 deletions(-) diff --git a/docs/scenario_files.md b/docs/scenario_files.md index 7e932a3..353e09f 100644 --- a/docs/scenario_files.md +++ b/docs/scenario_files.md @@ -1,2 +1,95 @@ -## Testing with scenario files -Using a scenario file to configure tests offers several advantages over setting the parameters via the command line. \ No newline at end of file +# Scenario files +Using a scenario file to configure tests offers several advantages over setting the parameters via the command line. Scenario files provide more flexibility in defining tests, greater ease of modifying tests, improved test reproducibility, and enable the creation of more complex test runs. Once a scenario file is created, that scenario can also easily be exported for future use. + +## Scenario file format + + global: + + systems: + system1: + + +The outline above can also easily be extended to incorporate multiple systems within a single scenario, like so: + + global: + + systems: + system1: + + system2: + + +## Scenario file options +The options in the parameters file maps directly back to the CLI options, minus the dash. + + +## Using scenario files + +Scenario files are placed within your Zathras working directory, and can be given any name. For example, given a scenario file `my_scenario`, the following command would begin a Zathras run using that scenario file: + + ./bin/burden --scenario my_scenario + +## Examples + +### Scenario file for bare metal +Example 1: This scenario would run the STREAM benchmark on a local bare metal system named "test_sys". + + global: + result_prefix: just_a_test + system_type: local + systems: + system1: + tests: streams + host_config: "test_sys" + +### Scenario files for cloud +Example 2: Create an AWS system, list system information, then delete the instance. + + global: + ssh_key_file: replace_your_ssh_key + terminate_cloud: 0 + os_vendor: rhel + results_prefix: create_only + os_vendor: rhel + system_type: aws + systems: + system1: + cloud_os_id: ami-078cbc4c2d057c244 + host_config: "m6a.xlarge" + +Example 3: This scenario would run a test on public cloud using AWS with two hosts, "m5.xlarge" and "m5.12xlarge". + + global: + ssh_key_file: /home/test_user/permissions/aws_region_2_ssh_key + terminate_cloud: 1 + cloud_os_id: aminumber + os_vendor: rhel + results_prefix: linpack + system_type: aws + systems: + system1: + java_version: java-8 + tests: linpack + system_type: aws + host_config: "m5.xlarge" + system2: + java_version: java-8 + tests: linpack + system_type: aws + host_config: "m5.4xlarge" + +Run fio on AWS m5.xlarge (2 disks) and m5.12xlarge (4 disks); disks are 6TB, and type is gp2: + + global: + ssh_key_file: /home/test_user/permissions/aws_region_2_ssh_key + terminate_cloud: 1 + cloud_os_id: ami-0fdea47967124a409 + os_vendor: rhel + results_prefix: just_a_test + system_type: aws + systems: + system1: + tests: fio + host_config: "m5.xlarge:Disks;number=2;size=6000;type=gp2,m12.xlarge:Disks;number=4;size=6000;type=gp2 + +Run fio on AWS m5.xlarge with 2 disks, and then run uperf on AWS m5.xlarge, 1 network. Note the SYS_BARRIER that indicates we will pause at that point, wait for all tests to finish, and then start the next batch. The m5.xlarge created for fio will be terminated at the end of the fio run and a new m5.xlarge for the uperf run will be created: \ No newline at end of file From 267c7bd3864f59dcd2d120dbc7cfcb2a4e88b700 Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Thu, 1 Aug 2024 14:14:24 -0400 Subject: [PATCH 17/27] Reformat examples --- docs/scenario_files.md | 30 ++++++++++++++++++++++++++---- 1 file changed, 26 insertions(+), 4 deletions(-) diff --git a/docs/scenario_files.md b/docs/scenario_files.md index 353e09f..6902866 100644 --- a/docs/scenario_files.md +++ b/docs/scenario_files.md @@ -32,7 +32,8 @@ Scenario files are placed within your Zathras working directory, and can be give ## Examples ### Scenario file for bare metal -Example 1: This scenario would run the STREAM benchmark on a local bare metal system named "test_sys". +#### Example 1 +This scenario would run the STREAM benchmark on a local bare metal system named "test_sys". global: result_prefix: just_a_test @@ -43,7 +44,8 @@ Example 1: This scenario would run the STREAM benchmark on a local bare metal sy host_config: "test_sys" ### Scenario files for cloud -Example 2: Create an AWS system, list system information, then delete the instance. +#### Example 1 +Create an AWS system, list system information, then delete the instance. global: ssh_key_file: replace_your_ssh_key @@ -57,7 +59,8 @@ Example 2: Create an AWS system, list system information, then delete the instan cloud_os_id: ami-078cbc4c2d057c244 host_config: "m6a.xlarge" -Example 3: This scenario would run a test on public cloud using AWS with two hosts, "m5.xlarge" and "m5.12xlarge". +#### Example 2 +This scenario would run a test on public cloud using AWS with two hosts, "m5.xlarge" and "m5.12xlarge". global: ssh_key_file: /home/test_user/permissions/aws_region_2_ssh_key @@ -78,6 +81,7 @@ Example 3: This scenario would run a test on public cloud using AWS with two hos system_type: aws host_config: "m5.4xlarge" +#### Example 3 Run fio on AWS m5.xlarge (2 disks) and m5.12xlarge (4 disks); disks are 6TB, and type is gp2: global: @@ -92,4 +96,22 @@ Run fio on AWS m5.xlarge (2 disks) and m5.12xlarge (4 disks); disks are 6TB, and tests: fio host_config: "m5.xlarge:Disks;number=2;size=6000;type=gp2,m12.xlarge:Disks;number=4;size=6000;type=gp2 -Run fio on AWS m5.xlarge with 2 disks, and then run uperf on AWS m5.xlarge, 1 network. Note the SYS_BARRIER that indicates we will pause at that point, wait for all tests to finish, and then start the next batch. The m5.xlarge created for fio will be terminated at the end of the fio run and a new m5.xlarge for the uperf run will be created: \ No newline at end of file +#### Example 4 +Run fio on AWS m5.xlarge with 2 disks, and then run uperf on AWS m5.xlarge, 1 network. Note the SYS_BARRIER that indicates we will pause at that point, wait for all tests to finish, and then start the next batch. The m5.xlarge created for fio will be terminated at the end of the fio run and a new m5.xlarge for the uperf run will be created: + + global: + ssh_key_file: /home/test_user/permissions/aws_region_2_ssh_key + terminate_cloud: 1 + cloud_os_id: ami-0fdea47967124a409 + os_vendor: rhel + results_prefix: just_a_test + system_type: aws + systems: + system1: + tests: fio + host_config: "m5.xlarge:Disks;number=2;size=6000;type=gp2" + system2: + host_config: "SYS_BARRIER" + system3: + tests: uperf + host_config: "m5.xlarge:Networks;number=1" \ No newline at end of file From 3201f12b16fcc6ca614e70e6ccf13fd818ac7958 Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Thu, 1 Aug 2024 14:17:12 -0400 Subject: [PATCH 18/27] Mention test config file page --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 81b8e63..0f0c461 100644 --- a/README.md +++ b/README.md @@ -72,6 +72,8 @@ Note: for most installations the install script should be run as a user, not roo ### Test config files Test config files, stored in the config/ directory of the Zathras installation, are the connective tissue between Zathras and the test wrappers. Currently this Zathras repo does not contain any defined test config files. Before you can use this tool, you will need to write your own test configs using the included example.yml, or add the configs from an external source if one is available to you. +The process of creating test configuration files is further explained here. + ### Testing Quickstarts In general, testing with Zathras follows a similar set of steps across modalities: configure the tests to run, then run burden via the command line. From ab705b8cb5a331ceedfbc7c69c4feb673eb0a5b5 Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Thu, 1 Aug 2024 14:28:57 -0400 Subject: [PATCH 19/27] Add test config files doc --- docs/test_config_files.md | 142 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 142 insertions(+) create mode 100644 docs/test_config_files.md diff --git a/docs/test_config_files.md b/docs/test_config_files.md new file mode 100644 index 0000000..3770208 --- /dev/null +++ b/docs/test_config_files.md @@ -0,0 +1,142 @@ +# Test configuration files +The test configuration file is used to define a given test to Zathras, and specifies test options to use. + +Test config files are stored within the `config/` directory of your Zathras installation. + +## Test config file contents +The following shows all the possible options a test configuration file could contain (fields are explained below). In practice, a test config file does not require every field be included. + + test_defs: + test: + test_template: + test_name: + test_description: + repository_type: + location: + reboot_system: "yes/no" + test_run_from: "local/remote" + repo_file: + version: + test_grouping: + os_supported: + rhel_pkgs: + ubuntu_pkgs: + amazon_pkgs: + pbench_required: + pbench_local_results: + storage_required: + network_required: + java_required: + archive_results: + test_script_to_run: + test_specific: + upload_extra: + +## Field definitions +test: is a unique numerical value from the other tests. Other than for creation of the yml file, it has no bearing on the test execution. + +test_template: Is a pointer to a specific test template that will be brought in. Purpose is that when we need to make changes for a specifc test, we update the test_template file not the test_defs.yml. + +test_name: designates the name that zathras knows the test by. This name is what is passed in with the --tests option. + +test_description: A brief description of what the test does. This information is displayed when the --show_tests option is provided to burden. + +repository_type: The type of repository that the test is located in. Valid values include: + + git_tar: Mainly used when pulling an entire git repository for a test. + + git_file: Designated when pulling a single file from a git repository. + + git_tag: Will pull based on the git_tag. + + file: Indicates we are copying a file. + + tar: Indicates we are copying a tarball. + +location: Is the location at which the test can be found. Note in the case of git, it will be everything minus the actual location name. For example, if we have: + + https://github.com/redhat-performance/streams-wrapper/ + +We will drop the streams-wrapper on the end and have + + https://github.com/redhat-performance/ + + + +version: The version of the test to use. For a listing of versions available, you can issue the following command: + + $ ./burden --test_versions streams + ============================ + streams + ============================ + v1.0 Merge pull request #2 from redhat-performance/fix_usage + + + +test_grouping : Is the grouping name of the tests. In the case of a git repo, it will be the actual location name. For example, if we have: + + https://github.com/redhat-performance/streams-wrapper/ + +we drop everything but streams-wrapper and use "streams-wrapper" + +os_supported: What Operating Systems are supported. rhel, ubuntu, amazon or all. + +reboot_system: This is for cloud systems, and will reboot the cloud system once it has been created. + +test_run_from: Some tests need to run from the local system, not the cloud system (boot timing test is one example). If local is specified, the test will run locally, if remote is specified the test will run on the test system. + +repo_file: The archive file to be downloaded. In the case of git repos, it will be .zip + +rhel_pkgs: RHEL packages that are required to run the test; “none” if there aren’t any packaging requirements + +ubuntu_pkgs: Ubuntu packages that are required to run the test; “none” if there aren’t any packaging requirements. + +amazon_pkgs: Amazon packages that are required to run the test; “none” if there aren’t any packaging requirements. + +pbench_required: If set to yes, the test requires pbench to be installed to run. Zathras will install pbench on the test system for you. Only supported for RHEL/Fedora. Note: it is the responsibility of the user to make sure the pbench repos are set up properly. + +pbench_local_result: If set to yes, will tell the various wrapper scripts to consolidate the pbench results into a results tarball in /tmp with the format: + + results__.tar +e.g. + + results_pbench_linpack_tuned_virtual-guest_sys_file_none.tar + +storage_required: If set to yes, Zathras will check to make sure storage has been designated for the system before provisioning it. + +network_required: If set to yes, Zathras will check to make sure a network (other than the defaults) has been designated for the system before provisioning it. + +java_required: If set to yes, Zathras will make sure that a Java Version has been provided before provisioning any system. + +archive_results: Tell zathras that it has test results to retrieve from the test system. + +test_script_to_run: The script that Zathras is to run. + +test_specific: Test options that are specific to the test. + +upload_extra: If not set to “none”, uploads the each file in the comma separated list to the test system. + +## Example + + test3: + test_name: streams + exec_dir: "streams-wrapper-1.0/streams" + location: https://github.com/redhat-performance/streams-wrapper/archive/refs/tags + reboot_system: "no" + test_run_from: "remote" + os_supported: all + repo_file: "v1.0.zip" + rhel_pkgs: gcc,bc + ubuntu_pkgs: gcc,build-essential,libnuma-dev,zip,unzip + amazon_pkgs: gcc,bc,git,unzip,zip + pbench_required: "no" + pbench_local_results: "no" + storage_required: "no" + network_required: "no" + java_required: "no" + archive_results: "yes" + test_script_to_run: streams_run + test_specific: "" + upload_extra: "none" + post_script: "none" + pre_setup: "none" \ No newline at end of file From 0d18e2deb3a1f2051495cfb00b3895e24b96202c Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Thu, 1 Aug 2024 14:31:03 -0400 Subject: [PATCH 20/27] Link to test config files doc --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 0f0c461..050aeb9 100644 --- a/README.md +++ b/README.md @@ -72,7 +72,7 @@ Note: for most installations the install script should be run as a user, not roo ### Test config files Test config files, stored in the config/ directory of the Zathras installation, are the connective tissue between Zathras and the test wrappers. Currently this Zathras repo does not contain any defined test config files. Before you can use this tool, you will need to write your own test configs using the included example.yml, or add the configs from an external source if one is available to you. -The process of creating test configuration files is further explained here. +The process of creating test configuration files is further explained [here](https://github.com/redhat-performance/zathras/blob/documentation-rewrite/docs/test_config_files.md). ### Testing Quickstarts In general, testing with Zathras follows a similar set of steps across modalities: configure the tests to run, then run burden via the command line. From 72c16242b5e99760d5c5060e4380b9b607fba4ad Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Thu, 1 Aug 2024 14:32:09 -0400 Subject: [PATCH 21/27] Remove license from README.md --- README.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/README.md b/README.md index 050aeb9..c1f16c6 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ - [Installation](#installation) - [Test config files](#test-config-files) - [Testing Quick Start](#installation-and-quick-start) -- [License](#license) + ## About Zathras is a test automation framework for use with bare metal, cloud, and virtualized systems. Tests are defined by wrappers - external, self-contained scripts that enable easy extension of Zathras's functionality. @@ -83,5 +83,3 @@ However, there are some aspects that are particular to the testing environment b - Testing on bare metal - Testing on cloud - Testing on virtualization - -## License From e78b8da5330e4e582ca173b826420e025d5fd780 Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Thu, 1 Aug 2024 14:37:52 -0400 Subject: [PATCH 22/27] Update field definitions formatting --- docs/test_config_files.md | 46 +++++++++++++++++++-------------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/docs/test_config_files.md b/docs/test_config_files.md index 3770208..e8fadec 100644 --- a/docs/test_config_files.md +++ b/docs/test_config_files.md @@ -35,13 +35,13 @@ The following shows all the possible options a test configuration file could con ## Field definitions test: is a unique numerical value from the other tests. Other than for creation of the yml file, it has no bearing on the test execution. -test_template: Is a pointer to a specific test template that will be brought in. Purpose is that when we need to make changes for a specifc test, we update the test_template file not the test_defs.yml. +**test_template**: Is a pointer to a specific test template that will be brought in. Purpose is that when we need to make changes for a specifc test, we update the test_template file not the test_defs.yml. -test_name: designates the name that zathras knows the test by. This name is what is passed in with the --tests option. +**test_name**: designates the name that zathras knows the test by. This name is what is passed in with the --tests option. -test_description: A brief description of what the test does. This information is displayed when the --show_tests option is provided to burden. +**test_description**: A brief description of what the test does. This information is displayed when the --show_tests option is provided to burden. -repository_type: The type of repository that the test is located in. Valid values include: +**repository_type**: The type of repository that the test is located in. Valid values include: git_tar: Mainly used when pulling an entire git repository for a test. @@ -53,7 +53,7 @@ repository_type: The type of repository that the test is located in. Valid value tar: Indicates we are copying a tarball. -location: Is the location at which the test can be found. Note in the case of git, it will be everything minus the actual location name. For example, if we have: +**location**: Is the location at which the test can be found. Note in the case of git, it will be everything minus the actual location name. For example, if we have: https://github.com/redhat-performance/streams-wrapper/ @@ -63,7 +63,7 @@ We will drop the streams-wrapper on the end and have -version: The version of the test to use. For a listing of versions available, you can issue the following command: +**version**: The version of the test to use. For a listing of versions available, you can issue the following command: $ ./burden --test_versions streams ============================ @@ -73,48 +73,48 @@ version: The version of the test to use. For a listing of versions available, yo -test_grouping : Is the grouping name of the tests. In the case of a git repo, it will be the actual location name. For example, if we have: +**test_grouping**: Is the grouping name of the tests. In the case of a git repo, it will be the actual location name. For example, if we have: https://github.com/redhat-performance/streams-wrapper/ we drop everything but streams-wrapper and use "streams-wrapper" -os_supported: What Operating Systems are supported. rhel, ubuntu, amazon or all. +**os_supported**: What Operating Systems are supported. rhel, ubuntu, amazon or all. -reboot_system: This is for cloud systems, and will reboot the cloud system once it has been created. +**reboot_system**: This is for cloud systems, and will reboot the cloud system once it has been created. -test_run_from: Some tests need to run from the local system, not the cloud system (boot timing test is one example). If local is specified, the test will run locally, if remote is specified the test will run on the test system. +**test_run_from**: Some tests need to run from the local system, not the cloud system (boot timing test is one example). If local is specified, the test will run locally, if remote is specified the test will run on the test system. -repo_file: The archive file to be downloaded. In the case of git repos, it will be .zip +**repo_file**: The archive file to be downloaded. In the case of git repos, it will be .zip -rhel_pkgs: RHEL packages that are required to run the test; “none” if there aren’t any packaging requirements +**rhel_pkgs**: RHEL packages that are required to run the test; “none” if there aren’t any packaging requirements -ubuntu_pkgs: Ubuntu packages that are required to run the test; “none” if there aren’t any packaging requirements. +**ubuntu_pkgs**: Ubuntu packages that are required to run the test; “none” if there aren’t any packaging requirements. -amazon_pkgs: Amazon packages that are required to run the test; “none” if there aren’t any packaging requirements. +**amazon_pkgs**: Amazon packages that are required to run the test; “none” if there aren’t any packaging requirements. -pbench_required: If set to yes, the test requires pbench to be installed to run. Zathras will install pbench on the test system for you. Only supported for RHEL/Fedora. Note: it is the responsibility of the user to make sure the pbench repos are set up properly. +**pbench_required**: If set to yes, the test requires pbench to be installed to run. Zathras will install pbench on the test system for you. Only supported for RHEL/Fedora. Note: it is the responsibility of the user to make sure the pbench repos are set up properly. -pbench_local_result: If set to yes, will tell the various wrapper scripts to consolidate the pbench results into a results tarball in /tmp with the format: +**pbench_local_result**: If set to yes, will tell the various wrapper scripts to consolidate the pbench results into a results tarball in /tmp with the format: results__.tar e.g. results_pbench_linpack_tuned_virtual-guest_sys_file_none.tar -storage_required: If set to yes, Zathras will check to make sure storage has been designated for the system before provisioning it. +**storage_required**: If set to yes, Zathras will check to make sure storage has been designated for the system before provisioning it. -network_required: If set to yes, Zathras will check to make sure a network (other than the defaults) has been designated for the system before provisioning it. +**network_required**: If set to yes, Zathras will check to make sure a network (other than the defaults) has been designated for the system before provisioning it. -java_required: If set to yes, Zathras will make sure that a Java Version has been provided before provisioning any system. +**java_required**: If set to yes, Zathras will make sure that a Java Version has been provided before provisioning any system. -archive_results: Tell zathras that it has test results to retrieve from the test system. +**archive_results**: Tell zathras that it has test results to retrieve from the test system. -test_script_to_run: The script that Zathras is to run. +**test_script_to_run**: The script that Zathras is to run. -test_specific: Test options that are specific to the test. +**test_specific**: Test options that are specific to the test. -upload_extra: If not set to “none”, uploads the each file in the comma separated list to the test system. +**upload_extra**: If not set to “none”, uploads the each file in the comma separated list to the test system. ## Example From ed73e709027533888678f9bd8a80dc75d2c3133b Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Thu, 1 Aug 2024 14:42:43 -0400 Subject: [PATCH 23/27] Update README.md with testing quickstart links --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index c1f16c6..9ca8dfc 100644 --- a/README.md +++ b/README.md @@ -80,6 +80,6 @@ In general, testing with Zathras follows a similar set of steps across modalitie However, there are some aspects that are particular to the testing environment being used. Please refer to the following documentation that matches your use case: -- Testing on bare metal -- Testing on cloud -- Testing on virtualization +- [Testing on bare metal](https://github.com/redhat-performance/zathras/blob/documentation-rewrite/docs/test_config_files.md) +- [Testing on cloud](https://github.com/redhat-performance/zathras/blob/documentation-rewrite/docs/test_config_files.md) +- [Testing on virtualization](https://github.com/redhat-performance/zathras/blob/documentation-rewrite/docs/test_config_files.md) From ce025f365bc1ab252d398ce61bc25bcbc7a003db Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Thu, 1 Aug 2024 14:43:30 -0400 Subject: [PATCH 24/27] Update README.md typo --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 9ca8dfc..eda9fd2 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ - [Requirements](#requirements) - [Installation](#installation) - [Test config files](#test-config-files) - - [Testing Quick Start](#installation-and-quick-start) + - [Testing Quick Starts](#installation-and-quick-start) ## About From 24de68792fbc009f65c0c66c97227963e24ac139 Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Tue, 10 Sep 2024 13:06:08 -0400 Subject: [PATCH 25/27] add cloud testing info --- docs/testing_quickstart.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/testing_quickstart.md b/docs/testing_quickstart.md index b6c6a9b..2ab1c54 100644 --- a/docs/testing_quickstart.md +++ b/docs/testing_quickstart.md @@ -9,6 +9,11 @@ ## Testing on cloud +Currently supported public clouds: +- AWS cloud +- Azure cloud +- GCP cloud + ### Testing example using AWS: We could define a scenario file, "aws_example", as follows: From d50636d4f615e9bdb964add4a23da601e006ea70 Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Tue, 10 Sep 2024 13:12:48 -0400 Subject: [PATCH 26/27] combine documentation in one directory --- {documentation => docs/old_documentation}/README | 0 .../old_documentation}/aws_create_disks | 0 .../old_documentation}/aws_create_disks.pdf | Bin .../old_documentation}/burden_flow.jpg | Bin .../old_documentation}/connect_systems | 0 .../old_documentation}/connect_systems.pdf | Bin .../old_documentation}/example_usage | 0 .../old_documentation}/files_created | 0 {documentation => docs/old_documentation}/flow.jpg | Bin .../old_documentation}/kickoff_flow.jpg | Bin .../old_documentation}/ten_of_us_flow.jpg | Bin .../old_documentation}/ten_of_us_overview | 0 .../old_documentation}/ten_of_us_overview.pdf | Bin .../old_documentation}/test_exec_flow.jpg | Bin .../old_documentation}/zathras_doc.adoc | 0 .../old_documentation}/zathras_flow | 0 .../old_documentation}/zathras_flow.pdf | Bin 17 files changed, 0 insertions(+), 0 deletions(-) rename {documentation => docs/old_documentation}/README (100%) rename {documentation => docs/old_documentation}/aws_create_disks (100%) rename {documentation => docs/old_documentation}/aws_create_disks.pdf (100%) rename {documentation => docs/old_documentation}/burden_flow.jpg (100%) rename {documentation => docs/old_documentation}/connect_systems (100%) rename {documentation => docs/old_documentation}/connect_systems.pdf (100%) rename {documentation => docs/old_documentation}/example_usage (100%) rename {documentation => docs/old_documentation}/files_created (100%) rename {documentation => docs/old_documentation}/flow.jpg (100%) rename {documentation => docs/old_documentation}/kickoff_flow.jpg (100%) rename {documentation => docs/old_documentation}/ten_of_us_flow.jpg (100%) rename {documentation => docs/old_documentation}/ten_of_us_overview (100%) rename {documentation => docs/old_documentation}/ten_of_us_overview.pdf (100%) rename {documentation => docs/old_documentation}/test_exec_flow.jpg (100%) rename {documentation => docs/old_documentation}/zathras_doc.adoc (100%) rename {documentation => docs/old_documentation}/zathras_flow (100%) rename {documentation => docs/old_documentation}/zathras_flow.pdf (100%) diff --git a/documentation/README b/docs/old_documentation/README similarity index 100% rename from documentation/README rename to docs/old_documentation/README diff --git a/documentation/aws_create_disks b/docs/old_documentation/aws_create_disks similarity index 100% rename from documentation/aws_create_disks rename to docs/old_documentation/aws_create_disks diff --git a/documentation/aws_create_disks.pdf b/docs/old_documentation/aws_create_disks.pdf similarity index 100% rename from documentation/aws_create_disks.pdf rename to docs/old_documentation/aws_create_disks.pdf diff --git a/documentation/burden_flow.jpg b/docs/old_documentation/burden_flow.jpg similarity index 100% rename from documentation/burden_flow.jpg rename to docs/old_documentation/burden_flow.jpg diff --git a/documentation/connect_systems b/docs/old_documentation/connect_systems similarity index 100% rename from documentation/connect_systems rename to docs/old_documentation/connect_systems diff --git a/documentation/connect_systems.pdf b/docs/old_documentation/connect_systems.pdf similarity index 100% rename from documentation/connect_systems.pdf rename to docs/old_documentation/connect_systems.pdf diff --git a/documentation/example_usage b/docs/old_documentation/example_usage similarity index 100% rename from documentation/example_usage rename to docs/old_documentation/example_usage diff --git a/documentation/files_created b/docs/old_documentation/files_created similarity index 100% rename from documentation/files_created rename to docs/old_documentation/files_created diff --git a/documentation/flow.jpg b/docs/old_documentation/flow.jpg similarity index 100% rename from documentation/flow.jpg rename to docs/old_documentation/flow.jpg diff --git a/documentation/kickoff_flow.jpg b/docs/old_documentation/kickoff_flow.jpg similarity index 100% rename from documentation/kickoff_flow.jpg rename to docs/old_documentation/kickoff_flow.jpg diff --git a/documentation/ten_of_us_flow.jpg b/docs/old_documentation/ten_of_us_flow.jpg similarity index 100% rename from documentation/ten_of_us_flow.jpg rename to docs/old_documentation/ten_of_us_flow.jpg diff --git a/documentation/ten_of_us_overview b/docs/old_documentation/ten_of_us_overview similarity index 100% rename from documentation/ten_of_us_overview rename to docs/old_documentation/ten_of_us_overview diff --git a/documentation/ten_of_us_overview.pdf b/docs/old_documentation/ten_of_us_overview.pdf similarity index 100% rename from documentation/ten_of_us_overview.pdf rename to docs/old_documentation/ten_of_us_overview.pdf diff --git a/documentation/test_exec_flow.jpg b/docs/old_documentation/test_exec_flow.jpg similarity index 100% rename from documentation/test_exec_flow.jpg rename to docs/old_documentation/test_exec_flow.jpg diff --git a/documentation/zathras_doc.adoc b/docs/old_documentation/zathras_doc.adoc similarity index 100% rename from documentation/zathras_doc.adoc rename to docs/old_documentation/zathras_doc.adoc diff --git a/documentation/zathras_flow b/docs/old_documentation/zathras_flow similarity index 100% rename from documentation/zathras_flow rename to docs/old_documentation/zathras_flow diff --git a/documentation/zathras_flow.pdf b/docs/old_documentation/zathras_flow.pdf similarity index 100% rename from documentation/zathras_flow.pdf rename to docs/old_documentation/zathras_flow.pdf From 2a0e70019b78a82abd7a25347484c3441aafc385 Mon Sep 17 00:00:00 2001 From: Greg Dumas Date: Tue, 10 Sep 2024 13:23:35 -0400 Subject: [PATCH 27/27] move original README to the documentation archive --- README.bak => docs/old_documentation/README.bak | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename README.bak => docs/old_documentation/README.bak (100%) diff --git a/README.bak b/docs/old_documentation/README.bak similarity index 100% rename from README.bak rename to docs/old_documentation/README.bak