Skip to content

Commit

Permalink
feature flags; 'show performance latency'; up cli
Browse files Browse the repository at this point in the history
* revise/improve `ais show performance latency`
* add metrics
* add docs/feature-flags

Signed-off-by: Alex Aizman <[email protected]>
  • Loading branch information
alex-aizman committed Nov 1, 2023
1 parent c3bfd04 commit e9e9cdd
Show file tree
Hide file tree
Showing 6 changed files with 115 additions and 4 deletions.
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@ With a little effort, they all could be extracted and used outside.
- Configuration
- [Configuration](/docs/configuration.md)
- [CLI: `ais config`](/docs/cli/config.md)
- [Feature flags](/docs/feature_flags.md)
- Observability
- [Observability](/docs/metrics.md)
- [Prometheus](/docs/prometheus.md)
Expand All @@ -169,6 +170,7 @@ With a little effort, they all could be extracted and used outside.
- [CLI: `ais job`](/docs/cli/job.md) and [`ais show job`](/docs/cli/show.md)
- Topics
- [System files](/docs/sysfiles.md)
- [Feature flags](/docs/feature_flags.md)
- [`aisnode` command line](/docs/command_line.md)
- [Traffic patterns](/docs/traffic_patterns.md)
- [Highly available control plane](/docs/ha.md)
Expand Down
2 changes: 1 addition & 1 deletion cmd/cli/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ go 1.21

// direct
require (
github.com/NVIDIA/aistore v1.3.21-0.20231101164003-3b6ff26cee2b
github.com/NVIDIA/aistore v1.3.21-0.20231101170837-c3bfd04e196f
github.com/fatih/color v1.15.0
github.com/json-iterator/go v1.1.12
github.com/onsi/ginkgo v1.16.5
Expand Down
4 changes: 2 additions & 2 deletions cmd/cli/go.sum
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
code.cloudfoundry.org/bytefmt v0.0.0-20190710193110-1eb035ffe2b6/go.mod h1:wN/zk7mhREp/oviagqUXY3EwuHhWyOvAdsn5Y4CzOrc=
github.com/BurntSushi/toml v1.3.2/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ=
github.com/NVIDIA/aistore v1.3.21-0.20231101164003-3b6ff26cee2b h1:ukBNwqXrHr08evzaZh14WsBdReFI/JtGPBbggUUqMBk=
github.com/NVIDIA/aistore v1.3.21-0.20231101164003-3b6ff26cee2b/go.mod h1:+iSnZg0ovMaLgaT9fLAs2WmYBP7IfeTW1WYkbKrwP4g=
github.com/NVIDIA/aistore v1.3.21-0.20231101170837-c3bfd04e196f h1:0jqWqTQv0WnI448eT1Qlcw3sErlssJev9KnMpl8lRSc=
github.com/NVIDIA/aistore v1.3.21-0.20231101170837-c3bfd04e196f/go.mod h1:+iSnZg0ovMaLgaT9fLAs2WmYBP7IfeTW1WYkbKrwP4g=
github.com/OneOfOne/xxhash v1.2.8 h1:31czK/TI9sNkxIKfaUfGlU47BAxQ0ztGgd9vPyqimf8=
github.com/OneOfOne/xxhash v1.2.8/go.mod h1:eZbhyaAYD41SGSSsnmcpxVoRiQ/MPUTjUdIIOT9Um7Q=
github.com/VividCortex/ewma v1.1.1/go.mod h1:2Tkkvm3sRDVXaiyucHiACn4cqf7DpdyLvmxzcbUokwA=
Expand Down
1 change: 0 additions & 1 deletion cmn/feat/feat.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ const (

var All = []string{
"Enforce-IntraCluster-Access",
"Do-not-HEAD-Remote-Bucket",
"Skip-Loading-VersionChecksum-MD",
"Do-not-Auto-Detect-FileShare",
"Provide-S3-API-via-Root",
Expand Down
29 changes: 29 additions & 0 deletions docs/cli/performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,35 @@ $ ais performance <TAB-TAB>
counters throughput latency capacity disk
```

## `ais show performance latency`

Example usage:

```console
$ ais show performance latency --refresh 10

latency ------------------ 13:03:58.329680
TARGET GET-COLD(n) GET-COLD-RW(t) GET-COLD(total/avg size) GET(n) GET(t) GET-REDIR(t) GET(total/avg size)
t[EkMt8081] 151 2.01s 145.00MiB 983.31KiB 154 2.13s 1.156551ms 154.00MiB 1.00MiB

latency ------------------ 13:04:08.335764
TARGET GET-COLD(n) GET-COLD-RW(t) GET-COLD(total/avg size) GET(n) GET(t) GET-REDIR(t) GET(total/avg size)
t[EkMt8081] 189 2.04s 181.00MiB 980.66KiB 190 1.86s 892.015µs 190.00MiB 1.00MiB
```

Notice naming conventions:

* (n) - counter (total number of operations of a given kind)
* (t) - time (latency of the operation)

Other notable semantics includes:

| metric | comment |
| ------ | ------- |
| `GET-COLD-RW(t)` | denotes (remote read, local write) latency, which is a _part_ of the total latency _not_ including the time it takes to transmit requested payload to user |
| `GET(t)` | GET latency (for cold GETs includes the above) |
| `GET-REDIR(t)` | time that passes between ais gateway _redirecting_ GET operation to specific target, and this target _starting_ to handle the request |

## `ais show performance counters`

```console
Expand Down
81 changes: 81 additions & 0 deletions docs/feature_flags.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
---
layout: post
title: FEATURE FLAGS
permalink: /docs/feature-flags
redirect_from:
- /feature_flags.md/
- /docs/feature_flags.md/
---

## Feature Flags

`Feature flags` is a 64-bit (bit-wise) field in aistore cluster configuration denoting assorted (named) capabilities that can be individually enabled at runtime.

The features themselves are enumerated below. By default, all features are disabled, and the corresponding 64-bit field is set to zero.

## How to see, how to view, how to change

```console
$ ais config cluster features <TAB-TAB>

Enforce-IntraCluster-Access Provide-S3-API-via-Root Dont-Allow-Passing-FQN-to-ETL none
Do-not-HEAD-Remote-Bucket Fsync-PUT Ignore-LimitedCoexistence-Conflicts
Skip-Loading-VersionChecksum-MD LZ4-Block-1MB Dont-Rm-via-Validate-Warm-GET
Do-not-Auto-Detect-FileShare LZ4-Frame-Checksum Disable-Fast-Cold-GET
```

For example:

```console
$ ais config cluster features Provide-S3-API-via-Root Dont-Rm-via-Validate-Warm-GET Ignore-LimitedCoexistence-Conflicts
PROPERTY VALUE
features Provide-S3-API-via-Root,Ignore-LimitedCoexistence-Conflicts,Dont-Rm-via-Validate-Warm-GET

Cluster config updated
```

> **Tip**: to select multiple features, type the first letter (or the few first letters) of the one you select, press `<TAB-TAB>` to complete, and then press `<TAB-TAB>` again to keep going, if need be.
To view the current (configured) setting, type the same command and hit `Enter`:

```console
$ ais config cluster features
PROPERTY VALUE
features Provide-S3-API-via-Root,Ignore-LimitedCoexistence-Conflicts,Dont-Rm-via-Validate-Warm-GET
```

The same in JSON:

```console
$ ais config cluster features --json
{
"features": "1552"
}
```

Finally, to reset feature flags back to the system defaults, `<TAB-TAB>` to select `none` from the completion list, or simply run:

```console
$ ais config cluster features none

PROPERTY VALUE
features none

Cluster config updated
```

## Names and comments

| name | comment |
| ---- | ------- |
| `Enforce-IntraCluster-Access` | when enabled, aistore targets will make sure _not_ to execute direct (ie., not redirected) API calls |
| `Provide-S3-API-via-Root` | handle S3 requests via `aistore-hostname/` (whereby the default: `aistore-hostname/s3`) |
| `Dont-Allow-Passing-FQN-to-ETL` | do not allow passing fully-qualified name of a locally stored object to (local) ETL containers |
| `Fsync-PUT` | PUT and cold-GET: commit (or sync) the object payload to stable storage |
| `Ignore-LimitedCoexistence-Conflicts` | run in presence of "limited coexistence" type conflicts |
| `Skip-Loading-VersionChecksum-MD` | skip loading existing object's metadata, Version and Checksum (VC) in particular |
| `LZ4-Block-1MB` | .tar.lz4 format, lz4 compression: maximum uncompressed block size=1MB (default: 256K) |
| `LZ4-Frame-Checksum` | checksum lz4 frames |
| `Dont-Rm-via-Validate-Warm-GET` | when version validation (`versioning.validate_warm_get`) is enabled GET(object) shall _not_ delete the object if its remote source doesn't exist |
| `Do-not-Auto-Detect-FileShare` | do not auto-detect file share (NFS, SMB) when _promoting_ shared files to AIS |
| `Disable-Fast-Cold-GET` | use regular datapath to execute cold-GET operations |

0 comments on commit e9e9cdd

Please sign in to comment.