Benchmarking
Due to Skytable having in-house implementations of almost everything, starting from the protocol, the storage engine and query
-language — we have our own custom load testing tool called sky-bench that is distributed with the bundle.
Setting up for benchmarking
-Quick notes:
+Benchmarking
All of Skytable's components are developed from scratch in-house, which makes it hard (if not impossible) to utilize other benchmarking suites. This is why all of Skytable's release bundles come with the sky-bench benchmarking tool. It's important to understand how the tool works to understand what you're benchmarking and what the results mean.
The overall goal with the benchmark tool is to simulate how the database would perform in the real-world and reduce synthetic factors.
+The ability to simulate different workloads is currently being worked on. This means that across releases, the default workload that the benchmark tool runs may vary. See the release notes to see if the benchmark workload has changed.
Benchmark workload
+The workload that the engine currently uses (as of v0.8.1) is the following:
-
-
- The benchmark tool will create: +
- A model
bench.benchis created with a primary key (of typestringand with a column of typeuint8)
+ - Multiple clients are created (simulating "application servers") +
- Queries are run against unique rows. This means that unlike
redis-benchmarkevery query group touches a different row, as it would generally do in the real-world
+ - The following queries (collectively a "query group") are run for each unique row:
-
-
- a space called
bench
- - a model called
bench
+ - The row is first created with an
INSERT
+ - All columns of the row are returned with a
SELECT
+ - The integer column is incremented with an
UPDATE
+ - The row is finally removed with a
DELETE
- - a space called
- Be sure that these objects don't already exist! But don't worry, if they do — the benchmark tool will error. You won't lose any data -
- Once the benchmark is complete, the
benchspace is removed!
- - Do not use the
benchspace during the benchmark: You might lose data
+ - By default, 1,000,000 rows are created and manipulated +
- The time taken for each row to be sent, read back and decoded into a readable form is recorded (time taken to parse into actual language structures such as maps and lists) towards the total time taken, once again unlike many other benchmark tools +
- In total 4,000,000 queries are run (by default)
The benchmark tool will:
--
-
- Run a total of 4,000,000 queries (don't worry, they run pretty fast!):
-
-
-
- Run 1,000,000
INSERTs
- - Run 1,000,000
SELECTs
- - Run 1,000,000
UPDATEs
- - Run 1,000,000
DELETEs
-
- - Run 1,000,000
- The model used has the declaration
(un: string, pw: uint8)
- - The
SELECTwill select and return all fields
- - The
UPDATEwill increment the value ofpwlike thispw += 1
- - The
DELETEremoves the entire collection
- - The default primary key size is 7 bytes. All generated keys will be padded with zeros from to 0 to 999,999 like this:
0000000or0999999
-
Off to the races
+The benchmark tool will create a space bench and a model bench and will completely remove the space and associated data once the benchmark is complete. Do not use this space!
Running the benchmark
+Now that you know how the benchmark engine works, go ahead and benchmark for yourself.
-
-
- Start up the database server -
- Run
./sky-bench --password <your root password>. We need yourrootpassword because only the root account can create, alter and drop models and the benchmark tool needs to run these queries
- - Wait for it to happen. You may not believe your eyes, so we recommend that you keep your eyes hydrated 🔥🚀✨💣 -
sky-bench --password <root_password>
--password argument if you have already set it using the SKYDB_PASSWORD environment variableYou can tune the number of threads, connections, rows created and such to your liking to simulate the environment that you think matches your production setting.