The throughput is lower (worse) when run the test with 3.17.0 cassandra-stress version compared to 3.13.0 version

[juliayakovlev]

Before start using 3.17.0 cassandra-stress version in performance test, we ran the throughput/latency tests (mixed and read load) the few times and compared its results with results of the same test that ran with same Scylla version but 3.13.0 c-s version

Scylla version 2024.3.0~dev-20241106
Every test ran 3 untrottled cycles on the same cluster

Tablets disabled

1. Max throughput in mixed load:

c-s version	cycle 1	cycle 2	cycle 3	link
3.13.0	561599	561712	557011	https://argus.scylladb.com/tests/scylla-cluster-tests/516bfdc7-215b-4bdd-bd1c-be795c389134
3.17.0	533610	550041	553213	https://argus.scylladb.com/tests/scylla-cluster-tests/f667b4e3-bb37-43cf-aa40-edb6930480aa

2. Max throughput in read load:

c-s version	cycle 1	cycle 2	cycle 3	link
3.13.0	899256	885138	881635	https://argus.scylladb.com/tests/scylla-cluster-tests/0d7b9ac8-c847-456a-ae47-1e8e312cdac8
3.17.0	873856	854535	875792	https://argus.scylladb.com/tests/scylla-cluster-tests/f1e28be1-e166-423e-bde5-86852f5a3f02

Tablets enabled

1. Max throughput in mixed load:

c-s version	cycle 1	cycle 2	cycle 3	link
3.13.0	543966	542706	543258	https://argus.scylladb.com/tests/scylla-cluster-tests/aacd976a-21b8-4c95-a00c-cdeea51d7ba8
3.17.0	525962	523390	525102	https://argus.scylladb.com/tests/scylla-cluster-tests/b449b191-8816-445f-8f48-86ff6451835f

2. Max throughput in read load:

c-s version	cycle 1	cycle 2	cycle 3	link
3.13.0	780703	786261	782665	https://argus.scylladb.com/tests/scylla-cluster-tests/076805d0-3cd8-4b12-8697-49599cafc43d
3.17.0	644307	639954	640790	https://argus.scylladb.com/tests/scylla-cluster-tests/54f0e5d0-af63-46cf-ba97-4b71057a1f16

Conclusion:
throughput is lower (worse) when run the test with 3.17.0 cassandra-stress version

[fruch]

those are the changes between the versions:
v3.13.0...v3.17.0

I'll guess this is one of the following:

• c09fd08
• f87206a

[dkropachev]

@juliayakovlev , @fruch , is there a way to see CPU and memory load on loaders ?

[fruch]

@dkropachev @Bouncheck

do we expect ReplicaOrdering.RANDOM to affect results like that ?

[fruch]

@juliayakovlev , @fruch , is there a way to see CPU and memory load on loaders ?

in the os metrics dashboard, filter them by ip address (cross change with argus resources tab)

example from the last test from the table above:

[dkropachev]

@dkropachev @Bouncheck

do we expect ReplicaOrdering.RANDOM to affect results like that ?

From what I see loaders fail to generate requests with same speed as before, so I would say it could be, but only if it starved of CPU.

[fruch]

@roydahan what are we gonna do with this one ?

[dkropachev]

@roydahan what are we gonna do with this one ?

I am going to look at it soon

[roydahan]

@dkropachev as we discussed, let's try to look into it.