Skip to content
This repository has been archived by the owner on Mar 28, 2020. It is now read-only.

Commit

Permalink
tuning the liveness check time of etcd pod
Browse files Browse the repository at this point in the history
liveness probe needs to take care of the following steps' time:
* time to transmit a 4GB snapshot from leader to member -- 4000MB / 50MB/s = 80s
* time to load snapshot —— 4000MB / 200MB/s = 20s
* time to set up streams and follow up leader revisions -- 100s * 10 heartbeat/s * 1KB/heartbeat / 1024KB/MB / 50MB/s ~= 20s

total time -- 120s
  • Loading branch information
armstrongli authored and Li, Jianqiang committed Aug 5, 2019
1 parent 8347d27 commit 9e417e5
Showing 1 changed file with 6 additions and 1 deletion.
7 changes: 6 additions & 1 deletion pkg/util/k8sutil/pod_util.go
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,11 @@ func containerWithRequirements(c v1.Container, r v1.ResourceRequirements) v1.Con
return c
}

// liveness probe needs to take care of the following steps' time:
// * time to transmit a 4GB snapshot from leader to member -- 4000MB / 50MB/s = 80s
// * time to load snapshot —— 4000MB / 200MB/s = 20s
// * time to set up streams and follow up leader revisions -- 100s * 10 heartbeat/s * 1KB/heartbeat / 1024KB/MB / 50MB/s ~= 20s
// total time -- 120s
func newEtcdProbe(isSecure bool) *v1.Probe {
// etcd pod is healthy only if it can participate in consensus
cmd := "ETCDCTL_API=3 etcdctl endpoint status"
Expand All @@ -81,7 +86,7 @@ func newEtcdProbe(isSecure bool) *v1.Probe {
Command: []string{"/bin/sh", "-ec", cmd},
},
},
InitialDelaySeconds: 10,
InitialDelaySeconds: 120,
TimeoutSeconds: 10,
PeriodSeconds: 60,
FailureThreshold: 3,
Expand Down

0 comments on commit 9e417e5

Please sign in to comment.