Initial implementation for the Reconcile Logic #17
Conversation
Signed-off-by: Gaurav Dasson <[email protected]>
|
@gdasson Thanks for the PR. Could you please resolve the comments and ensure the PR is in the best shape you think before marking it as ready for review and also remove "draft" from the title? I will take a second round of review once you mark it as ready for review. |
Signed-off-by: Gaurav Dasson <[email protected]>
gdasson
changed the title
Signed-off-by: Gaurav Dasson <[email protected]>
Signed-off-by: Gaurav Dasson <[email protected]>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: gdasson The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
|
||
| // Check if the size of the stateful set is less than expected size | ||
| // Or if there is a pending learner to be promoted | ||
| if *sts.Spec.Replicas < int32(etcdCluster.Spec.Size) && memberCnt > 0 { |
There was a problem hiding this comment.
| if *sts.Spec.Replicas < int32(etcdCluster.Spec.Size) && memberCnt > 0 { | |
| if *sts.Spec.Replicas < int32(etcdCluster.Spec.Size) { |
We should guarantee the following two conditions before performing the scale in & out:
sts.Spec.Replica == memberCnt (returned from etcd cluster)is alwaystrue. refer to https://github.com/ahrtr/etcd-operator/blob/f3b16f5e0b8e2751a1ddc35e0c6a8ccaf4afe568/controller.go#L302-L315sts.Spec.Replica >= 1is always true
There was a problem hiding this comment.
@ahrtr : I am a little confused on this one. In the reference code you have given the inverse condition i.e replica != memberCnt but in the comment you are suggesting for it to be true? I am also thinking the pros/cons for the validation of sts.Spec.Replica == memberCnt before statefulset scale in/out. What additional assurance would this check provide that the reconcile logic is not currently guaranteeing?
There was a problem hiding this comment.
Normally sts.Spec.Replica == memberCnt (returned from etcd cluster) should be always true after etcd-operator finishes each round of Reconcile.
If it isn't true (not equal), then it means there are something wrong happened in previous Reconcile round, possible reasons:
- etcd-operator crashes right after we add a new member (call MemberAdd) and before reconcile the statefulSet (update the replica). In this case,
sts.Spec.Replica < memberCntis true. - etcd-operator crashes right after we delete a member (call MemberDelete) and before reconcile the statefulSet (update the replica). In this case,
sts.Spec.Replica > memberCntis true.
We should always fix any problems coming from previous reconcile rounds before we do something new in current reconcile round. So once we get to current round's scale in & out steps, we can assume that sts.Spec.Replica == memberCnt (returned from etcd cluster) is true.
You can add a TODO item for now and do it in a followup PR, just similar to what I did in https://github.com/ahrtr/etcd-operator/blob/f3b16f5e0b8e2751a1ddc35e0c6a8ccaf4afe568/controller.go#L302-L315.
|
@hakman @justinsb @jmhbnz @ArkaSaha30 Please take a look at this PR. It will be the base for the following PRs. Thanks. cc @ivanvc Also I am most concerned about the e2e test as mentioned in #17 (comment) |
Signed-off-by: Gaurav Dasson <[email protected]>
|
@gdasson please rebase this PR, thx |
| } | ||
|
|
||
| if etcdCluster.Spec.Size == 0 { | ||
| logger.Info("EtcdCluster size is 0..Skipping next steps") |
There was a problem hiding this comment.
nit, a warning message might a little better. It is unusual to create a EtcdCluster with 0 member.
|
@gdasson please rebase this PR and let's merge this PR. Also squash the commits (I can also do it for you if you have any difficulties). I will create more following tasks. |
This PR is an intial implementation of the design mentioned here
The code is now in working state and able to create an etcd cluster. However, the code will continue to be refined and made more production ready as we review and incorporate feedback.