Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

after restart storage cluster, why we balance leader for many times #424

Closed
jinyingsunny opened this issue Jan 24, 2024 · 2 comments
Closed
Assignees
Labels
affects/master PR/issue: this bug affects master version. process/done Process of bug severity/minor Severity of bug type/bug Type: something is unexpected

Comments

@jinyingsunny
Copy link

the follow picture show ,after restart 3 storaged-cluster, during 10 seconds, balance leader execute for 9 times.
image

operator log:
image

触发场景:

metad、graphd、storage三个组件 一起滚动,提交的顺序是:storaged->graphd->metad。实际看现象是:meta先执行(meta2-meta1-meta0),然后执行storage(storage2-storage1-storage0),最后执行graph(graphd2->graphd1->graphd0)

root@k8s-master:/home/sunny.liu/k8s_file# kubectl -n nebula annotate sts nebula2-storaged nebula-graph.io/restart-timestamp="1706087297" --overwrite
statefulset.apps/nebula2-storaged annotate

root@k8s-master:/home/sunny.liu/k8s_file# kubectl -n nebula annotate sts nebula2-graphd nebula-graph.io/restart-timestamp="1706087297" --overwrite
statefulset.apps/nebula2-graphd annotate

root@k8s-master:/home/sunny.liu/k8s_file# kubectl -n nebula annotate sts nebula2-metad nebula-graph.io/restart-timestamp="1706087297" --overwrite
statefulset.apps/nebula2-metad annotate 

只有第一笔是storaged没ready,后面就都说是graphd 没ready了
image

another question:when restart storage cluster, why we balance leader twice?
image

Your Environments (required)

operator:snap-1.30

Expected behavior

execute balance leader once

@jinyingsunny jinyingsunny added type/bug Type: something is unexpected affects/master PR/issue: this bug affects master version. severity/minor Severity of bug labels Jan 24, 2024
@jinyingsunny jinyingsunny changed the title why we balance leader for many times after restart storage cluster, why we balance leader for many times Jan 24, 2024
@MegaByte875 MegaByte875 self-assigned this Feb 4, 2024
@MegaByte875
Copy link
Contributor

#435

@jinyingsunny
Copy link
Author

用operator:snap-1.35 check,已经不再做多次。

image
相应的日志:
image

@github-actions github-actions bot added the process/fixed Process of bug label Feb 20, 2024
@jinyingsunny jinyingsunny added process/done Process of bug and removed process/fixed Process of bug labels Feb 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects/master PR/issue: this bug affects master version. process/done Process of bug severity/minor Severity of bug type/bug Type: something is unexpected
Projects
None yet
Development

No branches or pull requests

2 participants