Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[e2e case] scale in failed #480

Open
MegaByte875 opened this issue Mar 15, 2024 · 3 comments
Open

[e2e case] scale in failed #480

MegaByte875 opened this issue Mar 15, 2024 · 3 comments
Assignees
@MegaByte875
Labels
affects/none PR/issue: this bug affects none version. severity/none Severity of bug type/bug Type: something is unexpected

Comments

@MegaByte875
Copy link
Contributor

MegaByte875 commented Mar 15, 2024 •
edited
Loading

Please check the FAQ documentation before raising an issue

Describe the bug (required)

status:
  conditions:
  - lastTransitionTime: "2024-03-14T20:35:45Z"
    lastUpdateTime: "2024-03-14T20:35:45Z"
    message: Storaged is not healthy
    reason: StoragedUnhealthy
    status: "False"
    type: Ready
  graphd:
    phase: Running
    version: v3.7.0
    workload:
      availableReplicas: 5
      collisionCount: 0
      currentReplicas: 5
      currentRevision: e2e-nc-1-d25acfc9fb10c7c39dfd60b-graphd-75fb5d7847
      observedGeneration: 3
      readyReplicas: 5
      replicas: 5
      updateRevision: e2e-nc-1-d25acfc9fb10c7c39dfd60b-graphd-75fb5d7847
      updatedReplicas: 5
  metad:
    phase: Running
    version: v3.7.0
    workload:
      availableReplicas: 3
      collisionCount: 0
      currentReplicas: 3
      currentRevision: e2e-nc-1-d25acfc9fb10c7c39dfd60b-metad-987f9cb99
      observedGeneration: 1
      readyReplicas: 3
      replicas: 3
      updateRevision: e2e-nc-1-d25acfc9fb10c7c39dfd60b-metad-987f9cb99
      updatedReplicas: 3
  observedGeneration: 6
  storaged:
    balancedSpaces:
    - 1
    lastBalanceJob:
      jobID: 2
      spaceID: 1
    phase: ScaleIn
    removedSpaces:
    - 1
    version: v3.7.0
    workload:
      availableReplicas: 5
      collisionCount: 0
      currentReplicas: 5
      currentRevision: e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-65546f5855
      observedGeneration: 3
      readyReplicas: 5
      replicas: 5
      updateRevision: e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-65546f5855
      updatedReplicas: 5
  version: 3.7.0-ent
(root@nebula) [e2e_ldbc_snb]> show jobs
+--------+------------------+------------+----------------------------+----------------------------+
| Job Id | Command          | Status     | Start Time                 | Stop Time                  |
+--------+------------------+------------+----------------------------+----------------------------+
| 3      | "LEADER_BALANCE" | "FINISHED" | 2024-03-14T20:36:24.000000 | 2024-03-14T20:36:29.000000 |
| 2      | "DATA_BALANCE"   | "FINISHED" | 2024-03-14T20:36:04.000000 | 2024-03-14T20:36:18.000000 |
+--------+------------------+------------+----------------------------+----------------------------+
Got 2 rows (time spent 7.443ms/8.043798ms)

(root@nebula) [e2e_ldbc_snb]> show spaces;
+----------------+
| Name           |
+----------------+
| "e2e_ldbc_snb" |
+----------------+


(root@nebula) [e2e_ldbc_snb]> show job 2;
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+----------------------------+----------------------------+-------------+
| Job Id(spaceId:partId) | Command(src->dst)                                                                                                                                                                                                                                                                                                | Status      | Start Time                 | Stop Time                  | State       |
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+----------------------------+----------------------------+-------------+
| 2                      | "DATA_BALANCE"                                                                                                                                                                                                                                                                                                   | "FINISHED"  | 2024-03-14T20:36:04.000000 | 2024-03-14T20:36:18.000000 | "SUCCEEDED" |
| "2, 1:1"               | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-0.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-3.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:04.000000 | 2024-03-14T20:36:15.000000 | "SUCCEEDED" |
| "2, 1:1"               | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-1.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-4.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:03.000000 | 2024-03-14T20:36:04.000000 | "SUCCEEDED" |
| "2, 1:2"               | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-0.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-3.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:04.000000 | 2024-03-14T20:36:06.000000 | "SUCCEEDED" |
| "2, 1:2"               | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-1.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-4.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:03.000000 | 2024-03-14T20:36:04.000000 | "SUCCEEDED" |
| "2, 1:3"               | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-0.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-3.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:05.000000 | 2024-03-14T20:36:17.000000 | "SUCCEEDED" |
| "2, 1:3"               | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-1.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-4.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:03.000000 | 2024-03-14T20:36:05.000000 | "SUCCEEDED" |
| "2, 1:4"               | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-0.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-3.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:06.000000 | 2024-03-14T20:36:08.000000 | "SUCCEEDED" |
| "2, 1:4"               | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-1.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-4.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:05.000000 | 2024-03-14T20:36:06.000000 | "SUCCEEDED" |
| "2, 1:5"               | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-0.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-3.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:17.000000 | 2024-03-14T20:36:17.000000 | "SUCCEEDED" |
| "2, 1:5"               | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-1.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-4.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:06.000000 | 2024-03-14T20:36:17.000000 | "SUCCEEDED" |
| "2, 1:6"               | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-0.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-3.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:04.000000 | 2024-03-14T20:36:05.000000 | "SUCCEEDED" |
| "2, 1:6"               | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-1.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-4.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:03.000000 | 2024-03-14T20:36:04.000000 | "SUCCEEDED" |
| "2, 1:7"               | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-0.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-3.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:05.000000 | 2024-03-14T20:36:07.000000 | "SUCCEEDED" |
| "2, 1:7"               | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-1.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-4.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:03.000000 | 2024-03-14T20:36:05.000000 | "SUCCEEDED" |
| "2, 1:8"               | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-0.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-3.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:06.000000 | 2024-03-14T20:36:07.000000 | "SUCCEEDED" |
| "2, 1:8"               | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-1.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-4.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:07.000000 | 2024-03-14T20:36:08.000000 | "SUCCEEDED" |
| "2, 1:9"               | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-0.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-3.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:03.000000 | 2024-03-14T20:36:04.000000 | "SUCCEEDED" |
| "2, 1:9"               | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-1.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-4.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:04.000000 | 2024-03-14T20:36:05.000000 | "SUCCEEDED" |
| "2, 1:10"              | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-0.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-3.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:05.000000 | 2024-03-14T20:36:10.000000 | "SUCCEEDED" |
| "2, 1:10"              | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-1.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-4.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:05.000000 | 2024-03-14T20:36:05.000000 | "SUCCEEDED" |
| "2, 1:11"              | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-0.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-3.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:04.000000 | 2024-03-14T20:36:06.000000 | "SUCCEEDED" |
| "2, 1:11"              | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-1.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-4.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:03.000000 | 2024-03-14T20:36:04.000000 | "SUCCEEDED" |
| "2, 1:12"              | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-0.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-3.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:04.000000 | 2024-03-14T20:36:06.000000 | "SUCCEEDED" |
| "2, 1:12"              | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-1.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-4.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:03.000000 | 2024-03-14T20:36:04.000000 | "SUCCEEDED" |
| "2, 1:13"              | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-0.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-3.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:05.000000 | 2024-03-14T20:36:10.000000 | "SUCCEEDED" |
| "2, 1:13"              | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-1.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-4.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:03.000000 | 2024-03-14T20:36:05.000000 | "SUCCEEDED" |
| "2, 1:14"              | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-0.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-3.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:05.000000 | 2024-03-14T20:36:15.000000 | "SUCCEEDED" |
| "2, 1:14"              | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-1.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-4.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:03.000000 | 2024-03-14T20:36:05.000000 | "SUCCEEDED" |
| "2, 1:15"              | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-0.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-3.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:06.000000 | 2024-03-14T20:36:17.000000 | "SUCCEEDED" |
| "2, 1:15"              | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-1.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779->e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-4.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local:9779" | "SUCCEEDED" | 2024-03-14T20:36:06.000000 | 2024-03-14T20:36:06.000000 | "SUCCEEDED" |
| "Total:30"             | "Succeeded:30"                                                                                                                                                                                                                                                                                                   | "Failed:0"  | "In Progress:0"            | "Invalid:0"                | ""          |
+------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+----------------------------+----------------------------+-------------+
Got 32 rows (time spent 2.673ms/5.486582ms)


(root@nebula) [e2e_ldbc_snb]> show hosts;
+-----------------------------------------------------------------------------------------------------------------------------------------------------+------+----------+--------------+---------------------+------------------------+---------+-------------+
| Host                                                                                                                                                | Port | Status   | Leader count | Leader distribution | Partition distribution | Zone    | Version     |
+-----------------------------------------------------------------------------------------------------------------------------------------------------+------+----------+--------------+---------------------+------------------------+---------+-------------+
| "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-0.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local" | 9779 | "ONLINE" | 6            | "e2e_ldbc_snb:6"    | "e2e_ldbc_snb:15"      | "zone1" | "3.7.0-ent" |
| "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-1.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local" | 9779 | "ONLINE" | 5            | "e2e_ldbc_snb:5"    | "e2e_ldbc_snb:15"      | "zone2" | "3.7.0-ent" |
| "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-2.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local" | 9779 | "ONLINE" | 9            | "e2e_ldbc_snb:9"    | "e2e_ldbc_snb:30"      | "zone3" | "3.7.0-ent" |
| "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-3.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local" | 9779 | "ONLINE" | 5            | "e2e_ldbc_snb:5"    | "e2e_ldbc_snb:15"      | "zone1" | "3.7.0-ent" |
| "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-4.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local" | 9779 | "ONLINE" | 5            | "e2e_ldbc_snb:5"    | "e2e_ldbc_snb:15"      | "zone2" | "3.7.0-ent" |
+-----------------------------------------------------------------------------------------------------------------------------------------------------+------+----------+--------------+---------------------+------------------------+---------+-------------+
Got 5 rows (time spent 2.08ms/3.014541ms)


(root@nebula) [e2e_ldbc_snb]> show zones;
+---------+-----------------------------------------------------------------------------------------------------------------------------------------------------+------+
| Name    | Host                                                                                                                                                | Port |
+---------+-----------------------------------------------------------------------------------------------------------------------------------------------------+------+
| "zone1" | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-0.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local" | 9779 |
| "zone1" | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-3.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local" | 9779 |
| "zone2" | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-1.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local" | 9779 |
| "zone2" | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-4.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local" | 9779 |
| "zone3" | "e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-2.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local" | 9779 |
+---------+-----------------------------------------------------------------------------------------------------------------------------------------------------+------+
Got 5 rows (time spent 1.314ms/2.198187ms)

zone_metad_err_logs.tar.gz

Your Environments (required)

  • OS: uname -a
  • Commit id (e.g. a3ffc7d8)

How To Reproduce(required)

Steps to reproduce the behavior:

  1. Step 1
  2. Step 2
  3. Step 3

Expected behavior

Additional context
@MegaByte875 MegaByte875 added the type/bug Type: something is unexpected label Mar 15, 2024
@MegaByte875 MegaByte875 self-assigned this Mar 15, 2024
@github-actions github-actions bot added affects/none PR/issue: this bug affects none version. severity/none Severity of bug labels Mar 15, 2024
@MegaByte875
Copy link
Contributor Author

MegaByte875 commented Mar 15, 2024 •
edited
Loading

the failed reconcile process in e2e log

I0314 20:36:04.255820       1 meta_client.go:359] balance job running now
E0314 20:36:04.255858       1 data_balancer.go:51] storaged cluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged] balance data in zone error: waiting for balance job 2 finished
I0314 20:36:04.274492       1 nebulacluster.go:128] NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] status updated successfully
E0314 20:36:04.274578       1 storaged_cluster.go:236] scale storaged cluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged] failed: waiting for balance job 2 finished
E0314 20:36:04.274600       1 nebula_cluster_control.go:152] reconcile storaged cluster failed: waiting for balance job 2 finished
I0314 20:36:04.290464       1 nebulacluster.go:128] NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] status updated successfully
I0314 20:36:04.290499       1 nebula_cluster_controller.go:181] NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] reconcile details: waiting for balance job 2 finished
I0314 20:36:04.290504       1 nebula_cluster_controller.go:181] NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] reconcile details: waiting for nebulacluster ready
I0314 20:36:04.290613       1 nebula_cluster_controller.go:154] Finished reconciling NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] (142.998501ms), result: {false 10s}
I0314 20:36:04.290846       1 nebula_cluster_controller.go:171] Start to reconcile NebulaCluster
I0314 20:36:04.372304       1 nebulacluster.go:128] NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] status updated successfully
E0314 20:36:04.444513       1 storaged_cluster.go:236] scale storaged cluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged] failed: balance job still in progress, jobID 2, spaceID 1
E0314 20:36:04.444548       1 nebula_cluster_control.go:152] reconcile storaged cluster failed: balance job still in progress, jobID 2, spaceID 1
I0314 20:36:04.465856       1 nebulacluster.go:128] NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] status updated successfully
I0314 20:36:04.466117       1 nebula_cluster_controller.go:181] NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] reconcile details: balance job still in progress, jobID 2, spaceID 1
I0314 20:36:04.466139       1 nebula_cluster_controller.go:181] NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] reconcile details: waiting for nebulacluster ready
I0314 20:36:04.466144       1 nebula_cluster_controller.go:154] Finished reconciling NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] (175.372422ms), result: {false 10s}
I0314 20:36:14.291178       1 nebula_cluster_controller.go:171] Start to reconcile NebulaCluster
I0314 20:36:14.374694       1 nebulacluster.go:128] NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] status updated successfully
E0314 20:36:14.388530       1 storaged_cluster.go:236] scale storaged cluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged] failed: balance job still in progress, jobID 2, spaceID 1
E0314 20:36:14.388553       1 nebula_cluster_control.go:152] reconcile storaged cluster failed: balance job still in progress, jobID 2, spaceID 1
I0314 20:36:14.404823       1 nebulacluster.go:128] NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] status updated successfully
I0314 20:36:14.404866       1 nebula_cluster_controller.go:181] NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] reconcile details: balance job still in progress, jobID 2, spaceID 1
I0314 20:36:14.404872       1 nebula_cluster_controller.go:181] NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] reconcile details: waiting for nebulacluster ready
I0314 20:36:14.404877       1 nebula_cluster_controller.go:154] Finished reconciling NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] (113.833165ms), result: {false 10s}
I0314 20:36:24.405586       1 nebula_cluster_controller.go:171] Start to reconcile NebulaCluster
I0314 20:36:24.481082       1 nebulacluster.go:128] NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] status updated successfully
I0314 20:36:24.506584       1 meta_client.go:347] space 1 balance leader successfully
I0314 20:36:24.553075       1 nebulacluster.go:128] NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] status updated successfully
I0314 20:36:27.163945       1 nebulacluster.go:128] NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] status updated successfully
I0314 20:36:27.163979       1 nebula_cluster_controller.go:156] Finished reconciling NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b], spendTime: (2.75855131s)
I0314 20:36:27.164187       1 nebula_cluster_controller.go:171] Start to reconcile NebulaCluster
I0314 20:36:27.535952       1 nebulacluster.go:128] NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] status updated successfully
I0314 20:36:31.367354       1 nebulacluster.go:128] NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] status updated successfully
I0314 20:36:31.367473       1 nebula_cluster_controller.go:156] Finished reconciling NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b], spendTime: (4.203337066s)
I0314 20:36:31.367757       1 nebula_cluster_controller.go:171] Start to reconcile NebulaCluster
I0314 20:36:31.695297       1 nebulacluster.go:128] NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] status updated successfully
I0314 20:36:31.756840       1 nebulacluster.go:128] NebulaCluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b] status updated successfully
I0314 20:36:31.756883       1 storaged_scaler.go:188] storaged cluster [e2e-nc-1-64fce459e0fcd07f74cd501/e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged] remove hosts in the space e2e_ldbc_snb successfully
E0314 20:36:31.757877       1 storaged_scaler.go:191] drop hosts [HostAddr({Host:e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-4.e2e-nc-1-d25acfc9fb10c7c39dfd60b-storaged-headless.e2e-nc-1-64fce459e0fcd07f74cd501.svc.cluster.local Port:9779})] failed: metad client response code -2016 name <UNSET>

@MegaByte875
Copy link
Contributor Author

a normal reconcile process

E0315 07:19:08.334272       1 storaged_cluster.go:236] scale storaged cluster [default/nebula-storaged] failed: balance job still in progress, jobID 9, spaceID 1
E0315 07:19:08.334291       1 nebula_cluster_control.go:152] reconcile storaged cluster failed: balance job still in progress, jobID 9, spaceID 1
I0315 07:19:08.348778       1 nebulacluster.go:129] NebulaCluster [default/nebula] status updated successfully
I0315 07:19:08.348802       1 nebula_cluster_controller.go:181] NebulaCluster [default/nebula] reconcile details: balance job still in progress, jobID 9, spaceID 1
I0315 07:19:08.348807       1 nebula_cluster_controller.go:181] NebulaCluster [default/nebula] reconcile details: waiting for nebulacluster ready
I0315 07:19:08.348811       1 nebula_cluster_controller.go:154] Finished reconciling NebulaCluster [default/nebula] (349.725633ms), result: {false 10s}
I0315 07:19:08.348888       1 nebula_cluster_controller.go:171] Start to reconcile NebulaCluster
I0315 07:19:08.680821       1 nebulacluster.go:129] NebulaCluster [default/nebula] status updated successfully
E0315 07:19:08.706306       1 storaged_scaler.go:185] remove hosts [HostAddr({Host:nebula-storaged-4.nebula-storaged-headless.default.svc.cluster.local Port:9779})] failed: balance job still in progress, jobID 9, spaceID 1
E0315 07:19:08.706376       1 storaged_cluster.go:236] scale storaged cluster [default/nebula-storaged] failed: balance job still in progress, jobID 9, spaceID 1
E0315 07:19:08.706392       1 nebula_cluster_control.go:152] reconcile storaged cluster failed: balance job still in progress, jobID 9, spaceID 1
I0315 07:19:08.721868       1 nebulacluster.go:129] NebulaCluster [default/nebula] status updated successfully
I0315 07:19:08.721890       1 nebula_cluster_controller.go:181] NebulaCluster [default/nebula] reconcile details: balance job still in progress, jobID 9, spaceID 1
I0315 07:19:08.721896       1 nebula_cluster_controller.go:181] NebulaCluster [default/nebula] reconcile details: waiting for nebulacluster ready
I0315 07:19:08.721900       1 nebula_cluster_controller.go:154] Finished reconciling NebulaCluster [default/nebula] (373.044954ms), result: {false 10s}
I0315 07:19:08.721965       1 nebula_cluster_controller.go:171] Start to reconcile NebulaCluster
E0315 07:19:09.021422       1 storaged_scaler.go:185] remove hosts [HostAddr({Host:nebula-storaged-4.nebula-storaged-headless.default.svc.cluster.local Port:9779})] failed: balance job still in progress, jobID 9, spaceID 1
E0315 07:19:09.021501       1 storaged_cluster.go:236] scale storaged cluster [default/nebula-storaged] failed: balance job still in progress, jobID 9, spaceID 1
E0315 07:19:09.021514       1 nebula_cluster_control.go:152] reconcile storaged cluster failed: balance job still in progress, jobID 9, spaceID 1
I0315 07:19:09.036105       1 nebulacluster.go:129] NebulaCluster [default/nebula] status updated successfully
I0315 07:19:09.036149       1 nebula_cluster_controller.go:181] NebulaCluster [default/nebula] reconcile details: balance job still in progress, jobID 9, spaceID 1
I0315 07:19:09.036154       1 nebula_cluster_controller.go:181] NebulaCluster [default/nebula] reconcile details: waiting for nebulacluster ready
I0315 07:19:09.036158       1 nebula_cluster_controller.go:154] Finished reconciling NebulaCluster [default/nebula] (314.217905ms), result: {false 10s}
I0315 07:19:15.749622       1 nebula_cluster_controller.go:171] Start to reconcile NebulaCluster
I0315 07:19:15.851777       1 nebulacluster.go:129] NebulaCluster [default/nebula] status updated successfully
I0315 07:19:15.851799       1 storaged_scaler.go:188] storaged cluster [default/nebula-storaged] remove hosts in the space sf1 successfully
I0315 07:19:15.855477       1 storaged_scaler.go:194] storaged cluster [default/nebula-storaged] drop hosts [HostAddr({Host:nebula-storaged-4.nebula-storaged-headless.default.svc.cluster.local Port:9779})] successfully
I0315 07:19:15.859204       1 meta_client.go:347] space 1 balance leader successfully
I0315 07:19:15.875116       1 nebulacluster.go:129] NebulaCluster [default/nebula] status updated successfully
I0315 07:19:15.884828       1 pvc_deleter.go:82] component [default/nebula-storaged] set PVC storaged-data-nebula-storaged-4 annotations succeed
I0315 07:19:15.887436       1 storaged_scaler.go:229] storaged cluster [default/nebula-storaged] all used pvcs were reclaimed
I0315 07:19:15.916846       1 workload.go:122] workload StatefulSet default/nebula-storaged updated successfully
I0315 07:19:16.153313       1 nebulacluster.go:129] NebulaCluster [default/nebula] status updated successfully

@MegaByte875
Copy link
Contributor Author

#481

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MegaByte875 MegaByte875

Labels
affects/none PR/issue: this bug affects none version. severity/none Severity of bug type/bug Type: something is unexpected
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@MegaByte875