DAOS-16111 rebuild: enhance leader update_and_warn_for_slow_engines() #15778
+255
−135
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Ticket title is 'rebuild enhancement: uniform identifier in log messages' |
kccain
force-pushed
the
kccain/daos_16111_more_ps_rb_warn
branch
from
e271010 to
6a8ff39
Compare
With this change, the existing warning logic invoked by the PS leader engine during rebuild_leader_status_check() is enhanced to check for "almost done" and likely hung rebuilds. When 95% or more engines have completed their scan or pull phase, this logic will set a deadline of 2 minutes. If the relevant rebuild phase has not finished by the deadline, warning messages will be logged to indicate a potentially stuck rebuild, including a list of engines that have not completed. The determination of "almost done" is such that, depending on the scale of the system, the number of remaining engines that are being waited for is in a reasonable range of 1-20 engines. The daos_test -r rebuild tests include a new rebuild_long_scan_hang() to inject a single-engine scan hang with a > 2 minute delay, to exercise the new warning logic. Features: rebuild Signed-off-by: Kenneth Cain <[email protected]>
kccain
force-pushed
the
kccain/daos_16111_more_ps_rb_warn
branch
from
6a8ff39 to
fce3be5
Compare
|
thanks for the enhancement, basically looks fine to me. a small point is in rebuild_iv_ent_update() can add an INFO or even ERR log if one engine reports failure "src_iv->riv_status != 0". |
liuxuezhao
reviewed
Jan 24, 2025
| {"REBUILD35: destroy container then reintegrate", rebuild_cont_destroy_and_reintegrate, | ||
| rebuild_sub_6nodes_rf1_setup, rebuild_sub_teardown}, | ||
| {"REBUILD36: single engine scan lengthy hang", rebuild_long_scan_hang, rebuild_sub_setup, | ||
| rebuild_sub_teardown}, |
There was a problem hiding this comment.
i added another UT for incr reint, may need rebase later if my PR land first.
There was a problem hiding this comment.
what PR is it that added a test to this file? Latest master shows REBUILD35 is the last test in the list.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With this change, the existing warning logic invoked by the PS leader engine during rebuild_leader_status_check() is enhanced to check for "almost done" and likely hung rebuilds. When 95% or more engines have completed their scan or pull phase, this logic will set a deadline of 2 minutes. If the relevant rebuild phase has not finished by the deadline, warning messages will be logged to indicate a potentially stuck rebuild, including a list of engines that have not completed.
The determination of "almost done" is such that, depending on the scale of the system, the number of remaining engines that are being waited for is in a reasonable range of 1-20 engines.
The daos_test -r rebuild tests include a new rebuild_long_scan_hang() to inject a single-engine scan hang with a > 2 minute delay, to exercise the new warning logic.
Features: rebuild
Before requesting gatekeeper:
Features:(orTest-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.Gatekeeper: