Enable reorder read sequence for bk client by default #4139
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hangc0276
requested review from
merlimat,
jiazhai,
eolivelli,
zymap,
wenbingshen and
StevenLuMT
zymap
reviewed
Dec 1, 2023
| @@ -1057,6 +1057,9 @@ zkEnableSecurity=false | |||
| # Enable/disable having read operations for a ledger to be sticky to a single bookie. | |||
| stickyReadSEnabled=true | |||
|
|
|||
| # Enable/disable reordering read sequence on reading entries. | |||
| reorderReadSequenceEnabled=true | |||
There was a problem hiding this comment.
Should we change the default value in the clientconfiguration.java? I think it should be applied to all the clients, right? It will avoid reading a lost bookie as much as possible
There was a problem hiding this comment.
Good point. We can change the default value in the next major release.
wenbingshen
approved these changes
Dec 5, 2023
hangc0276
force-pushed
the
chenhang/enable_reorder_read_sequence_for_bk_client_by_default
branch
from
ac4777c to
0668143
Compare
merlimat
approved these changes
Dec 7, 2023
|
rerun failure checks |
1 similar comment
|
rerun failure checks |
zymap
approved these changes
Dec 12, 2023
merlimat
approved these changes
Jan 5, 2024
Ghatage
pushed a commit
to sijie/bookkeeper
that referenced
this pull request
Jul 12, 2024
### Motivation <!-- Explain here the context, and why you're making that change. What is the problem you're trying to solve. --> If one ledger's ensemble is [bk0, bk1] and bk0 is down, the bookie client may send a read request to bk0 first then fail with the following errors, and resend the read request to bk1 in the end. ``` 2023-10-19T18:33:52,042 - ERROR - [BookKeeperClientWorker-OrderedExecutor-3-0:PerChannelBookieClient@563] - Cannot connect to 192.168.31.216:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId 192.168.31.216:3181, bookie does not exist or it is not running 2023-10-19T18:33:52,042 - INFO - [BookKeeperClientWorker-OrderedExecutor-3-0:DefaultBookieAddressResolver@77] - Cannot resolve 192.168.31.216:3181, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available 2023-10-19T18:33:52,042 - INFO - [BookKeeperClientWorker-OrderedExecutor-3-0:PendingReadOp$LedgerEntryRequest@223] - Error: Bookie handle is not available while reading L6 E40 from bookie: 192.168.31.216:3181 ``` One of the related issues is in the auto-recovery decommission and there is one PR in the BookKeeper repo: apache#4113 However, the bookie client already knows the bk0 is down and we should send the read request to bk1 first. So we can reorder the read request based on the known bookie list. If one bookie is lost, it will reorder the lost bookie to the end of the read list. ### Modifications <!-- Describe the modifications you've done. --> Enable the `reorderReadSequence` by default for auto-recovery.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
If one ledger's ensemble is [bk0, bk1] and bk0 is down, the bookie client may send a read request to bk0 first then fail with the following errors, and resend the read request to bk1 in the end.
One of the related issues is in the auto-recovery decommission and there is one PR in the BookKeeper repo: #4113
However, the bookie client already knows the bk0 is down and we should send the read request to bk1 first. So we can reorder the read request based on the known bookie list. If one bookie is lost, it will reorder the lost bookie to the end of the read list.
Modifications
Enable the
reorderReadSequenceby default for auto-recovery.