4.0: use Ra checkpoints in rabbit_fifo for sub-linear time recovery of QQs on boot #10487
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
michaelklishin
changed the title
michaelklishin
changed the title
the-mikedavis
force-pushed
the
md-ra-checkpoints
branch
from
670c64c to
300ff68
Compare
the-mikedavis
changed the title
the-mikedavis
force-pushed
the
md-ra-checkpoints
branch
from
4e2b1a7 to
df6c9d7
Compare
the-mikedavis
force-pushed
the
md-ra-checkpoints
branch
3 times, most recently
from
6ef156b to
309600f
Compare
kjnilsson
force-pushed
the
qq-v4
branch
5 times, most recently
from
75291c7 to
bb89f2d
Compare
the-mikedavis
force-pushed
the
md-ra-checkpoints
branch
from
309600f to
17d7709
Compare
kjnilsson
force-pushed
the
qq-v4
branch
3 times, most recently
from
a479434 to
b6d9b85
Compare
kjnilsson
force-pushed
the
qq-v4
branch
3 times, most recently
from
e5db089 to
4bd8c1f
Compare
Create the new version but not including any changes yet. fix QQ: force delete followers after leader has terminated. Also try a longer sleep for mqtt_shared_SUITE so that the delete operation stands a chance to time out and move on to the forced deletion stage. In some mixed machine version scenarios some followers will never apply the poison pill command so we may as well force delete them just in case. QQ: skip test in amqp_client that cannot pass with mixed machine versions QQ: remove dead code Code relating to prior machine versions and state conversions. formatting / readability rabbit_fifo_prop_SUITE fixes
Also update rabbit_fifo_* suites to test more relevant code versions where applicable. add ff mock QQ: always use the updated credit mode format QQv4: use more compact consumer reference in settle, credit, return This introudces a new type: consumer_key() which is either the consumer_id or the raft index the checkout was processed at. If the consumer is using one of the updated credit spec formats rabbit_fifo will use the raft index as the primary key for the consumer such that the rabbit fifo client can then use the more space efficient integer index instead of the full consumer id in subsequent commands. There is compatibility code to still accept the consumer id in settle, return, discard and credit commands but this is slighlyt slower and of course less space efficient. The old form will be used in cases where the fifo client may have already remove the local consumer state (as happens after a cancel). Lots of test refactorings of the rabbit_fifo_SUITE to begin to use the new forms.
the-mikedavis
force-pushed
the
md-ra-checkpoints
branch
from
17d7709 to
70c13a5
Compare
the-mikedavis
force-pushed
the
md-ra-checkpoints
branch
from
70c13a5 to
34e0c5b
Compare
kjnilsson
force-pushed
the
qq-v4
branch
4 times, most recently
from
6be53b0 to
1bc6b22
Compare
kjnilsson
force-pushed
the
qq-v4
branch
3 times, most recently
from
6b296a1 to
93dfb0f
Compare
kjnilsson
force-pushed
the
qq-v4
branch
5 times, most recently
from
7753e09 to
84b15d3
Compare
kjnilsson
force-pushed
the
qq-v4
branch
2 times, most recently
from
5928e6c to
b487f7e
Compare
kjnilsson
force-pushed
the
qq-v4
branch
3 times, most recently
from
90d8cfa to
4fc8565
Compare
ansd
force-pushed
the
qq-v4
branch
2 times, most recently
from
5f1122e to
95d1994
Compare
|
This has been folded into #10637 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the companion PR for rabbitmq/ra#415
rabbit_fifocurrently has an ad-hoc checkpointing system where it saves{release_cursor, RaftIdx, State}effects in-memory periodically and emits them as the release cursor moves up. By building checkpointing intorawe can save the checkpoints on disk instead, reducing QQ memory footprint and enabling us to recover even very long queues in logarithmic time (w.r.t. the length of the queue). See rabbitmq/ra#415 for in-depth details about checkpoints.Connects #8261