refactor(s3stream/object-wal): complete appends sequentially #2426
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Should add unit tests to cover this refactor, and consider creating subsequent issues and PRs to simplify the upper-level logic. |
Chillax-0v0
force-pushed
the
refactor-object-wal-sequentially-succeed
branch
from
876bf28 to
fe5e8d0
Compare
Signed-off-by: Ning Yu <ningyu@automq.com>
Signed-off-by: Ning Yu <ningyu@automq.com>
…very Signed-off-by: Ning Yu <ningyu@automq.com>
Signed-off-by: Ning Yu <ningyu@automq.com>
Signed-off-by: Ning Yu <ningyu@automq.com>
Signed-off-by: Ning Yu <ningyu@automq.com>
Signed-off-by: Ning Yu <ningyu@automq.com>
Signed-off-by: Ning Yu <ningyu@automq.com>
Signed-off-by: Ning Yu <ningyu@automq.com>
Signed-off-by: Ning Yu <ningyu@automq.com>
Signed-off-by: Ning Yu <ningyu@automq.com>
Signed-off-by: Ning Yu <ningyu@automq.com>
Signed-off-by: Ning Yu <ningyu@automq.com>
Signed-off-by: Ning Yu <ningyu@automq.com>
Chillax-0v0
force-pushed
the
refactor-object-wal-sequentially-succeed
branch
from
fe5e8d0 to
93c5cb1
Compare
Signed-off-by: Ning Yu <ningyu@automq.com>
superhx
approved these changes
May 14, 2025
Chillax-0v0
added a commit
that referenced
this pull request
May 14, 2025
* chore: add todos Signed-off-by: Ning Yu <ningyu@automq.com> * refactor(s3stream/object-wal): sequentially succeed Signed-off-by: Ning Yu <ningyu@automq.com> * refactor(s3stream/object-wal): drop discontinuous objects during recovery Signed-off-by: Ning Yu <ningyu@automq.com> * test: introduce `MockObjectStorage` Signed-off-by: Ning Yu <ningyu@automq.com> * test: test sequentially succeed Signed-off-by: Ning Yu <ningyu@automq.com> * refactor: record endOffset in the object path Signed-off-by: Ning Yu <ningyu@automq.com> * feat: different version of wal object header Signed-off-by: Ning Yu <ningyu@automq.com> * refactor: adapt to the new object header format Signed-off-by: Ning Yu <ningyu@automq.com> * feat: recover from the trim offset Signed-off-by: Ning Yu <ningyu@automq.com> * test: test recover continuous objects from trim offset Signed-off-by: Ning Yu <ningyu@automq.com> * test: test marshal and unmarshal wal object header Signed-off-by: Ning Yu <ningyu@automq.com> * test: fix tests Signed-off-by: Ning Yu <ningyu@automq.com> * test: test recover from discontinuous objects Signed-off-by: Ning Yu <ningyu@automq.com> * test: test recover from v0 and v1 objects Signed-off-by: Ning Yu <ningyu@automq.com> * style: fix lint Signed-off-by: Ning Yu <ningyu@automq.com> --------- Signed-off-by: Ning Yu <ningyu@automq.com>
Chillax-0v0
added a commit
that referenced
this pull request
May 14, 2025
* chore: add todos Signed-off-by: Ning Yu <ningyu@automq.com> * refactor(s3stream/object-wal): sequentially succeed Signed-off-by: Ning Yu <ningyu@automq.com> * refactor(s3stream/object-wal): drop discontinuous objects during recovery Signed-off-by: Ning Yu <ningyu@automq.com> * test: introduce `MockObjectStorage` Signed-off-by: Ning Yu <ningyu@automq.com> * test: test sequentially succeed Signed-off-by: Ning Yu <ningyu@automq.com> * refactor: record endOffset in the object path Signed-off-by: Ning Yu <ningyu@automq.com> * feat: different version of wal object header Signed-off-by: Ning Yu <ningyu@automq.com> * refactor: adapt to the new object header format Signed-off-by: Ning Yu <ningyu@automq.com> * feat: recover from the trim offset Signed-off-by: Ning Yu <ningyu@automq.com> * test: test recover continuous objects from trim offset Signed-off-by: Ning Yu <ningyu@automq.com> * test: test marshal and unmarshal wal object header Signed-off-by: Ning Yu <ningyu@automq.com> * test: fix tests Signed-off-by: Ning Yu <ningyu@automq.com> * test: test recover from discontinuous objects Signed-off-by: Ning Yu <ningyu@automq.com> * test: test recover from v0 and v1 objects Signed-off-by: Ning Yu <ningyu@automq.com> * style: fix lint Signed-off-by: Ning Yu <ningyu@automq.com> --------- Signed-off-by: Ning Yu <ningyu@automq.com>
Chillax-0v0
added a commit
that referenced
this pull request
May 14, 2025
* chore: add todos Signed-off-by: Ning Yu <ningyu@automq.com> * refactor(s3stream/object-wal): sequentially succeed Signed-off-by: Ning Yu <ningyu@automq.com> * refactor(s3stream/object-wal): drop discontinuous objects during recovery Signed-off-by: Ning Yu <ningyu@automq.com> * test: introduce `MockObjectStorage` Signed-off-by: Ning Yu <ningyu@automq.com> * test: test sequentially succeed Signed-off-by: Ning Yu <ningyu@automq.com> * refactor: record endOffset in the object path Signed-off-by: Ning Yu <ningyu@automq.com> * feat: different version of wal object header Signed-off-by: Ning Yu <ningyu@automq.com> * refactor: adapt to the new object header format Signed-off-by: Ning Yu <ningyu@automq.com> * feat: recover from the trim offset Signed-off-by: Ning Yu <ningyu@automq.com> * test: test recover continuous objects from trim offset Signed-off-by: Ning Yu <ningyu@automq.com> * test: test marshal and unmarshal wal object header Signed-off-by: Ning Yu <ningyu@automq.com> * test: fix tests Signed-off-by: Ning Yu <ningyu@automq.com> * test: test recover from discontinuous objects Signed-off-by: Ning Yu <ningyu@automq.com> * test: test recover from v0 and v1 objects Signed-off-by: Ning Yu <ningyu@automq.com> * style: fix lint Signed-off-by: Ning Yu <ningyu@automq.com> --------- Signed-off-by: Ning Yu <ningyu@automq.com>
Chillax-0v0
added a commit
that referenced
this pull request
May 14, 2025
* chore: add todos Signed-off-by: Ning Yu <ningyu@automq.com> * refactor(s3stream/object-wal): sequentially succeed Signed-off-by: Ning Yu <ningyu@automq.com> * refactor(s3stream/object-wal): drop discontinuous objects during recovery Signed-off-by: Ning Yu <ningyu@automq.com> * test: introduce `MockObjectStorage` Signed-off-by: Ning Yu <ningyu@automq.com> * test: test sequentially succeed Signed-off-by: Ning Yu <ningyu@automq.com> * refactor: record endOffset in the object path Signed-off-by: Ning Yu <ningyu@automq.com> * feat: different version of wal object header Signed-off-by: Ning Yu <ningyu@automq.com> * refactor: adapt to the new object header format Signed-off-by: Ning Yu <ningyu@automq.com> * feat: recover from the trim offset Signed-off-by: Ning Yu <ningyu@automq.com> * test: test recover continuous objects from trim offset Signed-off-by: Ning Yu <ningyu@automq.com> * test: test marshal and unmarshal wal object header Signed-off-by: Ning Yu <ningyu@automq.com> * test: fix tests Signed-off-by: Ning Yu <ningyu@automq.com> * test: test recover from discontinuous objects Signed-off-by: Ning Yu <ningyu@automq.com> * test: test recover from v0 and v1 objects Signed-off-by: Ning Yu <ningyu@automq.com> * style: fix lint Signed-off-by: Ning Yu <ningyu@automq.com> --------- Signed-off-by: Ning Yu <ningyu@automq.com>
This was referenced May 15, 2025
superhx
pushed a commit
that referenced
this pull request
May 15, 2025
refactor(s3stream/object-wal): complete appends sequentially (#2426) * chore: add todos * refactor(s3stream/object-wal): sequentially succeed * refactor(s3stream/object-wal): drop discontinuous objects during recovery * test: introduce `MockObjectStorage` * test: test sequentially succeed * refactor: record endOffset in the object path * feat: different version of wal object header * refactor: adapt to the new object header format * feat: recover from the trim offset * test: test recover continuous objects from trim offset * test: test marshal and unmarshal wal object header * test: fix tests * test: test recover from discontinuous objects * test: test recover from v0 and v1 objects * style: fix lint --------- Signed-off-by: Ning Yu <ningyu@automq.com>
superhx
pushed a commit
that referenced
this pull request
May 15, 2025
refactor(s3stream/object-wal): complete appends sequentially (#2426) * chore: add todos * refactor(s3stream/object-wal): sequentially succeed * refactor(s3stream/object-wal): drop discontinuous objects during recovery * test: introduce `MockObjectStorage` * test: test sequentially succeed * refactor: record endOffset in the object path * feat: different version of wal object header * refactor: adapt to the new object header format * feat: recover from the trim offset * test: test recover continuous objects from trim offset * test: test marshal and unmarshal wal object header * test: fix tests * test: test recover from discontinuous objects * test: test recover from v0 and v1 objects * style: fix lint --------- Signed-off-by: Ning Yu <ningyu@automq.com>
superhx
pushed a commit
that referenced
this pull request
May 15, 2025
refactor(s3stream/object-wal): complete appends sequentially (#2426) * chore: add todos * refactor(s3stream/object-wal): sequentially succeed * refactor(s3stream/object-wal): drop discontinuous objects during recovery * test: introduce `MockObjectStorage` * test: test sequentially succeed * refactor: record endOffset in the object path * feat: different version of wal object header * refactor: adapt to the new object header format * feat: recover from the trim offset * test: test recover continuous objects from trim offset * test: test marshal and unmarshal wal object header * test: fix tests * test: test recover from discontinuous objects * test: test recover from v0 and v1 objects * style: fix lint --------- Signed-off-by: Ning Yu <ningyu@automq.com>
superhx
pushed a commit
that referenced
this pull request
May 15, 2025
* refactor(s3stream/object-wal): complete appends sequentially (#2426) * chore: add todos Signed-off-by: Ning Yu <ningyu@automq.com> * refactor(s3stream/object-wal): sequentially succeed Signed-off-by: Ning Yu <ningyu@automq.com> * refactor(s3stream/object-wal): drop discontinuous objects during recovery Signed-off-by: Ning Yu <ningyu@automq.com> * test: introduce `MockObjectStorage` Signed-off-by: Ning Yu <ningyu@automq.com> * test: test sequentially succeed Signed-off-by: Ning Yu <ningyu@automq.com> * refactor: record endOffset in the object path Signed-off-by: Ning Yu <ningyu@automq.com> * feat: different version of wal object header Signed-off-by: Ning Yu <ningyu@automq.com> * refactor: adapt to the new object header format Signed-off-by: Ning Yu <ningyu@automq.com> * feat: recover from the trim offset Signed-off-by: Ning Yu <ningyu@automq.com> * test: test recover continuous objects from trim offset Signed-off-by: Ning Yu <ningyu@automq.com> * test: test marshal and unmarshal wal object header Signed-off-by: Ning Yu <ningyu@automq.com> * test: fix tests Signed-off-by: Ning Yu <ningyu@automq.com> * test: test recover from discontinuous objects Signed-off-by: Ning Yu <ningyu@automq.com> * test: test recover from v0 and v1 objects Signed-off-by: Ning Yu <ningyu@automq.com> * style: fix lint Signed-off-by: Ning Yu <ningyu@automq.com> --------- Signed-off-by: Ning Yu <ningyu@automq.com> * test: fix tests Signed-off-by: Ning Yu <ningyu@automq.com> --------- Signed-off-by: Ning Yu <ningyu@automq.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolve #2432
Performance Test