Support VStream transaction chunking (prevents out of memory errors)#268
Open
twthorn wants to merge 2 commits into
Open
Support VStream transaction chunking (prevents out of memory errors)#268twthorn wants to merge 2 commits into
twthorn wants to merge 2 commits into
Conversation
|
Hi @twthorn, thanks for your contribution. Please prefix the commit message(s) with the debezium/dbz#xxx GitHub issue key. |
Signed-off-by: Thomas Thornton <thomaswilliamthornton@gmail.com>
…nabled Signed-off-by: Thomas Thornton <thomaswilliamthornton@gmail.com>
Contributor
Author
|
Pre-requisite PR so we can get the itest passing on this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes debezium/dbz#1519
Note: This PR will fail trivially until the vitess-grpc-client version is released & upgraded.
Add support for transaction chunking
Testing
Integration tests
TODO
debezium/dbz#1530
The second itest is intentionally failing and is from TDD to demonstrate the edge case of switching from using the current VGTID to trailing/last VGTID (only feasible one to send when chunking is enabled). I will add another PR that adds a feature called connector generation, which can be used to trigger an epoch bump when switching on chunking. Since epoch is the first variable compared between events, it will allow all events with trailing/last VGTID after chunking is enabled to still be evaluated as newer for downstream consumers. See here for more details on comparison logic: