Skip to content

Support VStream transaction chunking (prevents out of memory errors)#268

Open
twthorn wants to merge 2 commits into
debezium:mainfrom
twthorn:dbz#1519
Open

Support VStream transaction chunking (prevents out of memory errors)#268
twthorn wants to merge 2 commits into
debezium:mainfrom
twthorn:dbz#1519

Conversation

@twthorn
Copy link
Copy Markdown
Contributor

@twthorn twthorn commented Jan 10, 2026

Description

Fixes debezium/dbz#1519

Note: This PR will fail trivially until the vitess-grpc-client version is released & upgraded.

Add support for transaction chunking

Testing

Integration tests

  1. Verify that Debezium receives a write operation distributed across shards with chunking enabled as two atomic chunked transactions on each shard (not interleaved). And verifies that total order is tracked correctly across chunked transactions
  2. (Currently will fail) verify that enabling/migrating to chunking, ensures that the transaction ordering guarantees are preserved

TODO

debezium/dbz#1530
The second itest is intentionally failing and is from TDD to demonstrate the edge case of switching from using the current VGTID to trailing/last VGTID (only feasible one to send when chunking is enabled). I will add another PR that adds a feature called connector generation, which can be used to trigger an epoch bump when switching on chunking. Since epoch is the first variable compared between events, it will allow all events with trailing/last VGTID after chunking is enabled to still be evaluated as newer for downstream consumers. See here for more details on comparison logic:

  1. If the values for transaction_epoch are not equal, return the event with the higher transaction_epoch value. Otherwise, continue.
  2. If the values for transaction_rank are not equal, return the event with the higher transaction_rank value. Otherwise, continue.
  3. Return the event with a greater total_order value.

@github-actions
Copy link
Copy Markdown

Hi @twthorn, thanks for your contribution. Please prefix the commit message(s) with the debezium/dbz#xxx GitHub issue key.

Copy link
Copy Markdown
Contributor

@jpechane jpechane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@twthorn LGTM, thanks! Could you please fix the commit message and sign off the commit?

Signed-off-by: Thomas Thornton <thomaswilliamthornton@gmail.com>
…nabled

Signed-off-by: Thomas Thornton <thomaswilliamthornton@gmail.com>
@twthorn
Copy link
Copy Markdown
Contributor Author

twthorn commented Jan 20, 2026

Pre-requisite PR so we can get the itest passing on this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waits for other PR Can only be merged once another PR has been merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support for transaction chunking in vitess-connector

2 participants