debezium/dbz#1879 Add SMT EnforceRecordSize#284
Conversation
Signed-off-by: Thomas Thornton <thomaswilliamthornton@gmail.com>
There was a problem hiding this comment.
I wonder whether there should be more surgical algorithms, such as largest-first or greedy. I also wonder if the SMT should explicitly prefer a list of field names.
Let's say we have an event with 10 fields, 9 of which are 10 bytes each, and one is 5000 bytes, for a total of 5090 bytes. If we assume that the event's size exceeds a 2000-byte threshold, all 9 smaller fields are dropped to 6 bytes, which I think risks making the entire event potentially irrelevant.
Let's wait and see how Jiri views the PR behavior; maybe he sees something I am glossing over.
|
@twthorn Hi, thanks for the PR! Two comments
|
|
@Naros @jpechane thanks for the feedback. I opened debezium/debezium#7459 in debezium main repo and it addresses these comments as well. Please take a look when you get the chance, thank you. Once that is merged, I can update this PR to remove everything except the integration tests, which should still pass using the available SMT from the main repo. |
3 participants
debezium/dbz#1879
Add an SMT for enforcing record size with the current strategy of proportional column truncation.
We still need to performance test this. Some optimizations with caching may be needed.
We may support other strategies in the future, one such is payload offloading debezium/dbz#1949