[Draft] Add compression ratio calculation and per-column compression stats (#18184) by johnsolomonj · Pull Request #18185 · apache/pinot

Add forward index compression ratio and per-column compression stats to table size API

jsol-splunk

committed

Add comprehensive tests and fix redundant loop for compression stats feature

jsol-splunk

committed

Restructure compression stats API: nested DTOs, feature flag gating, tier breakdown, and stale metadata cleanup

jsol-splunk

committed

Add writer tracking gating, per-column compression stats in metadata API, and comprehensive tests

jsol-splunk

committed

Restructure columnCompressionStats to standard array DTO and gate on feature flag

jsol-splunk

committed

Fix default codec persistence, dictionary column stats, stale metrics, and metadata endpoint gaps

jsol-splunk

committed

Fix compression stats API structure, field names, and dict-only suppression

jsol-splunk

committed

Add compressionStats and storageBreakdown as proper DTO fields on metadata endpoint

jsol-splunk

committed

Fix metadata endpoint DTO immutability, coverage fields, and writer tracking accuracy

jsol-splunk

committed

Exclude old segments without stats from compression ratio denominator

jsol-splunk

committed

Fix writer uncompressed size tracking and harden compression stats API

jsol-splunk

committed

Fix negative ratio, tier gauge leak, and CLP codec persistence

jsol-splunk

committed

Consolidate CLP codec compression resolution into ForwardIndexType

jsol-splunk

committed

Preserve tier info when compression stats flag is off, skip stale columns

jsol-splunk

committed

Fix replica normalization for dict column sentinels and empty summaries

jsol-splunk

committed

Fix tier gauge leader check and exclude old raw segments from server stats

jsol-splunk

committed

Fix post-rebase import ordering, constant rename, and deprecated API

jsol-splunk

committed

Add missing test coverage for ColumnMetadataImpl and CompressionStatsSummary

jsol-splunk

committed

Expand test coverage for compression stats across all affected modules

jsol-splunk

committed

Polish TableSizeResourceTest: remove redundant queryParam and align assertion style

jsol-splunk

committed

Add ServerSegmentMetadataReader coverage: CompressionStatsSummary/StorageBreakdown aggregation and dict sentinel path

jsol-splunk

committed

Fix missing @Nullable import in ColumnMetadata after rebase onto #18470

jsol-splunk

committed

Add CLPForwardIndexCreatorV2 getUncompressedSize and setTrackUncompressedSize coverage

jsol-splunk

committed

Retrigger CI

jsol-splunk

committed

Fix hasDictionary=true wrongly reported for raw (no-dict) columns in mixed-age tables

jsol-splunk

committed

Revert unrelated ZkStarter timing changes from hasDictionary fix commit

jsol-splunk

committed

Fix isHasDictionary asymmetric JSON serialization and resolveCompressionType NPE risk

jsol-splunk

committed

Add default impl for isCompressionStatsEnabled in IndexCreationContext SPI

jsol-splunk

committed

Fix tier gauge using wrong metric API — use setOrUpdateTableGauge/removeTableGauge(name, key, gauge)

jsol-splunk

committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Draft] Add compression ratio calculation and per-column compression stats (#18184)#18185

[Draft] Add compression ratio calculation and per-column compression stats (#18184)#18185
johnsolomonj wants to merge 29 commits into
apache:masterfrom
johnsolomonj:feature/compression-stats-tracking

Commits on May 19, 2026

Add forward index compression ratio and per-column compression stats to table size API

Add comprehensive tests and fix redundant loop for compression stats feature

Restructure compression stats API: nested DTOs, feature flag gating, tier breakdown, and stale metadata cleanup

Add writer tracking gating, per-column compression stats in metadata API, and comprehensive tests

Restructure columnCompressionStats to standard array DTO and gate on feature flag

Fix default codec persistence, dictionary column stats, stale metrics, and metadata endpoint gaps

Fix compression stats API structure, field names, and dict-only suppression

Add compressionStats and storageBreakdown as proper DTO fields on metadata endpoint

Fix metadata endpoint DTO immutability, coverage fields, and writer tracking accuracy

Exclude old segments without stats from compression ratio denominator

Fix writer uncompressed size tracking and harden compression stats API

Fix negative ratio, tier gauge leak, and CLP codec persistence

Consolidate CLP codec compression resolution into ForwardIndexType

Preserve tier info when compression stats flag is off, skip stale columns

Fix replica normalization for dict column sentinels and empty summaries

Fix tier gauge leader check and exclude old raw segments from server stats

Fix post-rebase import ordering, constant rename, and deprecated API

Add missing test coverage for ColumnMetadataImpl and CompressionStatsSummary

Expand test coverage for compression stats across all affected modules

Polish TableSizeResourceTest: remove redundant queryParam and align assertion style

Add ServerSegmentMetadataReader coverage: CompressionStatsSummary/StorageBreakdown aggregation and dict sentinel path

Fix missing @Nullable import in ColumnMetadata after rebase onto #18470

Add CLPForwardIndexCreatorV2 getUncompressedSize and setTrackUncompressedSize coverage

Retrigger CI

Fix hasDictionary=true wrongly reported for raw (no-dict) columns in mixed-age tables

Revert unrelated ZkStarter timing changes from hasDictionary fix commit

Fix isHasDictionary asymmetric JSON serialization and resolveCompressionType NPE risk

Add default impl for isCompressionStatsEnabled in IndexCreationContext SPI

Fix tier gauge using wrong metric API — use setOrUpdateTableGauge/removeTableGauge(name, key, gauge)