[SPARK-56898] Rewrite COUNT(DISTINCT IF) to COUNT(DISTINCT) FILTER for Expand reduction

[xumingming]

What changes were proposed in this pull request?

Adds RewriteCountDistinctConditional optimizer rule that canonicalizes:

  COUNT(DISTINCT IF(cond, base, NULL))
  COUNT(DISTINCT CASE WHEN cond THEN base END)

into:

  COUNT(DISTINCT base) FILTER (WHERE cond)

This reduces the number of distinct groups seen by RewriteDistinctAggregates from N (one per unique conditional expression) down to 1 (all share the same base column), collapsing the Expand factor from Nx to 1x.

Gated by spark.sql.optimizer.rewriteCountDistinctConditional.enabled (default: false).

Includes comprehensive unit tests for rewrite patterns and safety boundaries.

Why are the changes needed?

When a query contains many COUNT(DISTINCT IF(cond_i, col, NULL)) expressions over the same base column, RewriteDistinctAggregates treats each unique IF(...) expression as a distinct group. N conditions → N distinct groups → N× Expand amplification. In production workloads with 25–60 such expressions, this produces multi-terabyte shuffles and hour-long runtimes.

SELECT
  user_id,
  COUNT(DISTINCT IF(dt >= '2026-04-16', order_id, NULL)) AS orders_30d,
  COUNT(DISTINCT IF(dt >= '2026-02-15', order_id, NULL)) AS orders_90d,
  COUNT(DISTINCT IF(pay_status = 'paid', order_id, NULL)) AS orders_paid,
  -- ... 50 more expressions
FROM transactions
GROUP BY user_id 

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit Test.

Was this patch authored or co-authored using generative AI tooling?

No.

[xumingming]

@LuciferYang Can you help take a look at this PR?