[pkg/ottl] The Substring function corrupts multibyte UTF-8 strings (byte based slicing)

[Nagato-Yuzuru]

Component(s)

pkg/ottl

What happened?

Description

Substring slices its target by bytes, not by characters. ASCII inputs happen to work. Multibyte UTF-8 inputs (like CJK or emoji) either return invalid UTF-8 when the slice cuts mid-rune or a string shorter than the user asked for.

Steps to Reproduce

use v0.151.0 (likely exists in all less current versions)

Reproducible on https://ottl.run/ with the config and payload below.

payload:

{
    "resourceLogs": [{
      "resource": {},
      "scopeLogs": [{
        "scope": {},
        "logRecords": [{
          "timeUnixNano": "1700000000000000000",
          "body": {"stringValue": "test"},
          "attributes": [
            {"key": "greeting", "value": {"stringValue": "日本語"}}
          ]
        }]
      }]
    }]
  }

config:

transform:
  log_statements:
    - context: log
      statements:
        - set(attributes["first_char"], Substring(attributes["greeting"], 0, 1))
        - set(attributes["three"], Substring(attributes["greeting"], 0, 3))

Expected Result

{
  "resourceLogs": [
    {
      "resource": {},
      "scopeLogs": [
        {
          "scope": {},
          "logRecords": [
            {
              "timeUnixNano": "1700000000000000000",
              "body": {
                "stringValue": "test"
              },
              "attributes": [
                {
                  "key": "greeting",
                  "value": {
                    "stringValue": "日本語"
                  }
                },
                {
                  "key": "first_char",
                  "value": {
                    "stringValue": "日"
                  }
                },
                {
                  "key": "three",
                  "value": {
                    "stringValue": "日本語"
                  }
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

Actual Result

{
  "resourceLogs": [
    {
      "resource": {},
      "scopeLogs": [
        {
          "scope": {},
          "logRecords": [
            {
              "timeUnixNano": "1700000000000000000",
              "body": {
                "stringValue": "test"
              },
              "attributes": [
                {
                  "key": "greeting",
                  "value": {
                    "stringValue": "日本語"
                  }
                },
                {
                  "key": "first_char",
                  "value": {
                    "stringValue": "�"
                  }
                },
                {
                  "key": "three",
                  "value": {
                    "stringValue": "日"
                  }
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

Collector version

v0.151.0

Environment information

OS: alpine:3.22.4

OpenTelemetry Collector configuration

Log output

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• pkg/ottl: @TylerHelmuth @evan-bradley @edmocosta @bogdandrutu

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[Nagato-Yuzuru]

Looks like the offending line is ottlfuncs/func_substring.go:57, where the byte slice is returned directly. Happy to send a PR if you're open to the direction.

[Dipanshusinghh]

Hey @Nagato-Yuzuru, good catch..

The issue happens because standard string slicing in Go (str[start:end]) operates on bytes, not actual characters. So it easily cuts multibyte UTF-8 runes (like CJK or emojis) right in the middle, returning invalid chars

To fix this, we need to convert the string into a slice of runes first before slicing it, something like []rune(str)[start:end]. We also need to add a quick length/bounds check on the rune slice so it doesnt panic if the requested substring length is bigger than the actual string

I would love to pick this up and open a PR if no one else is already working on it. Let me know