Skip to content

feat(new sink): new journald sink#23593

Open
wiktorsikora wants to merge 15 commits into
vectordotdev:masterfrom
wiktorsikora:add-journald-sink
Open

feat(new sink): new journald sink#23593
wiktorsikora wants to merge 15 commits into
vectordotdev:masterfrom
wiktorsikora:add-journald-sink

Conversation

@wiktorsikora
Copy link
Copy Markdown

@wiktorsikora wiktorsikora commented Aug 13, 2025

Summary

This PR introduces support for journald sink as discussed in #19177.

Vector configuration

# Journald Sink Example
# ------------------------------------------------------------------------------
# A simple example showing how to send logs to systemd journal.
# This demonstrates forwarding logs from a dummy source, parsing syslog messages,
# and mapping syslog severity levels to journal priority levels.

sources:
  dummy_logs:
    type: "demo_logs"
    format: "syslog"
    interval: 1

transforms:
  parse_syslog:
    type: "remap"
    inputs: ["dummy_logs"]
    source: |
      . = parse_syslog!(string!(.message))

  map_severity_to_priority:
    # Map the syslog severity to journal priority levels
    type: "remap"
    inputs: ["parse_syslog"]
    source: |
      .priority = if .severity == "emerg" {
        0
      } else if .severity == "alert" {
        1
      } else if .severity == "crit" {
        2
      } else if .severity == "err" {
        3
      } else if .severity == "warning" {
        4
      } else if .severity == "notice" {
        5
      } else if .severity == "info" {
        6
      } else if .severity == "debug" {
        7
      } else {
        6
      }

sinks:
  local_journal:
    inputs: ["map_severity_to_priority"]
    type: "journald"

How did you test this PR?

I tested the implementation locally on a Fedora 42 system.

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

Signed-off-by: Wiktor Sikora <wiktorsikora7@gmail.com>
Signed-off-by: Wiktor Sikora <wiktorsikora7@gmail.com>
Signed-off-by: Wiktor Sikora <wiktorsikora7@gmail.com>
Signed-off-by: Wiktor Sikora <wiktorsikora7@gmail.com>
Signed-off-by: Wiktor Sikora <wiktorsikora7@gmail.com>
@wiktorsikora wiktorsikora requested a review from a team as a code owner August 13, 2025 20:15
@github-actions github-actions Bot added the domain: sinks Anything related to the Vector's sinks label Aug 13, 2025
@thomasqueirozb
Copy link
Copy Markdown
Member

Hi @wiktorsikora, thank you for your submission! All new sinks/sources need documentation. Your would be in website/cue/reference/components/sinks/journald.cue. Also please run make generate-component-docs so that the config will also be generated. The Vector team will take a closer look at the code once this is added. Thanks!

@thomasqueirozb thomasqueirozb added sink: new Request or implementation of a new sink meta: awaiting author Pull requests that are awaiting their author. labels Aug 14, 2025
@github-actions github-actions Bot removed the meta: awaiting author Pull requests that are awaiting their author. label Aug 14, 2025
Signed-off-by: Wiktor Sikora <wiktorsikora7@gmail.com>
@wiktorsikora wiktorsikora requested a review from a team as a code owner August 14, 2025 19:25
@github-actions github-actions Bot added the domain: external docs Anything related to Vector's external, public documentation label Aug 14, 2025
cswatt
cswatt previously approved these changes Aug 15, 2025
Signed-off-by: Wiktor Sikora <wiktorsikora7@gmail.com>
@wiktorsikora
Copy link
Copy Markdown
Author

Thanks for the tips, I've added the documentation. I'd appreciate it if you could double-check it as I'm not entirely sure I've done everything correctly, especially the features section.

Signed-off-by: Wiktor Sikora <wiktorsikora7@gmail.com>
Signed-off-by: Wiktor Sikora <wiktorsikora7@gmail.com>
@wiktorsikora
Copy link
Copy Markdown
Author

@vectordotdev/vector just a gentle reminder about this PR :)

Copy link
Copy Markdown
Member

@pront pront left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @wiktorsikora, this looks good! Could we add tests here? Or a new integration test.

@pront pront added the meta: awaiting author Pull requests that are awaiting their author. label Oct 23, 2025
}

/// Sanitize a key and convert it to uppercase.
fn sanitize_key(&mut self, key: &str) {
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One small edge case I noticed: this may still produce field names that journald rejects.

The native receiver validates both plain and length-encoded fields with journal_field_valid(..., false): plain fields, length-encoded fields.

A few examples that look worth handling:

  • 🔥 can sanitize to an empty name, but empty names are rejected: l <= 0
  • _SYSTEMD_UNIT keeps the leading _, but client-provided protected fields are rejected: p[0] == '_'
  • 123abc keeps the leading digit, but digit-prefixed names are rejected: ascii_isdigit(p[0])
  • if an escape/fallback prefix is added, the final name still needs to stay within 64 bytes: l > 64

Comment thread src/sinks/mod.rs
pub mod humio;
#[cfg(any(feature = "sinks-influxdb", feature = "prometheus-integration-tests"))]
pub mod influxdb;
#[cfg(feature = "sinks-journald")]
Copy link
Copy Markdown

@DCjanus DCjanus May 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this should be Linux-only too, since the implementation depends on Linux/systemd-specific pieces such as memfd_create.

Suggested change
#[cfg(feature = "sinks-journald")]
#[cfg(all(target_os = "linux", feature = "sinks-journald"))]

Comment on lines +45 to +50
let bytes_sent = self.send_payload(&self.buf).await?;
// Clear the buffer after sending
// We could also keep the buffer for reuse, but by doing this we ensure that
// we don't allocate too much memory for long time in case of rare large payloads.
self.buf = vec![];
Ok(bytes_sent)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If send_payload returns an error here, self.buf is not cleared, so the next event may append to the previous failed payload.

Suggested change
let bytes_sent = self.send_payload(&self.buf).await?;
// Clear the buffer after sending
// We could also keep the buffer for reuse, but by doing this we ensure that
// we don't allocate too much memory for long time in case of rare large payloads.
self.buf = vec![];
Ok(bytes_sent)
let payload = std::mem::take(&mut self.buf);
let bytes_sent = self.send_payload(&payload).await?;
Ok(bytes_sent)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: external docs Anything related to Vector's external, public documentation domain: sinks Anything related to the Vector's sinks meta: awaiting author Pull requests that are awaiting their author. sink: new Request or implementation of a new sink

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Systemd journal sink

5 participants