Skip to content

out_file: Enabled record accessor for path and file configuration parameters #11794

@DanielPortovedo

Description

@DanielPortovedo

Describe the solution you'd like
Being able to dynamically create files according to some field in the log record would be very appreciated. For instance, when reading logs from Proxies it would be nice to store the logs in a file that would correspond to, for instance, the host of the request. The same would be for the path configuration parameter, for instance, we have multiple proxies and we want to create directories and store the records based on the proxy name which is inside the log record.

Accessing data from the Logs already happens in other plugins like rewrite_tag (See example bellow), and out_opensearch where you can define the index_name with data from the log itself, so this would be just a matter of having a bit more consistency rather than a completely new feature.

Describe alternatives you've considered
There is a current work around for the file parameter but it's ugly and tricky, which is to rewrite the TAG of the record to whatever we want. Later on, the out_file will name the file according to the TAG name if no File parameter was provided.

Additional context
Let's give an example:

I want to write my logs into a directory based on the proxy_name field and a file based on the hostname field of my logs. My proxies are: proxy1 & proxy2. My hosts are: host1 & host2.

Current work around solution

[INPUT]
    Name      forward
    Tag           proxy-requests
    Listen       0.0.0.0
    Port          24224

[FILTER]
    Name    rewrite_tag
    Match   proxy-requests

    Rule    $proxy_name ^(.*)$ $proxy_name.$hostname.log false

[OUTPUT]
    Match       proxy1*
    Name        file
    path        /proxy1/
    Format      plain

[OUTPUT]
    Match       proxy2*
    Name        file
    path        /proxy2/
    Format      plain

The above extended and ugly configuration will generate the following files which are not the best naming but work:

/proxy1/proxy1.host1.log
/proxy1/proxy1.host2.log
/proxy2/proxy2.host1.log
/proxy2/proxy2.host2.log

Desired

[INPUT]
    Name      forward
    Tag           proxy-requests
    Listen       0.0.0.0
    Port          24224

[OUTPUT]
    Match       *
    Name        file
    Path          /$proxy_name/
    File            $hostname.log
    Format      plain

The above simpler config would generate the desired behavior:

/proxy1/host1.log
/proxy1/host2.log
/proxy2/host1.log
/proxy2/host2.log

Edge cases & considerations
Missing field
If there referenced field doesn't exist, you could make it configurable like On_Missing_Field drop|fallback|use_default or just apply as default one of the following drop, fallback use_default

Data Validation
Data coming from the record should be validated to comply with filesystem naming standard, but I believe this could be something do use should be careful about. And a simple notice on the documentation would help.

High cardinality / file descriptor exhaustion
This again should be something the user needs to be careful about, or a new configuration field like max-open-files could be introduced. And again, a little notice on the documentation about this would work as well.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions