Describe the solution you'd like
Being able to dynamically create files according to some field in the log record would be very appreciated. For instance, when reading logs from Proxies it would be nice to store the logs in a file that would correspond to, for instance, the host of the request. The same would be for the path configuration parameter, for instance, we have multiple proxies and we want to create directories and store the records based on the proxy name which is inside the log record.
Accessing data from the Logs already happens in other plugins like rewrite_tag (See example bellow), and out_opensearch where you can define the index_name with data from the log itself, so this would be just a matter of having a bit more consistency rather than a completely new feature.
Describe alternatives you've considered
There is a current work around for the file parameter but it's ugly and tricky, which is to rewrite the TAG of the record to whatever we want. Later on, the out_file will name the file according to the TAG name if no File parameter was provided.
Additional context
Let's give an example:
I want to write my logs into a directory based on the proxy_name field and a file based on the hostname field of my logs. My proxies are: proxy1 & proxy2. My hosts are: host1 & host2.
Current work around solution
[INPUT]
Name forward
Tag proxy-requests
Listen 0.0.0.0
Port 24224
[FILTER]
Name rewrite_tag
Match proxy-requests
Rule $proxy_name ^(.*)$ $proxy_name.$hostname.log false
[OUTPUT]
Match proxy1*
Name file
path /proxy1/
Format plain
[OUTPUT]
Match proxy2*
Name file
path /proxy2/
Format plain
The above extended and ugly configuration will generate the following files which are not the best naming but work:
/proxy1/proxy1.host1.log
/proxy1/proxy1.host2.log
/proxy2/proxy2.host1.log
/proxy2/proxy2.host2.log
Desired
[INPUT]
Name forward
Tag proxy-requests
Listen 0.0.0.0
Port 24224
[OUTPUT]
Match *
Name file
Path /$proxy_name/
File $hostname.log
Format plain
The above simpler config would generate the desired behavior:
/proxy1/host1.log
/proxy1/host2.log
/proxy2/host1.log
/proxy2/host2.log
Edge cases & considerations
Missing field
If there referenced field doesn't exist, you could make it configurable like On_Missing_Field drop|fallback|use_default or just apply as default one of the following drop, fallback use_default
Data Validation
Data coming from the record should be validated to comply with filesystem naming standard, but I believe this could be something do use should be careful about. And a simple notice on the documentation would help.
High cardinality / file descriptor exhaustion
This again should be something the user needs to be careful about, or a new configuration field like max-open-files could be introduced. And again, a little notice on the documentation about this would work as well.
Describe the solution you'd like
Being able to dynamically create files according to some field in the log record would be very appreciated. For instance, when reading logs from Proxies it would be nice to store the logs in a file that would correspond to, for instance, the host of the request. The same would be for the
pathconfiguration parameter, for instance, we have multiple proxies and we want to create directories and store the records based on the proxy name which is inside the log record.Accessing data from the Logs already happens in other plugins like
rewrite_tag(See example bellow), andout_opensearchwhere you can define theindex_namewith data from the log itself, so this would be just a matter of having a bit more consistency rather than a completely new feature.Describe alternatives you've considered
There is a current work around for the file parameter but it's ugly and tricky, which is to rewrite the TAG of the record to whatever we want. Later on, the
out_filewill name the file according to the TAG name if noFileparameter was provided.Additional context
Let's give an example:
I want to write my logs into a directory based on the
proxy_namefield and a file based on thehostnamefield of my logs. My proxies are:proxy1&proxy2. My hosts are:host1&host2.Current work around solution
The above extended and ugly configuration will generate the following files which are not the best naming but work:
/proxy1/proxy1.host1.log
/proxy1/proxy1.host2.log
/proxy2/proxy2.host1.log
/proxy2/proxy2.host2.log
Desired
The above simpler config would generate the desired behavior:
/proxy1/host1.log
/proxy1/host2.log
/proxy2/host1.log
/proxy2/host2.log
Edge cases & considerations
Missing field
If there referenced field doesn't exist, you could make it configurable like
On_Missing_Field drop|fallback|use_defaultor just apply as default one of the followingdrop,fallbackuse_defaultData Validation
Data coming from the record should be validated to comply with filesystem naming standard, but I believe this could be something do use should be careful about. And a simple notice on the documentation would help.
High cardinality / file descriptor exhaustion
This again should be something the user needs to be careful about, or a new configuration field like
max-open-filescould be introduced. And again, a little notice on the documentation about this would work as well.