kv filter overwrites existing fields without warning/log entry

[awuetz]

The KV filter overwrites existing field values if the key is identical to the fieldname.

Assume you have the following log line:

url=http://www.google.com/my.jpeg,type=image/jpeg

And the following logstash configuration:

input { 
        file {  
                path => "/tmp/http_wwwrelay.csv"              
                type => "http_www"
        }
}

filter {
        if [type] == "http_www" {
                grok {    
                        match => { "message" => "^%{GREEDYDATA:kv_source}$" }
                }
                kv {
                        source => "kv_source"
                        field_split => ","
                        value_split => "="
                        remove_field => "kv_source"
        }
      }
}

output {
    if [type] == "http_www" {
              stdout {}
    }
}

At the input file statement we set "type" to "http_www".
After the kv filter processing, the field "type" with previous value "http_www" has the new value "image/jpeg". Thats why the output filter is not working.

The workaround is to use "prefix" or "target" at the kv filter, to find it out was quite hard.

The replacement of an existing field value should be logged.

[jsvd]

That's a problem indeed..finding overlapping fields on each event might be costly though..
Maybe we could issue a warning of possible overwrites on startup time if "kv" filter is used without prefix or target?

[awuetz]

Yes, that would be one solution. Maybe extending the documentation would be better.

I also thought about making the parameter prefix/target mandatory instead of optional. But i don't know if this is a good solution.

[widhalmt]

What about a blacklist of fields that can never be overwritten? Maybe even with some predefined excludes like "type"?

[suyograo]

Opened logstash-plugins/logstash-filter-kv#5