Make pattern elements matching integers emit integer fields

[magnusbaeck]

There are a number of patterns that match integers (or floats) via e.g. %{INT:foo} that are emitting string values for values that cannot be anything but numeric. This is an annoyance since it forces users to define their own Elasticsearch index templates with explicit mappings to get the fields correctly mapped in Elasticsearch. Users shouldn't have to do that if all they want to do is parse and visualize an Apache log; index templates should be for experienced users.

Example with problematic tokens highlighted:

HAPROXYHTTP %{SYSLOGTIMESTAMP:syslog_timestamp} %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:%{INT:client_port} [%{HAPROXYDATE:accept_date}] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_request}/%{INT:time_queue}/%{INT:time_backend_connect}/%{INT:time_backend_response}/%{NOTSPACE:time_duration} %{INT:http_status_code} %{NOTSPACE:bytes_read} %{DATA:captured_request_cookie} %{DATA:captured_response_cookie} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue} ({%{HAPROXYCAPTUREDREQUESTHEADERS}})?( )?({%{HAPROXYCAPTUREDRESPONSEHEADERS}})?( )?"(|(%{WORD:http_verb} (%{URIPROTO:http_proto}://)?(?:%{USER:http_user}(?::[^@]*)?@)?(?:%{URIHOST:http_host})?(?:%{URIPATHPARAM:http_request})?( HTTP/%{NUMBER:http_version})?))?"

[jordansissel]

Agreed we should have better defaults on built-in patterns that number/int/etc are represented by the appropriate types internally. However, changing this will break backwards compatibility, so we'll have to either be very careful with announcements or make new patterns (HAPROXYHTTP2)

[magnusbaeck]

Yep. While I don't like the idea of increasing the number of patterns it would be a backwards compatibility break. I doubt that many are relying on the fields to be strings, though.

[SamSaffron]

thanks, awesome, on a related note:

be very careful mucking with IP:client_ip ... cause its actually either ipv4 or ipv6 depending on what client accessed it. was in for a rude shock when I tried to map that to ip without a mutate

[purbon]

@jordansissel nowadays that we are aiming to release 2.0 soon, this would be a nice change we could easy introduce, so breaking backwards compatibility would not be that hard. What do you think?

[SamSaffron]

FYI this is what I am using at Discourse to parse our haproxy log, I am getting a bunch of grok failures so there are still some cases the grok pattern is bad:

https://gist.github.com/SamSaffron/0bcbcdcf16e84accd4d0

https://gist.github.com/SamSaffron/953d3b0de2662b8af205

The ipv4 splitting though needs some thinking about, I needed to add custom code for that

This is the most common failure pattern

Sep 8 08:02:38 tieinterceptor2 haproxy[15603]: 10.0.0.1:55876 [08/Sep/2015:08:02:38.427] http-in/9: SSL handshake failure

But there are some longer lines as well usually around truncated log lines

[colinmollenhour]

Additionally, IP addresses as bytes should be stored as int so that histograms can be built on them as per:

elastic/logstash#2868

It looks like for Apache #102 already resolves this but that only applies to Apache.