Support TIMESTAMP_ISO8601 in HAProxy patterns

[antaflos]

Rsyslog (and probably others) can be configured to use ISO8601
timestamps, usually high-precision with microsecond resolution. The
HAProxy patterns should take this possibility into account and not be
arbitrarily limited to just SYSLOGTIMESTAMP.

[antaflos]

This contains updated spec tests for the HAProxy pattern but could not get them to run because I seem to be doing something wrong with JRuby, RVM and related Gems.

[igalic]

igalic@levix ~/src/elastic/logstash-patterns-core (git)-[antaflos_haproxy-timestamp8601] % bundle exec rspec spec/patterns/*_spec.rb
Using Accessor#strict_set for specs
Run options: exclude {:redis=>true, :socket=>true, :performance=>true, :couchdb=>true, :elasticsearch=>true, :elasticsearch_secure=>true, :export_cypher=>true, :integration=>true, :windows=>true}
..............................................................................FFFF.......................................................................................................................................

Failures:

  1) HAPROXY Parsing HAPROXY log line from raw syslog line should include {"client_ip" => "127.0.0.1"}
     Failure/Error: it { should include("client_ip" => "127.0.0.1") }
       expected {"message" => "Dec  9 13:01:26 localhost haproxy[28029]: 127.0.0.1:39759 [09/Dec/2013:12:59:46.633] loadbalancer default/instance8 0/51536/1/48082/99627 200 83285 - - ---- 87/87/87/1/0 0/67 {77.24.148.74} \"GET /path/to/image HTTP/1.1\"", "@version" => "1", "@timestamp" => "2015-08-26T13:02:48.683Z", "syslog_timestamp" => "Dec  9 13:01:26"} to include {"client_ip" => "127.0.0.1"}
       Diff:
       @@ -1,2 +1,5 @@
       -[{"client_ip"=>"127.0.0.1"}]
       +"@timestamp" => "2015-08-26T13:02:48.683Z",
       +"@version" => "1",
       +"message" => "Dec  9 13:01:26 localhost haproxy[28029]: 127.0.0.1:39759 [09/Dec/2013:12:59:46.633] loadbalancer default/instance8 0/51536/1/48082/99627 200 83285 - - ---- 87/87/87/1/0 0/67 {77.24.148.74} \"GET /path/to/image HTTP/1.1\"",
       +"syslog_timestamp" => "Dec  9 13:01:26",
     # ./spec/patterns/haproxy_spec.rb:15:in `(root)'

  2) HAPROXY Parsing HAPROXY log line from raw syslog line should include {"program" => "haproxy"}
     Failure/Error: it { should include("program" => "haproxy") }
       expected {"message" => "Dec  9 13:01:26 localhost haproxy[28029]: 127.0.0.1:39759 [09/Dec/2013:12:59:46.633] loadbalancer default/instance8 0/51536/1/48082/99627 200 83285 - - ---- 87/87/87/1/0 0/67 {77.24.148.74} \"GET /path/to/image HTTP/1.1\"", "@version" => "1", "@timestamp" => "2015-08-26T13:02:48.847Z", "syslog_timestamp" => "Dec  9 13:01:26"} to include {"program" => "haproxy"}
       Diff:
       @@ -1,2 +1,5 @@
       -[{"program"=>"haproxy"}]
       +"@timestamp" => "2015-08-26T13:02:48.847Z",
       +"@version" => "1",
       +"message" => "Dec  9 13:01:26 localhost haproxy[28029]: 127.0.0.1:39759 [09/Dec/2013:12:59:46.633] loadbalancer default/instance8 0/51536/1/48082/99627 200 83285 - - ---- 87/87/87/1/0 0/67 {77.24.148.74} \"GET /path/to/image HTTP/1.1\"",
       +"syslog_timestamp" => "Dec  9 13:01:26",
     # ./spec/patterns/haproxy_spec.rb:14:in `(root)'

  3) HAPROXY Parsing HAPROXY log line from raw syslog line should include {"server_name" => "instance8"}
     Failure/Error: it { should include("server_name" => "instance8") }
       expected {"message" => "Dec  9 13:01:26 localhost haproxy[28029]: 127.0.0.1:39759 [09/Dec/2013:12:59:46.633] loadbalancer default/instance8 0/51536/1/48082/99627 200 83285 - - ---- 87/87/87/1/0 0/67 {77.24.148.74} \"GET /path/to/image HTTP/1.1\"", "@version" => "1", "@timestamp" => "2015-08-26T13:02:48.982Z", "syslog_timestamp" => "Dec  9 13:01:26"} to include {"server_name" => "instance8"}
       Diff:
       @@ -1,2 +1,5 @@
       -[{"server_name"=>"instance8"}]
       +"@timestamp" => "2015-08-26T13:02:48.982Z",
       +"@version" => "1",
       +"message" => "Dec  9 13:01:26 localhost haproxy[28029]: 127.0.0.1:39759 [09/Dec/2013:12:59:46.633] loadbalancer default/instance8 0/51536/1/48082/99627 200 83285 - - ---- 87/87/87/1/0 0/67 {77.24.148.74} \"GET /path/to/image HTTP/1.1\"",
       +"syslog_timestamp" => "Dec  9 13:01:26",
     # ./spec/patterns/haproxy_spec.rb:17:in `(root)'

  4) HAPROXY Parsing HAPROXY log line from raw syslog line should include {"http_verb" => "GET"}
     Failure/Error: it { should include("http_verb" => "GET") }
       expected {"message" => "Dec  9 13:01:26 localhost haproxy[28029]: 127.0.0.1:39759 [09/Dec/2013:12:59:46.633] loadbalancer default/instance8 0/51536/1/48082/99627 200 83285 - - ---- 87/87/87/1/0 0/67 {77.24.148.74} \"GET /path/to/image HTTP/1.1\"", "@version" => "1", "@timestamp" => "2015-08-26T13:02:49.089Z", "syslog_timestamp" => "Dec  9 13:01:26"} to include {"http_verb" => "GET"}
       Diff:
       @@ -1,2 +1,5 @@
       -[{"http_verb"=>"GET"}]
       +"@timestamp" => "2015-08-26T13:02:49.089Z",
       +"@version" => "1",
       +"message" => "Dec  9 13:01:26 localhost haproxy[28029]: 127.0.0.1:39759 [09/Dec/2013:12:59:46.633] loadbalancer default/instance8 0/51536/1/48082/99627 200 83285 - - ---- 87/87/87/1/0 0/67 {77.24.148.74} \"GET /path/to/image HTTP/1.1\"",
       +"syslog_timestamp" => "Dec  9 13:01:26",
     # ./spec/patterns/haproxy_spec.rb:16:in `(root)'

Finished in 27.98 seconds (files took 4.83 seconds to load)
217 examples, 4 failures

Failed examples:

rspec ./spec/patterns/haproxy_spec.rb:15 # HAPROXY Parsing HAPROXY log line from raw syslog line should include {"client_ip" => "127.0.0.1"}
rspec ./spec/patterns/haproxy_spec.rb:14 # HAPROXY Parsing HAPROXY log line from raw syslog line should include {"program" => "haproxy"}
rspec ./spec/patterns/haproxy_spec.rb:17 # HAPROXY Parsing HAPROXY log line from raw syslog line should include {"server_name" => "instance8"}
rspec ./spec/patterns/haproxy_spec.rb:16 # HAPROXY Parsing HAPROXY log line from raw syslog line should include {"http_verb" => "GET"}

Randomized with seed 22622

bundle exec rspec spec/patterns/*_spec.rb  110,09s user 2,87s system 218% cpu 51,741 total
1 igalic@levix ~/src/elastic/logstash-patterns-core (git)-[antaflos_haproxy-timestamp8601] %

[antaflos]

So the spec test I didn't touch fails? Weird.

[antaflos]

Ok, I think I see where I went wrong, will fix soon.

[igalic]

re-ran tests.
all is well now!

[purbon]

Thanks for your contribution @antaflos and many thanks @lgalic for validating the test in here! much appreciate it.

LGTM

[elasticsearch-bot]

Merged sucessfully into master!