optimize the optimizer

[Licenser]

Right now DQE always tries to dedublicate the query, with large queries this in itself is expensive, a better way might be to only do that when we have a decent ammount (10%?) of doublicated get's (this is easy to determin in the flattened form.

[Licenser]

on this guy we save about 400ms that is quite nice

After before optimizing the optimizer:

04-19 19:21:57.486 [debug] <0.1371.0>@dqe:pdebug:523 [dqe:parse|<0.1371.0>|0ms] Query parsed: SELECT sum(action.count IN fifo WHERE service=sniffle) LAST 1m
2016-04-19 19:21:57.486 [debug] <0.1371.0>@dqe:pdebug:523 [dqe:parse|<0.1371.0>|0ms] Times normalized.
2016-04-19 19:21:57.486 [debug] <0.1371.0>@dqe:pdebug:523 [dqe:parse|<0.1371.0>|0ms] Aliases resolved.
2016-04-19 19:21:57.486 [debug] <0.1371.0>@dqe:pdebug:523 [dqe:parse|<0.1371.0>|0ms] Preprocessor done.
2016-04-19 19:21:57.486 [debug] <0.1371.0>@dqe:pdebug:523 [dqe:parse|<0.1371.0>|0ms] Query flattened.
2016-04-19 19:21:57.486 [debug] <0.1371.0>@dqe:pdebug:523 [dqe:parse|<0.1371.0>|0ms] Time and resolutions adjusted.
2016-04-19 19:21:57.486 [debug] <0.1371.0>@dqe:pdebug:523 [dqe:parse|<0.1371.0>|0ms] Query Translated.
2016-04-19 19:21:57.486 [debug] <0.1371.0>@dqe:pdebug:523 [dqe:prepare|<0.1371.0>|0ms] Parsing done.
2016-04-19 19:21:57.486 [debug] <0.1371.0>@dqe:pdebug:523 [dqe:prepare|<0.1371.0>|0ms] Buckets analyzed (0).
2016-04-19 19:21:57.486 [debug] <0.1371.0>@dqe:pdebug:523 [dqe:prepare|<0.1371.0>|0ms] Buckets fetched.
2016-04-19 19:21:57.506 [debug] <0.1371.0>@dqe_idx_pg:lookup:27 [dqe_idx:pg:lookup] Query too 19.212872ms: SELECT DISTINCT bucket, key FROM metrics WHERE collection = $1 AND metric = $2 AND id IN (SELECT DISTINCT metric_id FROM tags WHERE  namespace = $3 AND name = $4 AND value = $5) <- [<<"fifo">>,<<6,97,99,116,105,111,110,5,99,111,117,110,116>>,<<>>,<<"service">>,<<"sniffle">>]
2016-04-19 19:21:57.506 [debug] <0.1371.0>@dqe:pdebug:523 [dqe:prepare|<0.1371.0>|20ms] Looked up 696 metrics for {in,<<"fifo">>,[<<"action">>,<<"count">>],{'=',{tag,<<>>,<<"service">>},<<"sniffle">>}}.
2016-04-19 19:21:57.507 [debug] <0.1371.0>@dqe:pdebug:523 [dqe:prepare|<0.1371.0>|22ms] Parts expanded.
2016-04-19 19:21:57.508 [debug] <0.1371.0>@dqe:pdebug:523 [dqe:prepare|<0.1371.0>|22ms] Naing applied.
2016-04-19 19:21:57.508 [debug] <0.1371.0>@dqe:pdebug:523 [dqe:query|<0.1371.0>|22ms] preperation done.
2016-04-19 19:21:57.953 [debug] <0.1371.0>@dqe:pdebug:523 [dqe:query|<0.1371.0>|467ms] flow generated.
2016-04-19 19:21:58.033 [debug] <0.1371.0>@dqe:pdebug:523 [dqe:query|<0.1371.0>|547ms] Query complete.

After:

2016-04-19 19:27:50.603 [debug] <0.334.0>@dqe:pdebug:526 [dqe:parse|<0.334.0>|9ms] Query parsed: SELECT sum(action.count IN fifo WHERE service=sniffle) LAST 1m
2016-04-19 19:27:50.603 [debug] <0.334.0>@dqe:pdebug:526 [dqe:parse|<0.334.0>|9ms] Times normalized.
2016-04-19 19:27:50.603 [debug] <0.334.0>@dqe:pdebug:526 [dqe:parse|<0.334.0>|9ms] Aliases resolved.
2016-04-19 19:27:50.603 [debug] <0.334.0>@dqe:pdebug:526 [dqe:parse|<0.334.0>|9ms] Preprocessor done.
2016-04-19 19:27:50.603 [debug] <0.334.0>@dqe:pdebug:526 [dqe:parse|<0.334.0>|9ms] Query flattened.
2016-04-19 19:27:50.603 [debug] <0.334.0>@dqe:pdebug:526 [dqe:parse|<0.334.0>|9ms] Time and resolutions adjusted.
2016-04-19 19:27:50.603 [debug] <0.334.0>@dqe:pdebug:526 [dqe:parse|<0.334.0>|9ms] Query Translated.
2016-04-19 19:27:50.603 [debug] <0.334.0>@dqe:pdebug:526 [dqe:prepare|<0.334.0>|9ms] Parsing done.
2016-04-19 19:27:50.603 [debug] <0.334.0>@dqe:pdebug:526 [dqe:prepare|<0.334.0>|9ms] Buckets analyzed (0).
2016-04-19 19:27:50.603 [debug] <0.334.0>@dqe:pdebug:526 [dqe:prepare|<0.334.0>|9ms] Buckets fetched.
2016-04-19 19:27:50.642 [debug] <0.334.0>@dqe_idx_pg:lookup:27 [dqe_idx:pg:lookup] Query too 35.441993000000004ms: SELECT DISTINCT bucket, key FROM metrics WHERE collection = $1 AND metric = $2 AND id IN (SELECT DISTINCT metric_id FROM tags WHERE  namespace = $3 AND name = $4 AND value = $5) <- [<<"fifo">>,<<6,97,99,116,105,111,110,5,99,111,117,110,116>>,<<>>,<<"service">>,<<"sniffle">>]
2016-04-19 19:27:50.642 [debug] <0.334.0>@dqe:pdebug:526 [dqe:prepare|<0.334.0>|48ms] Looked up 696 metrics for {in,<<"fifo">>,[<<"action">>,<<"count">>],{'=',{tag,<<>>,<<"service">>},<<"sniffle">>}}.
2016-04-19 19:27:50.643 [debug] <0.334.0>@dqe:pdebug:526 [dqe:prepare|<0.334.0>|49ms] Parts expanded.
2016-04-19 19:27:50.644 [debug] <0.334.0>@dqe:pdebug:526 [dqe:prepare|<0.334.0>|50ms] Counting parts 696 total and 696 unique.
2016-04-19 19:27:50.644 [debug] <0.334.0>@dqe:pdebug:526 [dqe:prepare|<0.334.0>|50ms] Naing applied.
2016-04-19 19:27:50.644 [debug] <0.334.0>@dqe:pdebug:526 [dqe:query|<0.334.0>|50ms] preperation done.
2016-04-19 19:27:50.666 [debug] <0.334.0>@dqe:pdebug:526 [dqe:query|<0.334.0>|72ms] flow generated.
2016-04-19 19:27:50.736 [debug] <0.334.0>@dqe:pdebug:526 [dqe:query|<0.334.0>|142ms] Query complete.

[Licenser]

probably want to take total length into account for short queries (number of points per series) deduplication is a lot less important then for long queries but that needs some decent heuristics.

[Licenser]

optimizer is re-written this should solves hits.