We are seeing query OOM kill in thanos promql engine with code path:
vectorOperator.initJoinTables() -> vectorOperator.resultmetric() -> labels.Builder.Labels()
The large heap allocation was caused by labels of the join table output.
Example query:
sum by (namespace, label_beta_kubernetes_io_instance_type) (
kube_pod_info
* on (cluster, node) group_left(label_beta_kubernetes_io_instance_type)
max by (node, cluster, label_beta_kubernetes_io_instance_type) (kube_node_labels)
)
The query is an aggregation query with an inner join. The join is a group left join so it tries to keep full labels from LHS and some labels from RHS. In this case the LHS is a high cardinality queries with quite a few labels and keeping the full result labels result an OOM kill in this case.
For this specific query, there is an outer aggregation sum by (namespace, label_beta_kubernetes_io_instance_type) so we actually don't need to keep the full result labels in the join table. One optimization we can do is to push down the outer aggregation to the inner vector operator join to only keep useful labels. If the join table can only keep the required labels namespace and label_beta_kubernetes_io_instance_type in this case then we can save a lot of memory on heap.
Essentially, it is similar to rewriting the original query to apply the aggregation for LHS of the join to reduce join table output labels. The optimization is to do this using a logical optimizer instead of doing it manually.
sum by (namespace, label_beta_kubernetes_io_instance_type) (
sum by (cluster, node, namespace) (kube_pod_info) # see the sum by here
* on (cluster, node) group_left( label_beta_kubernetes_io_instance_type)
max by (node, cluster, label_beta_kubernetes_io_instance_type) (kube_node_labels)
)
We are seeing query OOM kill in thanos promql engine with code path:
The large heap allocation was caused by labels of the join table output.
Example query:
The query is an aggregation query with an inner join. The join is a group left join so it tries to keep full labels from LHS and some labels from RHS. In this case the LHS is a high cardinality queries with quite a few labels and keeping the full result labels result an OOM kill in this case.
For this specific query, there is an outer aggregation
sum by (namespace, label_beta_kubernetes_io_instance_type)so we actually don't need to keep the full result labels in the join table. One optimization we can do is to push down the outer aggregation to the inner vector operator join to only keep useful labels. If the join table can only keep the required labelsnamespaceandlabel_beta_kubernetes_io_instance_typein this case then we can save a lot of memory on heap.Essentially, it is similar to rewriting the original query to apply the aggregation for LHS of the join to reduce join table output labels. The optimization is to do this using a logical optimizer instead of doing it manually.