Skip to content

Releases: grafana/mimir

2.17.1

05 Sep 20:11
@zenador zenador
mimir-2.17.1
This tag was signed with the committer’s verified signature.
zenador
GPG key ID: F2D719C4BD7BDE3E
Verified
Learn about vigilant mode.
c5ebb8f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
2.17.1 Latest
Latest

This release contains 7 PRs from 5 authors. Thank you!

Changelog

2.17.1

Grafana Mimir

All changes in this release: mimir-2.17.0...mimir-2.17.1

Assets 118

2.17.0

15 Aug 17:03
@zenador zenador
mimir-2.17.0
This tag was signed with the committer’s verified signature.
zenador
GPG key ID: F2D719C4BD7BDE3E
Verified
Learn about vigilant mode.
e150acb
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
2.17.0

This release contains 735 PRs from 78 authors, including new contributors Bernaud Vincent, Carrie Edwards, danieleandreatta, David Vávra, Edgaras Giedrė, Gabija Bruzgaitė, Henrique Lourenço, Innokentii Konstantinov, Jasper Maes, Jeanette Tan, Joobi S B, Julius Hinze, Lukas Bischofberger, mihaelmiklec, mimir-vendoring[bot], Nasiel, pierremahot, rektabhi, sam clulow, Shay Pletcher, Thor K. Høgås, Toni Cárdenas, Yuran Ou, zhuoyuan-liu. Thank you!

Grafana Mimir version 2.17.0 release notes

Grafana Labs is excited to announce version 2.17 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

MQE is enabled by default in queriers. MQE provides benefits over the Prometheus engine, inluding reduced memory and CPU consumption and improved performance. To use the Prometheus engine instead of MQE, set -querier.query-engine=prometheus.

Grafana Mimir now supports using the Mimir Query Engine (MQE) in query-frontends in addition to queriers. You can enable MQE in query-frontends by setting the experimental CLI flag -query-frontend.query-engine=mimir or through the corresponding YAML option.

You can export the cortex_ingester_attributed_active_native_histogram_series and cortex_ingester_attributed_active_native_histogram_buckets native histogram cost attribution metrics to a custom Prometheus registry with user-specified labels.

Grafana Mimir supports converting OTel explicit bucket histograms to Prometheus native histograms with custom buckets using the distributor.otel-convert-histograms-to-nhcb flag.

The following experimental features have been removed:

  • The max_cost_attribution_labels_per_user cost attribution limit
  • Read-write deployment mode in the mixin

Important changes

In Grafana Mimir 2.17, the following behavior has changed:

The following default configuration values now apply to the memberlist KV store:

Key Value
memberlist.packet-dial-timeout 500ms
memberlist.packet-write-timeout 500ms
memberlist.max-concurrent-writes 5
memberlist.acquire-writer-timeout 1s

These values perform better but might cause long-running packets to be dropped in high-latency networks.

The -ruler-storage.cache.rule-group-enabled experimental CLI flag has been removed. Caching rule group contents is now always enabled when a cache is configured for the ruler.

The -ingester.ooo-native-histograms-ingestion-enabled CLI flag and corresponding ooo_native_histograms_ingestion_enabled runtime configuration option have been removed. Out-of-order native histograms are now enabled whenever both native histogram and out-of-order ingestion is enabled.

The -ingester.stream-chunks-when-using-blocks CLI flag and corresponding ingester_stream_chunks_when_using_blocks runtime configuration option have been deprecated and will be removed in a future release.

The cortex_distributor_label_values_with_newlines_total metric has been removed.

In the distributor, memberlist is marked as a stable option for backend storage for the high availability tracker. etcd has been deprecated for this purpose.

Experimental features

Grafana Mimir 2.17 includes some features that are experimental and disabled by default.
Use these features with caution and report any issues that you encounter:

  • Prometheus Remote-Write 2.0 protocol.
  • Duration expressions in PromQL. These are simple arithmetics on numbers in offset and range specification. For example, rate(http_requests_total[5m * 2]).
  • Promoting OTel scope metadata, including name, version, schema URL, and attributes, to metric labels, prefixed with otel_scope_. Enable this feature through the -distributor.otel-promote-scope-metadata flag.
  • Allowing primitive delta metrics ingestion through the OTLP endpoint with the -distributor.otel-native-delta-ingestion option.
  • Support for sort_by_label and sort_by_label_desc PromQL functions.
  • Support for cluster validation in HTTP calls. When enabled, the HTTP server verifies if a request coming from an HTTP client comes from an expected cluster. You can configure this validation with the following options:
    • -server.cluster-validation.label
    • -server.cluster-validation.http.enabled
    • -server.cluster-validation.http.soft-validation
    • -server.cluster-validation.http.exclude-paths

Bug fixes

For a detailed list of bug fixes, refer to the CHANGELOG.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is released independently.
Refer to the Grafana Mimir Helm chart documentation.

Changelog

2.17.0

Grafana Mimir

  • [CHANGE] Query-frontend: Ensure that cache keys generated from cardinality estimate middleware are less than 250 bytes in length by hashing the tenant IDs that are included in them. This change invalidates all cardinality estimates in the cache. #11568
  • [CHANGE] Ruler: Remove experimental CLI flag -ruler-storage.cache.rule-group-enabled to enable or disable caching the contents of rule groups. Caching rule group contents is now always enabled when a cache is configured for the ruler. #10949
  • [CHANGE] Ingester: Out-of-order native histograms are now enabled whenever both native histogram and out-of-order ingestion is enabled. The -ingester.ooo-native-histograms-ingestion-enabled CLI flag and corresponding ooo_native_histograms_ingestion_enabled runtime configuration option have been removed. #10956
  • [CHANGE] Distributor: removed the cortex_distributor_label_values_with_newlines_total metric. #10977
  • [CHANGE] Ingester/Distributor: renamed the experimental max_cost_attribution_cardinality_per_user config to max_cost_attribution_cardinality. #11092
  • [CHANGE] Frontend: The subquery spin-off feature is now enabled with -query-frontend.subquery-spin-off-enabled=true instead of -query-frontend.instant-queries-with-subquery-spin-off=.* #11153
  • [CHANGE] Overrides-exporter: Don't export per-tenant overrides that are set to their default values. #11173
  • [CHANGE] gRPC/HTTP clients: Rename metric cortex_client_request_invalid_cluster_validation_labels_total to cortex_client_invalid_cluster_validation_label_requests_total. #11237
  • [CHANGE] Querier: Use Mimir Query Engine (MQE) by default. Set -querier.query-engine=prometheus to continue using Prometheus' engine. #11501
  • [CHANGE] Memcached: Ignore initial DNS resolution failure, meaning don't depend on Memcached on startup. #11602
  • [CHANGE] Ingester: The -ingester.stream-chunks-when-using-blocks CLI flag and ingester_stream_chunks_when_using_blocks runtime configuration option have been deprecated and will be removed in a future release. #11711
  • [CHANGE] Distributor: track cortex_ingest_storage_writer_latency_seconds metric for failed writes too. Added outcome label to distinguish between success and failure. #11770
  • [CHANGE] Distributor: renamed few metrics used by experimental ingest storage. #11766
    • Renamed cortex_ingest_storage_writer_produce_requests_total to cortex_ingest_storage_writer_produce_records_enqueued_total
    • Renamed cortex_ingest_storage_writer_produce_failures_total to cortex_ingest_storage_writer_produce_records_failed_total
  • [CHANGE] Distributor: moved HA tracker timeout config to limits. #11774
    • Moved distributor.ha_tracker.ha_tracker_update_timeout to limits.ha_tracker_update_timeout.
    • Moved distributor.ha_tracker.ha_tracker_update_timeout_jitter_max to limits.ha_tracker_update_timeout_jitter_max.
    • Moved distributor.ha_tracker.ha_tracker_failover_timeout to limits.ha_tracker_failover_timeout.
  • [CHANGE] Distributor: Memberlist marked as stable as an option for backend storage for the HA tracker. #11861
  • [CHANGE] Distributor: etcd deprecated as an option for backend storage for the HA tracker. #12047
  • [CHANGE] Memberlist: Apply new default configuration values for MemberlistKV. This unlocks using it as backend storage for the HA Tracker. We have observed better performance with these defaults across different production loads. #11874
    • memberlist.packet-dial-timeout: 500ms
    • memberlist.packet-write-timeout: 500ms
    • memberlist.max-concurrent-writes: 5
    • memberlist.acquire-writer-timeout: 1s
      These defaults perform better but may cause long-running packets to be dropped in high-latency networks.
  • [CHANGE] Query-frontend: Apply query pruning and check for disabled experimental functions earlier in query processing. #11939
  • [FEATURE] Distributor: Experimental support for Prometheus Remote-Write 2.0 protocol. Limitations: Created timestamp is ignored, per series metadata is merged on metric family level automatically, ingestion might fail if client sends ProtoBuf fields out of order. The label version is added to the metric cortex_distributor_requests_in_total with a value of either 1.0 or 2.0 depending on the detected Remote-Write protocol. #11100 #11101 #11192 #11143
  • [FEATURE] Query-frontend: expand query-frontend.cache-errors and query-frontend.results-cache-ttl-for-errors configuration options to cache non-transient response failures for instant queries. #11120
  • [FEATURE] Query-frontend: Allow use of Mimir Query Engine (MQE) via the experimental CLI flags -query-frontend.query-engine or -query-frontend.enable-query-engine-fallback or corresponding YAML. #11417 #11775
  • [FEATURE] Querier, query-frontend, ruler: Enable experimental support for duration expressions in Pro...
Read more
Loading

2.17.0-rc.2

14 Aug 05:44
@zenador zenador
mimir-2.17.0-rc.2
This tag was signed with the committer’s verified signature.
zenador
GPG key ID: F2D719C4BD7BDE3E
Verified
Learn about vigilant mode.
1a8294e
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
2.17.0-rc.2 Pre-release
Pre-release

This release contains 7 PRs from 4 authors. Thank you!

Grafana Mimir version 2.17.0-rc.2 release notes

Grafana Labs is excited to announce version 2.17 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

MQE is enabled by default in queriers. MQE provides benefits over the Prometheus engine, inluding reduced memory and CPU consumption and improved performance. To use the Prometheus engine instead of MQE, set -querier.query-engine=prometheus.

Grafana Mimir now supports using the Mimir Query Engine (MQE) in query-frontends in addition to queriers. You can enable MQE in query-frontends by setting the experimental CLI flag -query-frontend.query-engine=mimir or through the corresponding YAML option.

You can export the cortex_ingester_attributed_active_native_histogram_series and cortex_ingester_attributed_active_native_histogram_buckets native histogram cost attribution metrics to a custom Prometheus registry with user-specified labels.

Grafana Mimir supports converting OTel explicit bucket histograms to Prometheus native histograms with custom buckets using the distributor.otel-convert-histograms-to-nhcb flag.

The following experimental features have been removed:

  • The max_cost_attribution_labels_per_user cost attribution limit
  • Read-write deployment mode in the mixin

Important changes

In Grafana Mimir 2.17, the following behavior has changed:

The following default configuration values now apply to the memberlist KV store:

Key Value
memberlist.packet-dial-timeout 500ms
memberlist.packet-write-timeout 500ms
memberlist.max-concurrent-writes 5
memberlist.acquire-writer-timeout 1s

These values perform better but might cause long-running packets to be dropped in high-latency networks.

The -ruler-storage.cache.rule-group-enabled experimental CLI flag has been removed. Caching rule group contents is now always enabled when a cache is configured for the ruler.

The -ingester.ooo-native-histograms-ingestion-enabled CLI flag and corresponding ooo_native_histograms_ingestion_enabled runtime configuration option have been removed. Out-of-order native histograms are now enabled whenever both native histogram and out-of-order ingestion is enabled.

The -ingester.stream-chunks-when-using-blocks CLI flag and corresponding ingester_stream_chunks_when_using_blocks runtime configuration option have been deprecated and will be removed in a future release.

The cortex_distributor_label_values_with_newlines_total metric has been removed.

In the distributor, memberlist is marked as a stable option for backend storage for the high availability tracker. etcd has been deprecated for this purpose.

Experimental features

Grafana Mimir 2.17 includes some features that are experimental and disabled by default.
Use these features with caution and report any issues that you encounter:

  • Prometheus Remote-Write 2.0 protocol.
  • Duration expressions in PromQL. These are simple arithmetics on numbers in offset and range specification. For example, rate(http_requests_total[5m * 2]).
  • Promoting OTel scope metadata, including name, version, schema URL, and attributes, to metric labels, prefixed with otel_scope_. Enable this feature through the -distributor.otel-promote-scope-metadata flag.
  • Allowing primitive delta metrics ingestion through the OTLP endpoint with the -distributor.otel-native-delta-ingestion option.
  • Support for sort_by_label and sort_by_label_desc PromQL functions.
  • Support for cluster validation in HTTP calls. When enabled, the HTTP server verifies if a request coming from an HTTP client comes from an expected cluster. You can configure this validation with the following options:
    • -server.cluster-validation.label
    • -server.cluster-validation.http.enabled
    • -server.cluster-validation.http.soft-validation
    • -server.cluster-validation.http.exclude-paths

Bug fixes

For a detailed list of bug fixes, refer to the CHANGELOG.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is released independently.
Refer to the Grafana Mimir Helm chart documentation.

Changelog

2.17.0-rc.2

Grafana Mimir

  • [ENHANCEMENT] Stagger head compaction intervals across zones to prevent compactions from aligning simultaneously, which could otherwise cause strong consistency queries to fail when experimental ingest storage is enabled. #12090
  • [ENHANCEMENT] Compactor: Add -compactor.update-blocks-concurrency flag to control concurrency for updating block metadata during bucket index updates, separate from deletion marker concurrency. #12117
  • [ENHANCEMENT] Query-frontend: Allow users to set the query-frontend.extra-propagated-headers flag to specify the extra headers allowed to pass through to the rest of the query path. #12174
  • [BUGFIX] Ingester: Fix issue where ingesters can exit read-only mode during idle compactions, resulting in write errors. #12128
  • [BUGFIX] otlp: Reverts #11889 which has a pooled memory re-use bug. #12266

All changes in this release: mimir-2.17.0-rc.1...mimir-2.17.0-rc.2

Loading

2.17.0-rc.1

07 Aug 14:42
@zenador zenador
mimir-2.17.0-rc.1
This tag was signed with the committer’s verified signature.
zenador
GPG key ID: F2D719C4BD7BDE3E
Verified
Learn about vigilant mode.
1169c97
Compare
Choose a tag to compare
2.17.0-rc.1 Pre-release
Pre-release

This release contains 81 PRs from 27 authors, including new contributors Edgaras Giedrė, Julius Hinze. Thank you!

Grafana Mimir version 2.17.0-rc.1 release notes

Grafana Labs is excited to announce version 2.17 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

MQE is enabled by default in queriers. MQE provides benefits over the Prometheus engine, inluding reduced memory and CPU consumption and improved performance. To use the Prometheus engine instead of MQE, set -querier.query-engine=prometheus.

Grafana Mimir now supports using the Mimir Query Engine (MQE) in query-frontends in addition to queriers. You can enable MQE in query-frontends by setting the experimental CLI flag -query-frontend.query-engine=mimir or through the corresponding YAML option.

You can export the cortex_ingester_attributed_active_native_histogram_series and cortex_ingester_attributed_active_native_histogram_buckets native histogram cost attribution metrics to a custom Prometheus registry with user-specified labels.

Grafana Mimir supports converting OTel explicit bucket histograms to Prometheus native histograms with custom buckets using the distributor.otel-convert-histograms-to-nhcb flag.

The following experimental features have been removed:

  • The max_cost_attribution_labels_per_user cost attribution limit
  • Read-write deployment mode in the mixin

Important changes

In Grafana Mimir 2.17, the following behavior has changed:

The following default configuration values now apply to the memberlist KV store:

Key Value
memberlist.packet-dial-timeout 500ms
memberlist.packet-write-timeout 500ms
memberlist.max-concurrent-writes 5
memberlist.acquire-writer-timeout 1s

These values perform better but might cause long-running packets to be dropped in high-latency networks.

The -ruler-storage.cache.rule-group-enabled experimental CLI flag has been removed. Caching rule group contents is now always enabled when a cache is configured for the ruler.

The -ingester.ooo-native-histograms-ingestion-enabled CLI flag and corresponding ooo_native_histograms_ingestion_enabled runtime configuration option have been removed. Out-of-order native histograms are now enabled whenever both native histogram and out-of-order ingestion is enabled.

The -ingester.stream-chunks-when-using-blocks CLI flag and corresponding ingester_stream_chunks_when_using_blocks runtime configuration option have been deprecated and will be removed in a future release.

The cortex_distributor_label_values_with_newlines_total metric has been removed.

In the distributor, memberlist is marked as a stable option for backend storage for the high availability tracker. etcd has been deprecated for this purpose.

Experimental features

Grafana Mimir 2.17 includes some features that are experimental and disabled by default.
Use these features with caution and report any issues that you encounter:

  • Prometheus Remote-Write 2.0 protocol.
  • Duration expressions in PromQL. These are simple arithmetics on numbers in offset and range specification. For example, rate(http_requests_total[5m * 2]).
  • Promoting OTel scope metadata, including name, version, schema URL, and attributes, to metric labels, prefixed with otel_scope_. Enable this feature through the -distributor.otel-promote-scope-metadata flag.
  • Allowing primitive delta metrics ingestion through the OTLP endpoint with the -distributor.otel-native-delta-ingestion option.
  • Support for sort_by_label and sort_by_label_desc PromQL functions.
  • Support for cluster validation in HTTP calls. When enabled, the HTTP server verifies if a request coming from an HTTP client comes from an expected cluster. You can configure this validation with the following options:
    • -server.cluster-validation.label
    • -server.cluster-validation.http.enabled
    • -server.cluster-validation.http.soft-validation
    • -server.cluster-validation.http.exclude-paths

Bug fixes

For a detailed list of bug fixes, refer to the CHANGELOG.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is released independently.
Refer to the Grafana Mimir Helm chart documentation.

Changelog

2.17.0-rc.1

Grafana Mimir

  • [FEATURE] Distributor: Add experimental -distributor.otel-native-delta-ingestion option to allow primitive delta metrics ingestion via the OTLP endpoint. #11631
  • [FEATURE] MQE: Add support for experimental sort_by_label and sort_by_label_desc PromQL functions. #11930
  • [FEATURE] Ingester/Block-builder: Handle the created timestamp field for remote-write requests. #11977
  • [FEATURE] Cost attribution: Labels specified in the limit configuration may specify an output label in order to override emitted label names. #12035
  • [ENHANCEMENT] Ingester: Display user grace interval in the tenant list obtained through the /ingester/tenants endpoint. #11961
  • [ENHANCEMENT] kafkatool: add consumer-group delete-offset command as a way to delete the committed offset for a consumer group. #11988
  • [ENHANCEMENT] Block-builder-scheduler: Detect gaps in scheduled and completed jobs. #11867
  • [ENHANCEMENT] Distributor: Experimental support for Prometheus Remote-Write 2.0 protocol has been updated. Created timestamps are now supported. This feature includes some limitations. If samples in a write request aren't ordered by time, the created timestamp might be dropped. Additionally, per-series metadata is automatically merged on the metric family level. Ingestion might fail if the client sends ProtoBuf fields out-of-order. The label version is added to the metric cortex_distributor_requests_in_total with a value of either 1.0 or 2.0, depending on the detected remote-write protocol. #11977
  • [ENHANCEMENT] Query-frontend: Added labels query optimizer that automatically removes redundant __name__!="" matchers from label names and label values queries, improving query performance. You can enable the optimizer per-tenant with the labels_query_optimizer_enabled runtime configuration flag. #12054 #12066 #12076 #12080
  • [ENHANCEMENT] Query-frontend: Standardise non-regex patterns in query blocking upon loading of config. #12102
  • [ENHANCEMENT] Ruler: Propagate GCS object mutation rate limit for rule group uploads. #12086
  • [BUGFIX] Distributor: Validate the RW2 symbols field and reject invalid requests that don't have an empty string as the first symbol. #11953
  • [BUGFIX] Distributor: Check max_inflight_push_requests_bytes before decompressing incoming requests. #11967
  • [BUGFIX] Query-frontend: Allow limit parameter to be 0 in label queries to explicitly request unlimited results. #12054
  • [BUGFIX] Distributor: Fix a possible panic in the OTLP push path while handling a gRPC status error. #12072
  • [BUGFIX] Query-frontend: Evaluate experimental duration expressions before sharding, splitting, and caching. Otherwise, the result is not correct. #12038
  • [BUGFIX] Block-builder-scheduler: Fix bugs in handling of partitions with no commit. #12130

Mixin

  • [CHANGE] Remove support for the experimental read-write deployment mode. #11975
  • [CHANGE] Alerts: Replace namespace with job label in golang_alerts. #11957
  • [FEATURE] Add an alert if the block-builder-scheduler detects that it has skipped data. #12118

Jsonnet

  • [CHANGE] Removed support for the experimental read-write deployment mode. #11974
  • [ENHANCEMENT] Add assertion to ensure ingester ScaledObject has minimum and maximum replicas set to a value greater than 0. #11979
  • [ENHANCEMENT] Add ingest_storage_migration_ignore_ingest_storage_errors and ingest_storage_migration_ingest_storage_max_wait_time configs to control error handling of the partition ingesters during ingest storage migrations. #12105
  • [ENHANCEMENT] Add block-builder job processing duration timings and offset-skipped errors to the Block-builder dashboard. #12118

Documentation

  • [ENHANCEMENT] Update the MimirIngestedDataTooFarInTheFuture runbook with a note about false positives and the endpoint to flush TSDB blocks by user. #11961

All changes in this release: mimir-2.17.0-rc.0...mimir-2.17.0-rc.1

Loading

2.17.0-rc.0

16 Jul 14:08
@zenador zenador
mimir-2.17.0-rc.0
0bef180
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
2.17.0-rc.0 Pre-release
Pre-release

This release contains 646 PRs from 75 authors, including new contributors Bernaud Vincent, Carrie Edwards, danieleandreatta, David Vávra, Gabija Bruzgaitė, Henrique Lourenço, Innokentii Konstantinov, Jasper Maes, Jeanette Tan, Joobi S B, Lukas Bischofberger, mihaelmiklec, mimir-vendoring[bot], Nasiel, pierremahot, rektabhi, sam clulow, Shay Pletcher, Thor K. Høgås, Toni Cárdenas, Yuran Ou, zhuoyuan-liu. Thank you!

Grafana Mimir version 2.17.0-rc.0 release notes

Grafana Labs is excited to announce version 2.17 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

MQE is enabled by default in queriers. MQE provides benefits over the Prometheus engine, inluding reduced memory and CPU consumption and improved performance. To use the Prometheus engine instead of MQE, set -querier.query-engine=prometheus.

Grafana Mimir now supports using the Mimir Query Engine (MQE) in query-frontends in addition to queriers. You can enable MQE in query-frontends by setting the experimental CLI flag -query-frontend.query-engine=mimir or through the corresponding YAML option.

You can export the cortex_ingester_attributed_active_native_histogram_series and cortex_ingester_attributed_active_native_histogram_buckets native histogram cost attribution metrics to a custom Prometheus registry with user-specified labels.

Grafana Mimir supports converting OTel explicit bucket histograms to Prometheus native histograms with custom buckets using the distributor.otel-convert-histograms-to-nhcb flag.

The following experimental features have been removed:

  • The max_cost_attribution_labels_per_user cost attribution limit
  • Read-write deployment mode in the mixin

Important changes

In Grafana Mimir 2.17, the following behavior has changed:

The following default configuration values now apply to the memberlist KV store:

Key Value
memberlist.packet-dial-timeout 500ms
memberlist.packet-write-timeout 500ms
memberlist.max-concurrent-writes 5
memberlist.acquire-writer-timeout 1s

These values perform better but might cause long-running packets to be dropped in high-latency networks.

The -ruler-storage.cache.rule-group-enabled experimental CLI flag has been removed. Caching rule group contents is now always enabled when a cache is configured for the ruler.

The -ingester.ooo-native-histograms-ingestion-enabled CLI flag and corresponding ooo_native_histograms_ingestion_enabled runtime configuration option have been removed. Out-of-order native histograms are now enabled whenever both native histogram and out-of-order ingestion is enabled.

The -ingester.stream-chunks-when-using-blocks CLI flag and corresponding ingester_stream_chunks_when_using_blocks runtime configuration option have been deprecated and will be removed in a future release.

The cortex_distributor_label_values_with_newlines_total metric has been removed.

In the distributor, memberlist is marked as a stable option for backend storage for the high availability tracker. etcd has been deprecated for this purpose.

Experimental features

Grafana Mimir 2.17 includes some features that are experimental and disabled by default.
Use these features with caution and report any issues that you encounter:

  • Prometheus Remote-Write 2.0 protocol.
  • Duration expressions in PromQL. These are simple arithmetics on numbers in offset and range specification. For example, rate(http_requests_total[5m * 2]).
  • Promoting OTel scope metadata, including name, version, schema URL, and attributes, to metric labels, prefixed with otel_scope_. Enable this feature through the -distributor.otel-promote-scope-metadata flag.
  • Allowing primitive delta metrics ingestion through the OTLP endpoint with the -distributor.otel-native-delta-ingestion option.
  • Support for sort_by_label and sort_by_label_desc PromQL functions.
  • Support for cluster validation in HTTP calls. When enabled, the HTTP server verifies if a request coming from an HTTP client comes from an expected cluster. You can configure this validation with the following options:
    • -server.cluster-validation.label
    • -server.cluster-validation.http.enabled
    • -server.cluster-validation.http.soft-validation
    • -server.cluster-validation.http.exclude-paths

Bug fixes

For a detailed list of bug fixes, refer to the CHANGELOG.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is released independently.
Refer to the Grafana Mimir Helm chart documentation.

Changelog

2.17.0-rc.0

Grafana Mimir

  • [CHANGE] Query-frontend: Ensure that cache keys generated from cardinality estimate middleware are less than 250 bytes in length by hashing the tenant IDs that are included in them. This change invalidates all cardinality estimates in the cache. #11568
  • [CHANGE] Ruler: Remove experimental CLI flag -ruler-storage.cache.rule-group-enabled to enable or disable caching the contents of rule groups. Caching rule group contents is now always enabled when a cache is configured for the ruler. #10949
  • [CHANGE] Ingester: Out-of-order native histograms are now enabled whenever both native histogram and out-of-order ingestion is enabled. The -ingester.ooo-native-histograms-ingestion-enabled CLI flag and corresponding ooo_native_histograms_ingestion_enabled runtime configuration option have been removed. #10956
  • [CHANGE] Distributor: removed the cortex_distributor_label_values_with_newlines_total metric. #10977
  • [CHANGE] Ingester/Distributor: renamed the experimental max_cost_attribution_cardinality_per_user config to max_cost_attribution_cardinality. #11092
  • [CHANGE] Frontend: The subquery spin-off feature is now enabled with -query-frontend.subquery-spin-off-enabled=true instead of -query-frontend.instant-queries-with-subquery-spin-off=.* #11153
  • [CHANGE] Overrides-exporter: Don't export per-tenant overrides that are set to their default values. #11173
  • [CHANGE] gRPC/HTTP clients: Rename metric cortex_client_request_invalid_cluster_validation_labels_total to cortex_client_invalid_cluster_validation_label_requests_total. #11237
  • [CHANGE] Querier: Use Mimir Query Engine (MQE) by default. Set -querier.query-engine=prometheus to continue using Prometheus' engine. #11501
  • [CHANGE] Memcached: Ignore initial DNS resolution failure, meaning don't depend on Memcached on startup. #11602
  • [CHANGE] Ingester: The -ingester.stream-chunks-when-using-blocks CLI flag and ingester_stream_chunks_when_using_blocks runtime configuration option have been deprecated and will be removed in a future release. #11711
  • [CHANGE] Distributor: track cortex_ingest_storage_writer_latency_seconds metric for failed writes too. Added outcome label to distinguish between success and failure. #11770
  • [CHANGE] Distributor: renamed few metrics used by experimental ingest storage. #11766
    • Renamed cortex_ingest_storage_writer_produce_requests_total to cortex_ingest_storage_writer_produce_records_enqueued_total
    • Renamed cortex_ingest_storage_writer_produce_failures_total to cortex_ingest_storage_writer_produce_records_failed_total
  • [CHANGE] Distributor: moved HA tracker timeout config to limits. #11774
    • Moved distributor.ha_tracker.ha_tracker_update_timeout to limits.ha_tracker_update_timeout.
    • Moved distributor.ha_tracker.ha_tracker_update_timeout_jitter_max to limits.ha_tracker_update_timeout_jitter_max.
    • Moved distributor.ha_tracker.ha_tracker_failover_timeout to limits.ha_tracker_failover_timeout.
  • [CHANGE] Distributor: Memberlist marked as stable as an option for backend storage for the HA tracker. #11861
  • [CHANGE] Distributor: etcd deprecated as an option for backend storage for the HA tracker. #12047
  • [CHANGE] Memberlist: Apply new default configuration values for MemberlistKV. This unlocks using it as backend storage for the HA Tracker. We have observed better performance with these defaults across different production loads. #11874
    • memberlist.packet-dial-timeout: 500ms
    • memberlist.packet-write-timeout: 500ms
    • memberlist.max-concurrent-writes: 5
    • memberlist.acquire-writer-timeout: 1s
      These defaults perform better but may cause long-running packets to be dropped in high-latency networks.
  • [CHANGE] Query-frontend: Apply query pruning and check for disabled experimental functions earlier in query processing. #11939
  • [FEATURE] Distributor: Experimental support for Prometheus Remote-Write 2.0 protocol. Limitations: Created timestamp is ignored, per series metadata is merged on metric family level automatically, ingestion might fail if client sends ProtoBuf fields out of order. The label version is added to the metric cortex_distributor_requests_in_total with a value of either 1.0 or 2.0 depending on the detected Remote-Write protocol. #11100 #11101 #11192 #11143
  • [FEATURE] Query-frontend: expand query-frontend.cache-errors and query-frontend.results-cache-ttl-for-errors configuration options to cache non-transient response failures for instant queries. #11120
  • [FEATURE] Query-frontend: Allow use of Mimir Query Engine (MQE) via the experimental CLI flags -query-frontend.query-engine or -query-frontend.enable-query-engine-fallback or corresponding YAML. #11417 #11775
  • [FEATURE] Querier, query-frontend, ruler: Enable experimental support for duration expressions in PromQ...
Read more
Loading

2.16.1

24 Jun 23:34
@56quarters 56quarters
mimir-2.16.1
This tag was signed with the committer’s verified signature.
56quarters Nick Pillitteri
GPG key ID: E7B3A3C53C87A973
Verified
Learn about vigilant mode.
876d470
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
2.16.1

This release contains 14 PRs from 8 authors. Thank you!

Changelog

2.16.1

Grafana Mimir

  • [BUGFIX] Update to Go v1.23.9 to address CVE-2025-22871. #11543
  • [BUGFIX] Update golang.org/x/net to v0.38.0 to address CVE-2025-22872. #11281
  • [BUGFIX] Query-frontend: Fix a panic in monolithic mode caused by a clash in labels of the cortex_client_invalid_cluster_validation_label_requests_total metric definition. #11455

All changes in this release: mimir-2.16.0...mimir-2.16.1

Loading

2.15.3

05 Jun 21:24
@chencs chencs
mimir-2.15.3
250a0bb
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
2.15.3

This release contains 10 PRs from 5 authors, including new contributors mimir-github-bot[bot]. Thank you!

Changelog

2.15.3

Grafana Mimir

Mimirtool

Mimir Continuous Test

All changes in this release: mimir-2.15.2...mimir-2.15.3

Loading

2.15.2

09 Apr 23:34
@chencs chencs
mimir-2.15.2
dff5c51
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
2.15.2

This release contains 4 PRs from 3 authors. Thank you!

Changelog

2.15.2

Grafana Mimir

All changes in this release: mimir-2.15.1...mimir-2.15.2

Loading

2.16.0

31 Mar 15:26
@56quarters 56quarters
mimir-2.16.0
This tag was signed with the committer’s verified signature.
56quarters Nick Pillitteri
GPG key ID: E7B3A3C53C87A973
Verified
Learn about vigilant mode.
b4f36da
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
2.16.0

This release contains 463 PRs from 69 authors, including new contributors Alessandro Verzicco, Alex Greenbank, André Pires, Bjorn Stout, Bruno FERNANDO, Casie Chen, Dustin Wilson, Edwin Tye, Kenny Trytek, Leszek Błażewski, Markus Opolka, Matthew Jacobson, Matt Veitas, mimir-github-bot[bot], Moustafa Baiou, Ryan Brady, TheRealNoob. Thank you!

Grafana Mimir version 2.16.0 release notes

Features and enhancements

In rulers, when rule concurrency is enabled for a rule group, its rules will now be reordered and run in batches based on their dependencies. This increases the number of rules that can potentially run concurrently. Note that the global and tenant-specific limits around the number of rule groups and rules per group still apply.

Using mimirtool to analyze Grafana dashboards now supports bar chart, pie chart, state timeline, status history, histogram, candlestick, canvas, flame graph, geomap, node graph, trend, and XY chart panels.

Important changes

In Grafana Mimir 2.16, the following behavior has changed:

Grafana Mimir only provides container images based on distroless images. Alpine Linux-based container images were deprecated in the 2.12 release and are no longer built.

How experimental PromQL functions are enabled has changed.

  • The experimental CLI flags -querier.promql-experimental-functions-enabled and -query-frontend.block-promql-experimental-functions and respective YAML configuration have been removed from query-frontends and queriers.
  • Experimental PromQL functions are disabled by default but can be enabled using only the per-tenant setting enabled_promql_experimental_functions.

Support for native histograms and out-of-order native histograms is enabled by default in ingesters.

Distributors discard float and histogram samples with duplicated timestamps from each timeseries in a request before the request is forwarded to ingesters. Discarded samples are tracked by cortex_discarded_samples_total metrics with the reason sample_duplicate_timestamp.

Experimental features

Grafana Mimir 2.16 includes some features that are experimental and disabled by default. Use these features with caution and report any issues that you encounter:

Distributors now include experimental support for the Influx line protocol.

Query-frontends now include experimental support to "spin off" subqueries as actual range queries, so that they benefit from query acceleration techniques such as sharding, splitting, and caching.

Bug fixes

  • Distributor: Use a boolean to track changes while merging the ReplicaDesc components, rather than comparing the objects directly. #10185
  • Querier: Fix timeout responding to the query-frontend when the response size is within a few hundred bytes of -querier.frontend-client.grpc-max-send-msg-size. #10154
  • Query-frontend and querier: Show warning and info annotations in some cases where they were missing (if a lazy querier was used). #10277
  • Query-frontend: Fix an issue where transient errors are inadvertently cached. #10537 #10631
  • Ruler: Fix indeterminate rules always, instead of never, running concurrently when -ruler.max-independent-rule-evaluation-concurrency is set. prometheus/prometheus#15560 #10258
  • PromQL: Fix various UTF-8 bugs related to quoting. prometheus/prometheus#15531 #10258
  • Ruler: Fix an issue when using the experimental -ruler.max-independent-rule-evaluation-concurrency feature, where if a rule group was eligible for concurrency, it would flap between running concurrently or not based on the time it took after running concurrently. #9726 #10189
  • Mimirtool: remote-read commands now return data. #10286
  • PromQL: Fix deriv, predict_linear and double_exponential_smoothing with histograms prometheus/prometheus#15686 #10383
  • MQE: Fix deriv with histograms. #10383
  • PromQL: Fix <aggr_over_time> functions with histograms. prometheus/prometheus#15711 #10400
  • MQE: Fix <aggr_over_time> functions with histograms. #10400
  • Distributor: Return HTTP status 415, Unsupported Media Type, instead of 200, Success, for Remote Write 2.0 until we support it. #10423 #10916
  • Query-frontend: Add -query-frontend.prom2-range-compat flag and corresponding YAML to rewrite queries with ranges that worked in Prometheus 2 but are invalid in Prometheus 3. #10445 #10461 #10502
  • Distributor: Fix edge case at the HA-tracker with memberlist as KVStore, where when a replica in the KVStore is marked as deleted but not yet removed, it fails to update the KVStore. #10443
  • Distributor: Fix panics in DurationWithJitter util functions when computed variance is zero. #10507
  • Ingester: Fixed a race condition in the PostingsForMatchers cache that may have infrequently returned expired cached postings. #10500
  • Distributor: Report partially converted OTLP requests with status 400, Bad Request. #10588
  • Ruler: Fix issue where rule evaluations could be missed while shutting down a ruler instance if that instance owns many rule groups. prometheus/prometheus#15804 #10762
  • Ingester: Add additional check on reactive limiter queue sizes. #10722
  • TSDB: Fix unknown series errors and possible lost data during WAL replay when series are removed from the head due to inactivity and reappear before the next WAL checkpoint. prometheus/prometheus#16060 #10824
  • Querier: Fix issue where label_join could incorrectly return multiple series with the same labels rather than failing with vector cannot contain metrics with the same labelset. prometheus/prometheus#15975 #10826
  • Querier: Fix issue where counter resets on native histograms could be incorrectly under or over-counted when using subqueries. prometheus/prometheus#15987 #10871
  • Ingester: Fix goroutines and memory leak when experimental ingest storage is enabled and a server-side error occurs during metrics ingestion. #10915
  • Mimirtool: Fix issue where MIMIR_HTTP_PREFIX environment variable was ignored and the value from MIMIR_MIMIR_HTTP_PREFIX was used instead. #10207

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is released independently.
Refer to the Grafana Mimir Helm chart documentation.

Changelog

2.16.0

Grafana Mimir

  • [CHANGE] Querier: pass context to queryable IsApplicable hook. #10451
  • [CHANGE] Distributor: OTLP and push handler replace all non-UTF8 characters with the unicode replacement character \uFFFD in error messages before propagating them. #10236
  • [CHANGE] Querier: pass query matchers to queryable IsApplicable hook. #10256
  • [CHANGE] Build: removed Mimir Alpine Docker image and related CI tests. #10469
  • [CHANGE] Query-frontend: Add topic label to cortex_ingest_storage_strong_consistency_requests_total, cortex_ingest_storage_strong_consistency_failures_total, and cortex_ingest_storage_strong_consistency_wait_duration_seconds metrics. #10220
  • [CHANGE] Ruler: cap the rate of retries for remote query evaluation to 170/sec. This is configurable via -ruler.query-frontend.max-retries-rate. #10375 #10403
  • [CHANGE] Query-frontend: Add topic label to cortex_ingest_storage_reader_last_produced_offset_requests_total, cortex_ingest_storage_reader_last_produced_offset_failures_total, cortex_ingest_storage_reader_last_produced_offset_request_duration_seconds, cortex_ingest_storage_reader_partition_start_offset_requests_total, cortex_ingest_storage_reader_partition_start_offset_failures_total, cortex_ingest_storage_reader_partition_start_offset_request_duration_seconds metrics. #10462
  • [CHANGE] Ingester: Set -ingester.ooo-native-histograms-ingestion-enabled to true by default. #10483
  • [CHANGE] Ruler: Add user and reason labels to cortex_ruler_write_requests_failed_total and cortex_ruler_queries_failed_total; add user to
    cortex_ruler_write_requests_total and cortex_ruler_queries_total metrics. #10536
  • [CHANGE] Querier / Query-frontend: Remove experimental -querier.promql-experimental-functions-enabled and -query-frontend.block-promql-experimental-functions CLI flags and respective YAML configuration options to enable experimental PromQL functions. Instead access to experimental PromQL functions is always blocked. You can enable them using the per-tenant setting enabled_promql_experimental_functions. #10660 #10712
  • [CHANGE] Store-gateway: Include posting sampling rate in sparse index headers. When the sampling rate isn't set in a sparse index header, store gateway rebuilds the sparse header with the configured blocks-storage.bucket-store.posting-offsets-in-mem-sampling value. If the sparse header's sampling rate is set but doesn't match the configured rate, store gateway either rebuilds the sparse header or downsamples to the configured sampling rate. #10684 #10878
  • [CHANGE] Distributor: Return specific error message when burst size limit is exceeded. #10835
  • [CHANGE] Ingester: enable native histograms ingestion by default, meaningingester.native-histograms-ingestion-enabled defaults to true. #10867
  • [FEATURE] Ingester/Distributor: Add support for exporting cost attribution metrics (cortex_ingester_attributed_active_series, cortex_distributor_received_attributed_samples_total, and cortex_discarded_attributed_samples_total) with labels specified by customers to a custom Prometheus registry. This feature enables more flexible billing data tracking. #10269 #10702
  • [FEATURE] Ruler: Added /ruler/tenants endpoints to list the discovered tenants with rule groups. #10738
  • [FEATURE] Distributor: Add experimental Influx handler. #10153
  • [ENHANCEMENT]...
Read more
Loading

2.15.1

13 Mar 22:12
@chencs chencs
mimir-2.15.1
0c1e06f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
2.15.1

This release contains 8 PRs from 3 authors. Thank you!

Changelog

2.15.1

Grafana Mimir

All changes in this release: mimir-2.15.0...mimir-2.15.1

Loading