elasticsearch2 schema refresh fails with TypeError on ES7+/8 mapping metadata keys (dynamic, dynamic_templates)

[tostavio]

Issue Summary

The elasticsearch2 query runner fails to load the schema and shows "Schema refresh failed" against Elasticsearch 7.x and 8.x whenever any index that is visible to the data source credentials carries mapping metadata keys such as dynamic or dynamic_templates at the mappings level. The data source can still run queries, because each query hits /{index}/_search directly, but the parsing of the GET /_mappings response crashes, so the autocomplete and the schema browser become unavailable for the entire data source.

Root Cause

ElasticSearch2._parse_mappings, in redash/query_runner/elasticsearch2.py, iterates over the keys found under mappings and assumes that each key is a mapping type whose value is a dictionary containing properties:

for index_name in mappings_data:
    mappings[index_name] = {}
    index_mappings = mappings_data[index_name]
    try:
        for m in index_mappings.get("mappings", {}):
            _parse_properties("", index_mappings["mappings"][m]["properties"])
    except KeyError:
        _parse_properties("", index_mappings["mappings"]["properties"])

This structure only matched Elasticsearch 6.x and earlier, where mappings was keyed by mapping types. Since mapping types were removed in Elasticsearch 7, the _mappings response places the metadata keys directly under mappings. For example:

{
  "my-index": {
    "mappings": {
      "dynamic": false,
      "dynamic_templates": [ { "strings": { "mapping": { "type": "keyword" } } } ],
      "properties": { "sku": { "type": "keyword" } }
    }
  }
}

When the loop reaches m == "dynamic", it evaluates mappings["dynamic"]["properties"], which is False["properties"], and this raises TypeError: 'bool' object is not subscriptable. The dynamic_templates key, which is a list, fails in the same way. The surrounding except KeyError does not catch TypeError, so the exception propagates and get_schema aborts for the entire data source instead of skipping the single offending key.

This situation is common in real clusters, because APM and Kibana system indices, such as .internal.alerts-*, .kibana-*, .ds-traces-apm-*, and .ds-metrics-apm-*, as well as ordinary user indices that set dynamic or dynamic_templates, all trigger it.

Steps to Reproduce

1. Point an elasticsearch2 data source at an Elasticsearch 7.x or 8.x cluster whose visible indices include at least one index that sets dynamic or dynamic_templates at the mappings level. Most APM and Kibana system indices qualify out of the box.
2. Open the query editor for that data source.
3. The schema fails to load and shows "Schema refresh failed", and the server logs show a TypeError: 'bool' object is not subscriptable raised from _parse_mappings.

You can also reproduce the parsing path directly:

from redash.query_runner.elasticsearch2 import ElasticSearch2

ElasticSearch2._parse_mappings({
    "my-index": {
        "mappings": {
            "dynamic": False,
            "dynamic_templates": [{"strings": {"mapping": {"type": "keyword"}}}],
            "properties": {"sku": {"type": "keyword"}},
        }
    }
})
# TypeError: 'bool' object is not subscriptable

Suggested Fix

Catch TypeError alongside KeyError, so that the loop falls back to reading mappings["properties"] directly, which is the correct shape for Elasticsearch 7 and 8. Queries are not affected, because only the schema parsing was broken. A pull request with a regression test follows.

Technical details

• Affected file: redash/query_runner/elasticsearch2.py, in the ElasticSearch2._parse_mappings method.
• Elasticsearch: 7.x and 8.x, reproduced against an 8.x cluster.
• Redash: reproduced on the current master branch.

Related

#5545 ("Schema refresh failed with Elasticsearch v7.x") reports the same user-facing symptom against the older elasticsearch runner. This issue is specific to the elasticsearch2 runner and to the TypeError raised by its _parse_mappings method.