Build Error <mock requires Python '>=3.6' but the running Python is 2.7.15>

[bcisse]

Hi,

I'm having trouble building the elastalert docker container on Ubuntu 18.04 using the instructions at https://github.com/bitsensor/elastalert. Initial builds failed with same error message mentioned in #147 but even after modifying the dockerfile to have "FROM alpine:3.8 as py-ea" instead of "FROM alpine:latest as py-ea" as recommended, builds are still failing but with a different error:

mock requires Python '>=3.6' but the running Python is 2.7.15
You are using pip version 10.0.1, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The command '/bin/sh -c sed -i 's/jira>=1.0.10/jira>=1.0.10,<1.0.15/g' setup.py && python setup.py install && pip install -r requirements.txt' returned a non-zero code: 1
Makefile:6: recipe for target 'build' failed
make: *** [build] Error 1

Any help or insight on how to resolve this will be greatly appreciated.

[Renny8]

Having the same issue myself on Ubuntu 18.04 after making the change "FROM alpine:latest as py-ea -> FROM alpine:3.8 as py-ea"

Did you have any luck resolving @bcisse ?

[bcisse]

I was able to build version v0.2.1 of the elastalert docker image by modifying the dockerfile to install python3 as follows:

*****Dockerfile *****
FROM alpine:latest as py-ea
ARG ELASTALERT_VERSION=v0.2.1
...
CHANGE:
RUN apk add --update --no-cache ca-certificates openssl-dev openssl python2-dev python2 py2-pip py2-yaml libffi-dev gcc musl-dev wget && \

TO:
RUN apk add --update --no-cache ca-certificates openssl-dev openssl python3-dev python3 py3-pip py3-yaml libffi-dev gcc musl-dev wget &&
...
CHANGE:
COPY --from=py-ea /usr/lib/python2.7/site-packages /usr/lib/python2.7/site-packages

TO:
COPY --from=py-ea /usr/lib/python3.8/site-packages /usr/lib/python3.8/site-packages

RUN sed -i 's/jira>=1.0.10/jira>=1.0.10,<1.0.15/g' setup.py &&
python3 setup.py install &&
pip3 install -r requirements.txt

When I try to start a container using the image, it fails however after a few seconds. The elastalert indices are created in elasticsearch but container fails shortly after with following error:

16:37:18.170Z INFO elastalert-server: Config: No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
16:37:18.173Z INFO elastalert-server: Config: Proceeding to look for normal config file.
16:37:18.174Z INFO elastalert-server: Config: A config file was found in /opt/elastalert-server/config/config.json. Using that config.
16:37:18.185Z INFO elastalert-server: Router: Listening for GET request on /.
16:37:18.186Z INFO elastalert-server: Router: Listening for GET request on /status.
16:37:18.186Z INFO elastalert-server: Router: Listening for GET request on /status/control/:action.
16:37:18.186Z INFO elastalert-server: Router: Listening for GET request on /status/errors.
16:37:18.186Z INFO elastalert-server: Router: Listening for GET request on /rules.
16:37:18.188Z INFO elastalert-server: Router: Listening for GET request on /rules/:id.
16:37:18.189Z INFO elastalert-server: Router: Listening for POST request on /rules/:id.
16:37:18.189Z INFO elastalert-server: Router: Listening for DELETE request on /rules/:id.
16:37:18.189Z INFO elastalert-server: Router: Listening for GET request on /templates.
16:37:18.189Z INFO elastalert-server: Router: Listening for GET request on /templates/:id.
16:37:18.189Z INFO elastalert-server: Router: Listening for POST request on /templates/:id.
16:37:18.193Z INFO elastalert-server: Router: Listening for DELETE request on /templates/:id.
16:37:18.194Z INFO elastalert-server: Router: Listening for POST request on /test.
16:37:18.194Z INFO elastalert-server: Router: Listening for GET request on /config.
16:37:18.194Z INFO elastalert-server: Router: Listening for POST request on /config.
16:37:18.194Z INFO elastalert-server: Router: Listening for POST request on /download.
16:37:18.194Z INFO elastalert-server: Router: Listening for GET request on /metadata/:type.
16:37:18.195Z INFO elastalert-server: Router: Listening for GET request on /mapping/:index.
16:37:18.195Z INFO elastalert-server: Router: Listening for POST request on /search/:index.
16:37:18.199Z INFO elastalert-server: ProcessController: Starting ElastAlert
16:37:18.199Z INFO elastalert-server: ProcessController: Creating index
16:37:21.448Z INFO elastalert-server:
ProcessController: Elastic Version:7
Mapping used for string:{'type': 'keyword'}

16:37:21.449Z ERROR elastalert-server:
ProcessController: Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/opt/elastalert/elastalert/create_index.py", line 275, in
main()
File "/opt/elastalert/elastalert/create_index.py", line 252, in main
es.indices.put_mapping(index=index, doc_type='elastalert', body=es_mapping)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/indices.py", line 339, in put_mapping
"PUT", _make_path(index, "_mapping", doc_type), params=params, body=body
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/transport.py", line 353, in perform_request
timeout=timeout,
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 143, in perform_request
self._raise_error(response.status_code, raw_data)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/base.py", line 168, in _raise_error
status_code, error_message, additional_info
elasticsearch.exceptions.RequestError: RequestError(400, u'illegal_argument_exception', u'Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true.')

16:37:21.449Z ERROR elastalert-server: ProcessController: Index create exited with code 1
16:37:21.449Z WARN elastalert-server: ProcessController: ElastAlert will start but might not be able to save its data!
16:37:21.450Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none]
16:37:21.459Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 36)
16:37:21.461Z INFO elastalert-server: Server: Server listening on port 3030
16:37:21.462Z INFO elastalert-server: Server: Websocket listening on port 3333
16:37:21.462Z INFO elastalert-server: Server: Server started
16:37:22.338Z ERROR elastalert-server:
ProcessController: Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main

16:37:22.338Z ERROR elastalert-server:
ProcessController: "main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/opt/elastalert/elastalert/elastalert.py", line 1929, in
sys.exit(main(sys.argv[1:]))
File "/opt/elastalert/elastalert/elastalert.py", line 1925, in main

16:37:22.338Z ERROR elastalert-server:
ProcessController: client.start()
File "/opt/elastalert/elastalert/elastalert.py", line 1106, in start

16:37:22.338Z ERROR elastalert-server:
ProcessController: self.run_all_rules()
File "/opt/elastalert/elastalert/elastalert.py", line 1158, in run_all_rules

16:37:22.339Z ERROR elastalert-server:
ProcessController: self.send_pending_alerts()
File "/opt/elastalert/elastalert/elastalert.py", line 1534, in send_pending_alerts

16:37:22.339Z ERROR elastalert-server:
ProcessController: pending_alerts = self.find_recent_pending_alerts(self.alert_time_limit)
File "/opt/elastalert/elastalert/elastalert.py", line 1526, in find_recent_pending_alerts

16:37:22.340Z ERROR elastalert-server:
ProcessController: size=1000)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped

16:37:22.340Z ERROR elastalert-server:
ProcessController: return func(*args, params=params, **kwargs)
TypeError: search() got an unexpected keyword argument 'doc_type'

16:37:22.384Z ERROR elastalert-server: ProcessController: ElastAlert exited with code 1
16:37:22.384Z INFO elastalert-server: Server: Stopping server
16:37:22.384Z INFO elastalert-server: ProcessController: ElastAlert is not running
16:37:22.384Z INFO elastalert-server: Server: Server stopped. Bye!

[andrewbierbaum]

In my testing I saw this error a few times. fwiw, I have been using #157 and it works great on python3. I would also prehaps prefer a base ubuntu. See the commits for a possible solution

[jibsonline]

For those who want to stick to python 2.7 the below works

FROM alpine:3.8 as py-ea
ARG ELASTALERT_VERSION=v0.1.39
ENV ELASTALERT_VERSION=${ELASTALERT_VERSION}
# URL from which to download Elastalert.
ARG ELASTALERT_URL=https://github.com/Yelp/elastalert/archive/$ELASTALERT_VERSION.zip
ENV ELASTALERT_URL=${ELASTALERT_URL}
# Elastalert home directory full path.
ENV ELASTALERT_HOME /opt/elastalert

WORKDIR /opt

RUN apk add --update --no-cache ca-certificates openssl-dev openssl python2-dev python2 py2-pip py2-yaml libffi-dev gcc musl-dev wget && \
# Download and unpack Elastalert.
    wget -O elastalert.zip "${ELASTALERT_URL}" && \
    unzip elastalert.zip && \
    rm elastalert.zip && \
    mv e* "${ELASTALERT_HOME}"

WORKDIR "${ELASTALERT_HOME}"

# Install Elastalert.
# see: https://github.com/Yelp/elastalert/issues/1654
RUN sed -i 's/jira>=1.0.10/jira>=1.0.10,<1.0.15/g' setup.py && sed -i 's/mock>=2.0.0/mock==3.0.5/g' setup.py  && \
    python setup.py install && \
    pip install apscheduler>=3.3.0 aws-requests-auth>=0.3.0 blist>=1.3.6 boto3>=1.4.4 cffi>=1.11.5 configparser>=3.5.0 croniter>=0.3.16 elasticsearch>=7.0.0 envparse>=0.2.0 exotel>=0.1.3 "jira>=1.0.10,<1.0.15" jsonschema>=3.0.2 mock==3.0.5 prison>=0.1.2 py-zabbix==1.1.3 PyStaticConfiguration>=0.10.3 "python-dateutil>=2.6.0,<2.7.0" python-magic>=0.4.15 PyYAML>=5.1 requests>=2.0.0 stomp.py==4.1.23 texttable>=0.8.8 thehive4py>=1.4.4 twilio==6.0.0

FROM node:alpine
LABEL maintainer="BitSensor <dev@bitsensor.io>"
# Set timezone for this container
ENV TZ Etc/UTC

RUN apk add --update --no-cache curl tzdata python2 make libmagic

COPY --from=py-ea /usr/lib/python2.7/site-packages /usr/lib/python2.7/site-packages
COPY --from=py-ea /opt/elastalert /opt/elastalert
COPY --from=py-ea /usr/bin/elastalert* /usr/bin/

WORKDIR /opt/elastalert-server
COPY . /opt/elastalert-server

RUN npm install --production --quiet
COPY config/elastalert.yaml /opt/elastalert/config.yaml
COPY config/elastalert-test.yaml /opt/elastalert/config-test.yaml
COPY config/config.json config/config.json
COPY rule_templates/ /opt/elastalert/rule_templates
COPY elastalert_modules/ /opt/elastalert/elastalert_modules

EXPOSE 3030
ENTRYPOINT ["npm", "start"]

[saroyaj]

@jibsonline Using your config I get this error. Am I missing something ?

Installed /usr/lib/python2.7/site-packages/croniter-0.3.37-py2.7.egg
Searching for configparser>=3.5.0
Reading https://pypi.org/simple/configparser/
Downloading https://files.pythonhosted.org/packages/3f/e7/9518720c56396179f8c63d7b5924c8463ed423828e54329be7f8cde5c364/configparser-5.0.1.tar.gz#sha256=005c3b102c96f4be9b8f40dafbd4997db003d07d1caa19f37808be8031475f2a
Best match: configparser 5.0.1
Processing configparser-5.0.1.tar.gz
Writing /tmp/easy_install-DYi9wW/configparser-5.0.1/setup.cfg
Running configparser-5.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-DYi9wW/configparser-5.0.1/egg-dist-tmp-rLCXdh
error: Setup script exited with error: 'egg_base' must be a directory name (got src)
The command '/bin/sh -c sed -i 's/jira>=1.0.10/jira>=1.0.10,<1.0.15/g' setup.py && sed -i 's/mock>=2.0.0/mock==3.0.5/g' setup.py && python setup.py install && pip install apscheduler>=3.3.0 aws-requests-auth>=0.3.0 blist>=1.3.6 boto3>=1.4.4 cffi>=1.11.5 configparser>=3.5.0 croniter>=0.3.16 elasticsearch>=7.0.0 envparse>=0.2.0 exotel>=0.1.3 "jira>=1.0.10,<1.0.15" jsonschema>=3.0.2 mock==3.0.5 prison>=0.1.2 py-zabbix==1.1.3 PyStaticConfiguration>=0.10.3 "python-dateutil>=2.6.0,<2.7.0" python-magic>=0.4.15 PyYAML>=5.1 requests>=2.0.0 stomp.py==4.1.23 texttable>=0.8.8 thehive4py>=1.4.4 twilio==6.0.0' returned a non-zero code: 1
Makefile:6: recipe for target 'build' failed
make: *** [build] Error 1