Elastic Logstash Kibana ELK from Docker

[hazartilirot]

Well, I took another route, speaking of simplicity - it's really easy to set up. However, something went wrong with it and I had to figure out. Now, I share with all of you what should be done in order to get the result as fast as you can.

Open the project and clone it into any directory you want, supposing you're on MacOs

open terminal and type:
cd ~/desktop/dev

then clone the project:
git clone git@github.com:deviantony/docker-elk.git

Go to ~/desktop/dev/docker-elk/logstash/pipeline/logstash.conf and open logstash.conf. Paste the following text:

input {
	beats {
		port => 5044
	}
	tcp {
		port => 50000
	}
	file {
	    type => "users-ws"
	    path => "/var/log/microservices/users-ws.log"
	    start_position => "beginning"
	}
	file {
    	    type => "albums-ws"
    	    path => "/var/log/microservices/albums-ws.log"
    	    start_position => "beginning"
    }
}
output {
    if [type] == "users-ws" {
        elasticsearch {
            hosts => ["elasticsearch:9200"]
            data_stream => "false"
            user => "logstash_internal"
            password => "${LOGSTASH_INTERNAL_PASSWORD}"
            index => "users-ws-%{+YYYY.MM.dd}"
        }
    } else if [type] == "albums-ws" {
        elasticsearch {
            hosts => ["elasticsearch:9200"]
            data_stream => "false"
            user => "logstash_internal"
            password => "${LOGSTASH_INTERNAL_PASSWORD}"
            index => "albums-ws-%{+YYYY.MM.dd}"
        }
    }
    stdout { codec => rubydebug }
}

Mind, we need to mount two directories so that they are available within our docker container
/var/log/microservices/users-ws.log
/var/log/microservices/albums-ws.log

Now, we go to ~/desktop/dev/docker-elk/setup/entrypoint.sh and open entrypoint.sh

Find the line:

[logstash_internal]='logstash_writer'
(it is almost at the beginning, look for it in the users_roles section)

and change the value to
[logstash_internal]='superuser'

I can't explain the reason as to why we have to change the role. Otherwise, there would be a constant error with logstash_writer

and the last file we are going to set up is .env. It's hidden. If you're trying to open it within the IntelliJ IDEA, open the directory ~/desktop/dev/docker-elk/.env and then press SHIFT + CMD + . (dot). Override "changeme" values with your own password, mine was 123456789 and set ELASTIC_VERSION to the latest. For no reason they don't support "latest", therefore we have to specify the actual version in numbers.

ELASTIC_VERSION=8.7.0

## Passwords for stack users
#

# User 'elastic' (built-in)
#
# Superuser role, full access to cluster management and data indices.
# https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-users.html
ELASTIC_PASSWORD='123456789'

# User 'logstash_internal' (custom)
#
# The user Logstash uses to connect and send data to Elasticsearch.
# https://www.elastic.co/guide/en/logstash/current/ls-security.html
LOGSTASH_INTERNAL_PASSWORD='123456789'

# User 'kibana_system' (built-in)
#
# The user Kibana uses to connect and communicate with Elasticsearch.
# https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-users.html
KIBANA_SYSTEM_PASSWORD='123456789'

# Users 'metricbeat_internal', 'filebeat_internal' and 'heartbeat_internal' (custom)
#
# The users Beats use to connect and send data to Elasticsearch.
# https://www.elastic.co/guide/en/beats/metricbeat/current/feature-roles.html
METRICBEAT_INTERNAL_PASSWORD=''
FILEBEAT_INTERNAL_PASSWORD=''
HEARTBEAT_INTERNAL_PASSWORD=''

# User 'monitoring_internal' (custom)
#
# The user Metricbeat uses to collect monitoring data from stack components.
# https://www.elastic.co/guide/en/elasticsearch/reference/current/how-monitoring-works.html
MONITORING_INTERNAL_PASSWORD=''

# User 'beats_system' (built-in)
#
# The user the Beats use when storing monitoring information in Elasticsearch.
# https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-users.html
BEATS_SYSTEM_PASSWORD=''

the last step is to share our logs files with the logstash's container

Open docker-compose.yml file in ~/desktop/dev/docker-elk/ find logstash: service in volumes: add two more directories:
- /ABSOLUTE_PATH_TO_THE_LOG_DIRECTORY/users-ws.log:/var/log/microservices/users-ws.log
- /ABSOLUTE_PATH_TO_THE_LOG_DIRECTORY/albums-ws.log:/var/log/microservices/albums-ws.log

the left side from the colon is your physical path to the file, the right side from the colon is the mounted path to the file within the docker-contaiter.

Now, we're good to go.

Open terminal in ~/desktop/dev/docker-elk/ and run docker-compose up -d the "-d" key for detached mode.

Eventually you will get:

The last command you will need to execute in the future (to dismount entirely the whole thing) is:

open terminal in ~/desktop/dev/docker-elk/ and run docker-compose down -v

It would be really quick provided you don't encounter any errors.