On this page
Every application produces logs, and most teams only read them in a crisis — over SSH, one server at a time. The ELK stack changes that. Elasticsearch stores and indexes events, Logstash collects and shapes them, and Kibana lets anyone filter and explore them in a browser.
This guide introduces each component, then runs the whole stack in Docker with security enabled from the first boot.
The three components
Diagram: application logs flow into Logstash, which runs input, filter and output stages before writing to Elasticsearch; Kibana reads from Elasticsearch to search and visualise.
- Elasticsearch
- A distributed engine for real-time search and analysis. It indexes data according to its type, which makes retrieval very fast. On first start it configures security automatically — generating certificates, a password for the
elasticuser and an enrolment token for Kibana. Keep them somewhere safe. - Logstash
- The collection engine. It gathers the data your applications produce, transforms it, and stores it in Elasticsearch. The simplest route is to keep logging to files as you already do and let Logstash read them.
- Kibana
- The window onto everything Elasticsearch holds. On first launch it prints a one-time link; open it, paste the enrolment token, then sign in as
elastic.
Anatomy of a pipeline
A Logstash pipeline is a .conf file with three sections:
- Input — where events come from. Input plugins read from a specific source, and codecs decode formats such as CSV on the way in.
- Filter — intermediary processing that makes events easier to index. Filters are often applied conditionally, depending on the characteristics of the event.
- Output — where events go. Here, the Elasticsearch plugin writes each event to an index you choose.
You can run several pipelines side by side by declaring them in pipelines.yml.
Running the stack in Docker
If your application already uses Docker Compose, merge the services below into your existing file; otherwise create a new one. The finished project looks like this:
.
├── docker-compose.yml
├── .env
├── elasticsearch/
│ ├── config/elasticsearch.yml
│ └── Dockerfile
├── kibana/
│ ├── config/kibana.yml
│ └── Dockerfile
├── logstash/
│ ├── config/logstash.yml
│ ├── pipeline/logstash.conf
│ └── Dockerfile
└── setup/
├── Dockerfile
├── entrypoint.sh
├── lib.sh
└── roles/logstash_writer.jsonservices:
# One-off job: creates the logstash_internal and kibana_system users
# with the passwords from .env, then exits on every later run.
setup:
build:
context: setup/
args:
ELASTIC_VERSION: \${ELASTIC_VERSION}
init: true
volumes:
- ./setup/entrypoint.sh:/entrypoint.sh:ro,Z
- ./setup/lib.sh:/lib.sh:ro,Z
- ./setup/roles:/roles:ro,Z
- setup:/state:Z
environment:
ELASTIC_PASSWORD: \${ELASTIC_PASSWORD:-}
LOGSTASH_INTERNAL_PASSWORD: \${LOGSTASH_INTERNAL_PASSWORD:-}
KIBANA_SYSTEM_PASSWORD: \${KIBANA_SYSTEM_PASSWORD:-}
networks: [elk]
depends_on: [elasticsearch]
elasticsearch:
build:
context: elasticsearch/
args:
ELASTIC_VERSION: \${ELASTIC_VERSION}
volumes:
- ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro,Z
- elasticsearch:/usr/share/elasticsearch/data:Z
ports: ["9200:9200", "9300:9300"]
environment:
node.name: elasticsearch
ES_JAVA_OPTS: -Xms512m -Xmx512m
ELASTIC_PASSWORD: \${ELASTIC_PASSWORD:-}
discovery.type: single-node
networks: [elk]
restart: unless-stopped
logstash:
build:
context: logstash/
args:
ELASTIC_VERSION: \${ELASTIC_VERSION}
volumes:
- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro,Z
- ./logstash/pipeline:/usr/share/logstash/pipeline:ro,Z
ports: ["5044:5044", "50000:50000/tcp", "50000:50000/udp", "9600:9600"]
environment:
LS_JAVA_OPTS: -Xms256m -Xmx256m
LOGSTASH_INTERNAL_PASSWORD: \${LOGSTASH_INTERNAL_PASSWORD:-}
networks: [elk]
depends_on: [elasticsearch]
restart: unless-stopped
kibana:
build:
context: kibana/
args:
ELASTIC_VERSION: \${ELASTIC_VERSION}
volumes:
- ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:ro,Z
ports: ["5601:5601"]
environment:
KIBANA_SYSTEM_PASSWORD: \${KIBANA_SYSTEM_PASSWORD:-}
networks: [elk]
depends_on: [elasticsearch]
restart: unless-stopped
networks:
elk:
driver: bridge
volumes:
setup:
elasticsearch:image or build?
A service can either pull a ready-made
imageorbuildfrom its own Dockerfile. Building each component separately makes it easy to add plugins to Elasticsearch, Kibana or Logstash later without touching the others.
Each component’s Dockerfile simply pins the official image to the version in .env — the place to add plugins later:
ARG ELASTIC_VERSION
FROM docker.elastic.co/logstash/logstash:\${ELASTIC_VERSION}
# Add Logstash plugins here, for example:
# RUN logstash-plugin install logstash-filter-jsonThe pipeline itself listens on TCP port 50000 and writes everything to Elasticsearch as the restricted logstash_internal user:
input {
tcp {
port => 50000
}
}
## Add filters / plugin configuration here
output {
elasticsearch {
hosts => "elasticsearch:9200"
user => "logstash_internal"
password => "\${LOGSTASH_INTERNAL_PASSWORD}"
}
}Versions and credentials live in one place. The setup service reads them on first boot to create users and roles; to add Filebeat or Metricbeat later, declare their passwords in entrypoint.sh and add a matching role file under setup/roles.
ELASTIC_VERSION=8.6.2
# Built-in superuser
ELASTIC_PASSWORD='changeme'
# Custom user Logstash uses to write to Elasticsearch
LOGSTASH_INTERNAL_PASSWORD='changeme'
# Built-in user Kibana uses to talk to Elasticsearch
KIBANA_SYSTEM_PASSWORD='changeme'Keep reading — it’s free
The remaining 1 min of this story is for Dispatch subscribers. Subscribing takes one field, and the whole story opens right here.
Subscribe to read the rest
Free. One email when there’s something worth reading.
Originally published at Aviox Blogs engineering handbook.
