Introduction
The Elastic Stack (formerly known as ELK Stack) has become the go-to solution for log analytics, search, and observability. This comprehensive guide walks you through everything from purchasing a subscription to configuring a production-ready environment with hands-on labs.
What is the Elastic Stack?
The Elastic Stack consists of four main components:
- Elasticsearch: Distributed search and analytics engine
- Logstash: Server-side data processing pipeline
- Kibana: Data visualization and exploration tool
- Beats: Lightweight data shippers
Subscription Options and Purchasing
Elastic Cloud vs Self-Managed
Elastic Cloud (Recommended for beginners)
- Fully managed service hosted on AWS, GCP, or Azure
- Automatic updates and maintenance
- Built-in security and monitoring
- Pay-as-you-go pricing
Self-Managed
- Install and manage on your own infrastructure
- More control and customization options
- Requires technical expertise for maintenance
- Various licensing tiers available
Subscription Tiers
Basic (Free)
- Core Elasticsearch, Logstash, Kibana, and Beats
- Basic security features
- Community support
Gold
- Advanced security features
- Alerting and notifications
- Graph analytics
- Standard support
Platinum
- Machine learning features
- Advanced security (SAML, LDAP)
- Cross-cluster replication
- Premium support
Enterprise
- All Platinum features
- Advanced machine learning
- FIPS 140-2 compliance
- Dedicated support
How to Purchase
- Visit elastic.co
- Choose between Cloud or Self-Managed
- Select your subscription tier
- For Elastic Cloud: Choose cloud provider and region
- Configure cluster size and storage requirements
- Complete payment and account setup
Installation Methods
Method 1: Elastic Cloud (Easiest)
After purchasing an Elastic Cloud subscription:
- Log into your Elastic Cloud console
- Click “Create Deployment”
- Choose your configuration template
- Select cloud provider and region
- Configure hardware specifications
- Deploy your cluster
Method 2: Self-Managed Installation
We’ll demonstrate installation on Ubuntu 20.04 LTS.
Prerequisites
# Update system packages
sudo apt update && sudo apt upgrade -y
# Install Java (required for Elasticsearch)
sudo apt install openjdk-11-jdk -y
# Verify Java installation
java -version
Installing Elasticsearch
# Import Elastic repository signing key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
# Add Elastic repository
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
# Update package index
sudo apt update
# Install Elasticsearch
sudo apt install elasticsearch -y
# Configure Elasticsearch to start on boot
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch
Installing Kibana
# Install Kibana
sudo apt install kibana -y
# Enable Kibana service
sudo systemctl enable kibana
Installing Logstash
# Install Logstash
sudo apt install logstash -y
# Enable Logstash service
sudo systemctl enable logstash
Installing Beats (Filebeat example)
# Install Filebeat
sudo apt install filebeat -y
# Enable Filebeat service
sudo systemctl enable filebeat
Basic Configuration
Elasticsearch Configuration
Edit /etc/elasticsearch/elasticsearch.yml
:
# Cluster configuration
cluster.name: my-elastic-cluster
node.name: node-1
# Network configuration
network.host: localhost
http.port: 9200
# Path configuration
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
# Memory settings (adjust based on your system)
bootstrap.memory_lock: true
# Security settings (Elasticsearch 8.x has security enabled by default)
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
Edit /etc/elasticsearch/jvm.options
:
# Set heap size (typically 50% of available RAM, max 32GB)
-Xms2g
-Xmx2g
Kibana Configuration
Edit /etc/kibana/kibana.yml
:
# Server configuration
server.port: 5601
server.host: "localhost"
# Elasticsearch configuration
elasticsearch.hosts: ["http://localhost:9200"]
# Security configuration
elasticsearch.username: "kibana_system"
elasticsearch.password: "your_password_here"
Starting Services
# Start Elasticsearch
sudo systemctl start elasticsearch
# Wait for Elasticsearch to start, then start Kibana
sudo systemctl start kibana
# Check service status
sudo systemctl status elasticsearch
sudo systemctl status kibana
Lab 1: Basic Setup Verification
Objective
Verify that Elasticsearch and Kibana are running correctly.
Steps
- Test Elasticsearch connectivity:
curl -X GET "localhost:9200/"
Expected output should show cluster information.
- Access Kibana web interface: Open your browser and navigate to
http://localhost:5601
- Create your first index:
curl -X PUT "localhost:9200/my-first-index" -H 'Content-Type: application/json' -d'
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
}
}
'
- Add sample data:
curl -X POST "localhost:9200/my-first-index/_doc/1" -H 'Content-Type: application/json' -d'
{
"title": "My First Document",
"content": "This is a test document for Elastic Stack",
"timestamp": "2024-01-15T10:30:00"
}
'
- Search the data:
curl -X GET "localhost:9200/my-first-index/_search?q=test"
Lab 2: Log Ingestion with Filebeat
Objective
Configure Filebeat to collect system logs and send them to Elasticsearch.
Steps
- Configure Filebeat: Edit
/etc/filebeat/filebeat.yml
:
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/syslog
- /var/log/auth.log
output.elasticsearch:
hosts: ["localhost:9200"]
username: "elastic"
password: "your_password"
setup.kibana:
host: "localhost:5601"
- Setup Filebeat dashboards:
sudo filebeat setup --dashboards
- Start Filebeat:
sudo systemctl start filebeat
sudo systemctl status filebeat
- Verify data ingestion: Check in Kibana under “Discover” for filebeat-* indices.
Lab 3: Custom Logstash Pipeline
Objective
Create a Logstash pipeline to parse custom application logs.
Steps
- Create sample log file:
sudo mkdir -p /var/log/myapp
echo '2024-01-15 10:30:00 INFO [UserService] User john.doe logged in from 192.168.1.100' | sudo tee /var/log/myapp/application.log
echo '2024-01-15 10:35:00 ERROR [PaymentService] Payment failed for user jane.smith - Invalid card' | sudo tee -a /var/log/myapp/application.log
- Create Logstash configuration: Create
/etc/logstash/conf.d/myapp.conf
:
input {
file {
path => "/var/log/myapp/application.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
grok {
match => {
"message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} \[%{DATA:service}\] %{GREEDYDATA:log_message}"
}
}
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss" ]
}
if [log_message] =~ /logged in from/ {
grok {
match => {
"log_message" => "User %{USERNAME:username} logged in from %{IP:client_ip}"
}
}
mutate {
add_tag => [ "login_event" ]
}
}
if [log_message] =~ /Payment failed/ {
grok {
match => {
"log_message" => "Payment failed for user %{USERNAME:username} - %{GREEDYDATA:error_reason}"
}
}
mutate {
add_tag => [ "payment_error" ]
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "myapp-logs-%{+YYYY.MM.dd}"
user => "elastic"
password => "your_password"
}
stdout {
codec => rubydebug
}
}
- Test Logstash configuration:
sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t
- Start Logstash:
sudo systemctl start logstash
sudo systemctl status logstash
- Verify data in Kibana: Create an index pattern for
myapp-logs-*
and explore the parsed data.
Lab 4: Creating Kibana Dashboards
Objective
Build a comprehensive dashboard for monitoring application logs.
Steps
- Create Index Pattern:
- Go to Kibana → Stack Management → Index Patterns
- Click “Create index pattern”
- Enter pattern:
myapp-logs-*
- Select timestamp field
- Save the index pattern
- Create Visualizations:
Login Events Over Time:
- Go to Kibana → Visualize → Create visualization
- Choose “Line chart”
- Select your index pattern
- Set Y-axis to Count
- Set X-axis to Date Histogram on timestamp field
- Add filter:
tags: login_event
Error Rate by Service:
- Create new visualization → Pie chart
- Set Metrics to Count
- Set Buckets to Terms aggregation on
service.keyword
field - Add filter:
level: ERROR
Geographic Login Distribution:
- Create new visualization → Maps
- Add layer → Choropleth
- Configure based on client_ip field (requires GeoIP processor)
- Build Dashboard:
- Go to Kibana → Dashboard → Create dashboard
- Add your created visualizations
- Arrange and resize as needed
- Save the dashboard
Advanced Configuration
Security Configuration
For production environments, enable security features:
# In elasticsearch.yml
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
Performance Tuning
Elasticsearch JVM Settings:
# For production with 16GB RAM
-Xms8g
-Xmx8g
Index Templates:
{
"index_patterns": ["myapp-logs-*"],
"template": {
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1,
"index.refresh_interval": "30s"
},
"mappings": {
"properties": {
"timestamp": { "type": "date" },
"level": { "type": "keyword" },
"service": { "type": "keyword" },
"username": { "type": "keyword" },
"client_ip": { "type": "ip" }
}
}
}
}
Monitoring and Alerting
Enable Stack Monitoring:
# In elasticsearch.yml
xpack.monitoring.collection.enabled: true
# In kibana.yml
xpack.monitoring.ui.container.elasticsearch.enabled: true
Create alerts in Kibana:
- Go to Stack Management → Rules and Connectors
- Create rule for error rate threshold
- Configure actions (email, Slack, etc.)
Production Considerations
Cluster Architecture
For production, consider a multi-node setup:
- Master nodes: 3 dedicated master nodes for cluster management
- Data nodes: Multiple data nodes for storing and searching data
- Ingest nodes: Dedicated nodes for data preprocessing
- Coordinating nodes: Load balancing and request routing
Backup Strategy
# Create snapshot repository
curl -X PUT "localhost:9200/_snapshot/my_backup" -H 'Content-Type: application/json' -d'
{
"type": "fs",
"settings": {
"location": "/mount/backups/my_backup"
}
}
'
# Create snapshot
curl -X PUT "localhost:9200/_snapshot/my_backup/snapshot_1"
Capacity Planning
Monitor these key metrics:
- CPU usage (should stay below 80%)
- Memory usage
- Disk space and I/O
- JVM heap usage
- Search and indexing rates
Troubleshooting Common Issues
Elasticsearch Won’t Start
Check logs:
sudo journalctl -u elasticsearch -f
Common fixes:
- Increase file descriptor limits
- Adjust JVM heap size
- Check disk space
- Verify network configuration
High Memory Usage
Monitor JVM heap:
curl -X GET "localhost:9200/_nodes/stats/jvm"
Solutions:
- Reduce heap size if too large
- Clear old indices
- Optimize mappings
- Reduce replica count
Slow Search Performance
Check cluster health:
curl -X GET "localhost:9200/_cluster/health"
Optimization techniques:
- Use appropriate data types
- Implement proper sharding strategy
- Enable caching
- Optimize queries
Best Practices
Index Management
- Use Index Lifecycle Management (ILM)
- Implement proper naming conventions
- Regular cleanup of old indices
- Monitor shard sizes
Security
- Enable authentication and authorization
- Use TLS encryption
- Regular security updates
- Implement network segmentation
Monitoring
- Set up comprehensive monitoring
- Create alerting rules
- Regular performance reviews
- Capacity planning
Conclusion
The Elastic Stack provides powerful capabilities for log analytics, search, and observability. This guide covered the complete journey from purchasing subscriptions to building production-ready deployments with hands-on labs.
Key takeaways:
- Choose the right subscription tier for your needs
- Follow security best practices from the start
- Implement proper monitoring and alerting
- Plan for scalability and performance
- Regular maintenance and updates are crucial
With these foundations, you’re ready to implement robust log analytics and search solutions using the Elastic Stack. Remember to continuously monitor performance, update your configurations based on changing requirements, and stay current with Elastic’s evolving feature set.
For additional resources, consult the official Elastic documentation and community forums for the latest updates and advanced configuration techniques.