The Complete Guide to Elastic Stack: From Subscription to Production-Ready Setup

Introduction

The Elastic Stack (formerly known as ELK Stack) has become the go-to solution for log analytics, search, and observability. This comprehensive guide walks you through everything from purchasing a subscription to configuring a production-ready environment with hands-on labs.

What is the Elastic Stack?

The Elastic Stack consists of four main components:

Elasticsearch: Distributed search and analytics engine
Logstash: Server-side data processing pipeline
Kibana: Data visualization and exploration tool
Beats: Lightweight data shippers

Subscription Options and Purchasing

Elastic Cloud vs Self-Managed

Elastic Cloud (Recommended for beginners)

Fully managed service hosted on AWS, GCP, or Azure
Automatic updates and maintenance
Built-in security and monitoring
Pay-as-you-go pricing

Self-Managed

Install and manage on your own infrastructure
More control and customization options
Requires technical expertise for maintenance
Various licensing tiers available

Subscription Tiers

Basic (Free)

Core Elasticsearch, Logstash, Kibana, and Beats
Basic security features
Community support

Gold

Advanced security features
Alerting and notifications
Graph analytics
Standard support

Platinum

Machine learning features
Advanced security (SAML, LDAP)
Cross-cluster replication
Premium support

Enterprise

All Platinum features
Advanced machine learning
FIPS 140-2 compliance
Dedicated support

How to Purchase

Visit elastic.co
Choose between Cloud or Self-Managed
Select your subscription tier
For Elastic Cloud: Choose cloud provider and region
Configure cluster size and storage requirements
Complete payment and account setup

Installation Methods

Method 1: Elastic Cloud (Easiest)

After purchasing an Elastic Cloud subscription:

Log into your Elastic Cloud console
Click “Create Deployment”
Choose your configuration template
Select cloud provider and region
Configure hardware specifications
Deploy your cluster

Method 2: Self-Managed Installation

We’ll demonstrate installation on Ubuntu 20.04 LTS.

Prerequisites

# Update system packages
sudo apt update && sudo apt upgrade -y

# Install Java (required for Elasticsearch)
sudo apt install openjdk-11-jdk -y

# Verify Java installation
java -version

Installing Elasticsearch

# Import Elastic repository signing key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

# Add Elastic repository
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

# Update package index
sudo apt update

# Install Elasticsearch
sudo apt install elasticsearch -y

# Configure Elasticsearch to start on boot
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch

Installing Kibana

# Install Kibana
sudo apt install kibana -y

# Enable Kibana service
sudo systemctl enable kibana

Installing Logstash

# Install Logstash
sudo apt install logstash -y

# Enable Logstash service
sudo systemctl enable logstash

Installing Beats (Filebeat example)

# Install Filebeat
sudo apt install filebeat -y

# Enable Filebeat service
sudo systemctl enable filebeat

Basic Configuration

Elasticsearch Configuration

Edit /etc/elasticsearch/elasticsearch.yml:

# Cluster configuration
cluster.name: my-elastic-cluster
node.name: node-1

# Network configuration
network.host: localhost
http.port: 9200

# Path configuration
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

# Memory settings (adjust based on your system)
bootstrap.memory_lock: true

# Security settings (Elasticsearch 8.x has security enabled by default)
xpack.security.enabled: true
xpack.security.enrollment.enabled: true

Edit /etc/elasticsearch/jvm.options:

# Set heap size (typically 50% of available RAM, max 32GB)
-Xms2g
-Xmx2g

Kibana Configuration

Edit /etc/kibana/kibana.yml:

# Server configuration
server.port: 5601
server.host: "localhost"

# Elasticsearch configuration
elasticsearch.hosts: ["http://localhost:9200"]

# Security configuration
elasticsearch.username: "kibana_system"
elasticsearch.password: "your_password_here"

Starting Services

# Start Elasticsearch
sudo systemctl start elasticsearch

# Wait for Elasticsearch to start, then start Kibana
sudo systemctl start kibana

# Check service status
sudo systemctl status elasticsearch
sudo systemctl status kibana

Lab 1: Basic Setup Verification

Objective

Verify that Elasticsearch and Kibana are running correctly.

Steps

Test Elasticsearch connectivity:

curl -X GET "localhost:9200/"

Expected output should show cluster information.

Access Kibana web interface: Open your browser and navigate to http://localhost:5601
Create your first index:

curl -X PUT "localhost:9200/my-first-index" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  }
}
'

Add sample data:

curl -X POST "localhost:9200/my-first-index/_doc/1" -H 'Content-Type: application/json' -d'
{
  "title": "My First Document",
  "content": "This is a test document for Elastic Stack",
  "timestamp": "2024-01-15T10:30:00"
}
'

Search the data:

curl -X GET "localhost:9200/my-first-index/_search?q=test"

Lab 2: Log Ingestion with Filebeat

Objective

Configure Filebeat to collect system logs and send them to Elasticsearch.

Steps

Configure Filebeat: Edit /etc/filebeat/filebeat.yml:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/syslog
    - /var/log/auth.log

output.elasticsearch:
  hosts: ["localhost:9200"]
  username: "elastic"
  password: "your_password"

setup.kibana:
  host: "localhost:5601"

Setup Filebeat dashboards:

sudo filebeat setup --dashboards

Start Filebeat:

sudo systemctl start filebeat
sudo systemctl status filebeat

Verify data ingestion: Check in Kibana under “Discover” for filebeat-* indices.

Lab 3: Custom Logstash Pipeline

Objective

Create a Logstash pipeline to parse custom application logs.

Steps

Create sample log file:

sudo mkdir -p /var/log/myapp
echo '2024-01-15 10:30:00 INFO [UserService] User john.doe logged in from 192.168.1.100' | sudo tee /var/log/myapp/application.log
echo '2024-01-15 10:35:00 ERROR [PaymentService] Payment failed for user jane.smith - Invalid card' | sudo tee -a /var/log/myapp/application.log

Create Logstash configuration: Create /etc/logstash/conf.d/myapp.conf:

input {
  file {
    path => "/var/log/myapp/application.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  grok {
    match => { 
      "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} \[%{DATA:service}\] %{GREEDYDATA:log_message}" 
    }
  }
  
  date {
    match => [ "timestamp", "yyyy-MM-dd HH:mm:ss" ]
  }
  
  if [log_message] =~ /logged in from/ {
    grok {
      match => { 
        "log_message" => "User %{USERNAME:username} logged in from %{IP:client_ip}" 
      }
    }
    mutate {
      add_tag => [ "login_event" ]
    }
  }
  
  if [log_message] =~ /Payment failed/ {
    grok {
      match => { 
        "log_message" => "Payment failed for user %{USERNAME:username} - %{GREEDYDATA:error_reason}" 
      }
    }
    mutate {
      add_tag => [ "payment_error" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "myapp-logs-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "your_password"
  }
  
  stdout { 
    codec => rubydebug 
  }
}

Test Logstash configuration:

sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t

Start Logstash:

sudo systemctl start logstash
sudo systemctl status logstash

Verify data in Kibana: Create an index pattern for myapp-logs-* and explore the parsed data.

Lab 4: Creating Kibana Dashboards

Objective

Build a comprehensive dashboard for monitoring application logs.

Steps

Create Index Pattern:
- Go to Kibana → Stack Management → Index Patterns
- Click “Create index pattern”
- Enter pattern: myapp-logs-*
- Select timestamp field
- Save the index pattern
Create Visualizations:

Login Events Over Time:

Go to Kibana → Visualize → Create visualization
Choose “Line chart”
Select your index pattern
Set Y-axis to Count
Set X-axis to Date Histogram on timestamp field
Add filter: tags: login_event

Error Rate by Service:

Create new visualization → Pie chart
Set Metrics to Count
Set Buckets to Terms aggregation on service.keyword field
Add filter: level: ERROR

Geographic Login Distribution:

Create new visualization → Maps
Add layer → Choropleth
Configure based on client_ip field (requires GeoIP processor)

Build Dashboard:
- Go to Kibana → Dashboard → Create dashboard
- Add your created visualizations
- Arrange and resize as needed
- Save the dashboard

Advanced Configuration

Security Configuration

For production environments, enable security features:

# In elasticsearch.yml
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

Performance Tuning

Elasticsearch JVM Settings:

# For production with 16GB RAM
-Xms8g
-Xmx8g

Index Templates:

{
  "index_patterns": ["myapp-logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "index.refresh_interval": "30s"
    },
    "mappings": {
      "properties": {
        "timestamp": { "type": "date" },
        "level": { "type": "keyword" },
        "service": { "type": "keyword" },
        "username": { "type": "keyword" },
        "client_ip": { "type": "ip" }
      }
    }
  }
}

Monitoring and Alerting

Enable Stack Monitoring:

# In elasticsearch.yml
xpack.monitoring.collection.enabled: true

# In kibana.yml
xpack.monitoring.ui.container.elasticsearch.enabled: true

Create alerts in Kibana:

Go to Stack Management → Rules and Connectors
Create rule for error rate threshold
Configure actions (email, Slack, etc.)

Production Considerations

Cluster Architecture

For production, consider a multi-node setup:

Master nodes: 3 dedicated master nodes for cluster management
Data nodes: Multiple data nodes for storing and searching data
Ingest nodes: Dedicated nodes for data preprocessing
Coordinating nodes: Load balancing and request routing

Backup Strategy

# Create snapshot repository
curl -X PUT "localhost:9200/_snapshot/my_backup" -H 'Content-Type: application/json' -d'
{
  "type": "fs",
  "settings": {
    "location": "/mount/backups/my_backup"
  }
}
'

# Create snapshot
curl -X PUT "localhost:9200/_snapshot/my_backup/snapshot_1"

Capacity Planning

Monitor these key metrics:

CPU usage (should stay below 80%)
Memory usage
Disk space and I/O
JVM heap usage
Search and indexing rates

Troubleshooting Common Issues

Elasticsearch Won’t Start

Check logs:

sudo journalctl -u elasticsearch -f

Common fixes:

Increase file descriptor limits
Adjust JVM heap size
Check disk space
Verify network configuration

High Memory Usage

Monitor JVM heap:

curl -X GET "localhost:9200/_nodes/stats/jvm"

Solutions:

Reduce heap size if too large
Clear old indices
Optimize mappings
Reduce replica count

Slow Search Performance

Check cluster health:

curl -X GET "localhost:9200/_cluster/health"

Optimization techniques:

Use appropriate data types
Implement proper sharding strategy
Enable caching
Optimize queries

Best Practices

Index Management

Use Index Lifecycle Management (ILM)
Implement proper naming conventions
Regular cleanup of old indices
Monitor shard sizes

Security

Enable authentication and authorization
Use TLS encryption
Regular security updates
Implement network segmentation

Monitoring

Set up comprehensive monitoring
Create alerting rules
Regular performance reviews
Capacity planning

Conclusion

The Elastic Stack provides powerful capabilities for log analytics, search, and observability. This guide covered the complete journey from purchasing subscriptions to building production-ready deployments with hands-on labs.

Key takeaways:

Choose the right subscription tier for your needs
Follow security best practices from the start
Implement proper monitoring and alerting
Plan for scalability and performance
Regular maintenance and updates are crucial

With these foundations, you’re ready to implement robust log analytics and search solutions using the Elastic Stack. Remember to continuously monitor performance, update your configurations based on changing requirements, and stay current with Elastic’s evolving feature set.

For additional resources, consult the official Elastic documentation and community forums for the latest updates and advanced configuration techniques.