The Complete Guide to Elastic Stack: From Subscription to Production-Ready Setup

Introduction

The Elastic Stack (formerly known as ELK Stack) has become the go-to solution for log analytics, search, and observability. This comprehensive guide walks you through everything from purchasing a subscription to configuring a production-ready environment with hands-on labs.

What is the Elastic Stack?

The Elastic Stack consists of four main components:

  • Elasticsearch: Distributed search and analytics engine
  • Logstash: Server-side data processing pipeline
  • Kibana: Data visualization and exploration tool
  • Beats: Lightweight data shippers

Subscription Options and Purchasing

Elastic Cloud vs Self-Managed

Elastic Cloud (Recommended for beginners)

  • Fully managed service hosted on AWS, GCP, or Azure
  • Automatic updates and maintenance
  • Built-in security and monitoring
  • Pay-as-you-go pricing

Self-Managed

  • Install and manage on your own infrastructure
  • More control and customization options
  • Requires technical expertise for maintenance
  • Various licensing tiers available

Subscription Tiers

Basic (Free)

  • Core Elasticsearch, Logstash, Kibana, and Beats
  • Basic security features
  • Community support

Gold

  • Advanced security features
  • Alerting and notifications
  • Graph analytics
  • Standard support

Platinum

  • Machine learning features
  • Advanced security (SAML, LDAP)
  • Cross-cluster replication
  • Premium support

Enterprise

  • All Platinum features
  • Advanced machine learning
  • FIPS 140-2 compliance
  • Dedicated support

How to Purchase

  1. Visit elastic.co
  2. Choose between Cloud or Self-Managed
  3. Select your subscription tier
  4. For Elastic Cloud: Choose cloud provider and region
  5. Configure cluster size and storage requirements
  6. Complete payment and account setup

Installation Methods

Method 1: Elastic Cloud (Easiest)

After purchasing an Elastic Cloud subscription:

  1. Log into your Elastic Cloud console
  2. Click “Create Deployment”
  3. Choose your configuration template
  4. Select cloud provider and region
  5. Configure hardware specifications
  6. Deploy your cluster

Method 2: Self-Managed Installation

We’ll demonstrate installation on Ubuntu 20.04 LTS.

Prerequisites

# Update system packages
sudo apt update && sudo apt upgrade -y

# Install Java (required for Elasticsearch)
sudo apt install openjdk-11-jdk -y

# Verify Java installation
java -version

Installing Elasticsearch

# Import Elastic repository signing key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

# Add Elastic repository
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

# Update package index
sudo apt update

# Install Elasticsearch
sudo apt install elasticsearch -y

# Configure Elasticsearch to start on boot
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch

Installing Kibana

# Install Kibana
sudo apt install kibana -y

# Enable Kibana service
sudo systemctl enable kibana

Installing Logstash

# Install Logstash
sudo apt install logstash -y

# Enable Logstash service
sudo systemctl enable logstash

Installing Beats (Filebeat example)

# Install Filebeat
sudo apt install filebeat -y

# Enable Filebeat service
sudo systemctl enable filebeat

Basic Configuration

Elasticsearch Configuration

Edit /etc/elasticsearch/elasticsearch.yml:

# Cluster configuration
cluster.name: my-elastic-cluster
node.name: node-1

# Network configuration
network.host: localhost
http.port: 9200

# Path configuration
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

# Memory settings (adjust based on your system)
bootstrap.memory_lock: true

# Security settings (Elasticsearch 8.x has security enabled by default)
xpack.security.enabled: true
xpack.security.enrollment.enabled: true

Edit /etc/elasticsearch/jvm.options:

# Set heap size (typically 50% of available RAM, max 32GB)
-Xms2g
-Xmx2g

Kibana Configuration

Edit /etc/kibana/kibana.yml:

# Server configuration
server.port: 5601
server.host: "localhost"

# Elasticsearch configuration
elasticsearch.hosts: ["http://localhost:9200"]

# Security configuration
elasticsearch.username: "kibana_system"
elasticsearch.password: "your_password_here"

Starting Services

# Start Elasticsearch
sudo systemctl start elasticsearch

# Wait for Elasticsearch to start, then start Kibana
sudo systemctl start kibana

# Check service status
sudo systemctl status elasticsearch
sudo systemctl status kibana

Lab 1: Basic Setup Verification

Objective

Verify that Elasticsearch and Kibana are running correctly.

Steps

  1. Test Elasticsearch connectivity:
curl -X GET "localhost:9200/"

Expected output should show cluster information.

  1. Access Kibana web interface: Open your browser and navigate to http://localhost:5601
  2. Create your first index:
curl -X PUT "localhost:9200/my-first-index" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  }
}
'
  1. Add sample data:
curl -X POST "localhost:9200/my-first-index/_doc/1" -H 'Content-Type: application/json' -d'
{
  "title": "My First Document",
  "content": "This is a test document for Elastic Stack",
  "timestamp": "2024-01-15T10:30:00"
}
'
  1. Search the data:
curl -X GET "localhost:9200/my-first-index/_search?q=test"

Lab 2: Log Ingestion with Filebeat

Objective

Configure Filebeat to collect system logs and send them to Elasticsearch.

Steps

  1. Configure Filebeat: Edit /etc/filebeat/filebeat.yml:
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/syslog
    - /var/log/auth.log

output.elasticsearch:
  hosts: ["localhost:9200"]
  username: "elastic"
  password: "your_password"

setup.kibana:
  host: "localhost:5601"
  1. Setup Filebeat dashboards:
sudo filebeat setup --dashboards
  1. Start Filebeat:
sudo systemctl start filebeat
sudo systemctl status filebeat
  1. Verify data ingestion: Check in Kibana under “Discover” for filebeat-* indices.

Lab 3: Custom Logstash Pipeline

Objective

Create a Logstash pipeline to parse custom application logs.

Steps

  1. Create sample log file:
sudo mkdir -p /var/log/myapp
echo '2024-01-15 10:30:00 INFO [UserService] User john.doe logged in from 192.168.1.100' | sudo tee /var/log/myapp/application.log
echo '2024-01-15 10:35:00 ERROR [PaymentService] Payment failed for user jane.smith - Invalid card' | sudo tee -a /var/log/myapp/application.log
  1. Create Logstash configuration: Create /etc/logstash/conf.d/myapp.conf:
input {
  file {
    path => "/var/log/myapp/application.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  grok {
    match => { 
      "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} \[%{DATA:service}\] %{GREEDYDATA:log_message}" 
    }
  }
  
  date {
    match => [ "timestamp", "yyyy-MM-dd HH:mm:ss" ]
  }
  
  if [log_message] =~ /logged in from/ {
    grok {
      match => { 
        "log_message" => "User %{USERNAME:username} logged in from %{IP:client_ip}" 
      }
    }
    mutate {
      add_tag => [ "login_event" ]
    }
  }
  
  if [log_message] =~ /Payment failed/ {
    grok {
      match => { 
        "log_message" => "Payment failed for user %{USERNAME:username} - %{GREEDYDATA:error_reason}" 
      }
    }
    mutate {
      add_tag => [ "payment_error" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "myapp-logs-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "your_password"
  }
  
  stdout { 
    codec => rubydebug 
  }
}
  1. Test Logstash configuration:
sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t
  1. Start Logstash:
sudo systemctl start logstash
sudo systemctl status logstash
  1. Verify data in Kibana: Create an index pattern for myapp-logs-* and explore the parsed data.

Lab 4: Creating Kibana Dashboards

Objective

Build a comprehensive dashboard for monitoring application logs.

Steps

  1. Create Index Pattern:
    • Go to Kibana → Stack Management → Index Patterns
    • Click “Create index pattern”
    • Enter pattern: myapp-logs-*
    • Select timestamp field
    • Save the index pattern
  2. Create Visualizations:

Login Events Over Time:

  • Go to Kibana → Visualize → Create visualization
  • Choose “Line chart”
  • Select your index pattern
  • Set Y-axis to Count
  • Set X-axis to Date Histogram on timestamp field
  • Add filter: tags: login_event

Error Rate by Service:

  • Create new visualization → Pie chart
  • Set Metrics to Count
  • Set Buckets to Terms aggregation on service.keyword field
  • Add filter: level: ERROR

Geographic Login Distribution:

  • Create new visualization → Maps
  • Add layer → Choropleth
  • Configure based on client_ip field (requires GeoIP processor)
  1. Build Dashboard:
    • Go to Kibana → Dashboard → Create dashboard
    • Add your created visualizations
    • Arrange and resize as needed
    • Save the dashboard

Advanced Configuration

Security Configuration

For production environments, enable security features:

# In elasticsearch.yml
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

Performance Tuning

Elasticsearch JVM Settings:

# For production with 16GB RAM
-Xms8g
-Xmx8g

Index Templates:

{
  "index_patterns": ["myapp-logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "index.refresh_interval": "30s"
    },
    "mappings": {
      "properties": {
        "timestamp": { "type": "date" },
        "level": { "type": "keyword" },
        "service": { "type": "keyword" },
        "username": { "type": "keyword" },
        "client_ip": { "type": "ip" }
      }
    }
  }
}

Monitoring and Alerting

Enable Stack Monitoring:

# In elasticsearch.yml
xpack.monitoring.collection.enabled: true

# In kibana.yml
xpack.monitoring.ui.container.elasticsearch.enabled: true

Create alerts in Kibana:

  1. Go to Stack Management → Rules and Connectors
  2. Create rule for error rate threshold
  3. Configure actions (email, Slack, etc.)

Production Considerations

Cluster Architecture

For production, consider a multi-node setup:

  • Master nodes: 3 dedicated master nodes for cluster management
  • Data nodes: Multiple data nodes for storing and searching data
  • Ingest nodes: Dedicated nodes for data preprocessing
  • Coordinating nodes: Load balancing and request routing

Backup Strategy

# Create snapshot repository
curl -X PUT "localhost:9200/_snapshot/my_backup" -H 'Content-Type: application/json' -d'
{
  "type": "fs",
  "settings": {
    "location": "/mount/backups/my_backup"
  }
}
'

# Create snapshot
curl -X PUT "localhost:9200/_snapshot/my_backup/snapshot_1"

Capacity Planning

Monitor these key metrics:

  • CPU usage (should stay below 80%)
  • Memory usage
  • Disk space and I/O
  • JVM heap usage
  • Search and indexing rates

Troubleshooting Common Issues

Elasticsearch Won’t Start

Check logs:

sudo journalctl -u elasticsearch -f

Common fixes:

  • Increase file descriptor limits
  • Adjust JVM heap size
  • Check disk space
  • Verify network configuration

High Memory Usage

Monitor JVM heap:

curl -X GET "localhost:9200/_nodes/stats/jvm"

Solutions:

  • Reduce heap size if too large
  • Clear old indices
  • Optimize mappings
  • Reduce replica count

Slow Search Performance

Check cluster health:

curl -X GET "localhost:9200/_cluster/health"

Optimization techniques:

  • Use appropriate data types
  • Implement proper sharding strategy
  • Enable caching
  • Optimize queries

Best Practices

Index Management

  • Use Index Lifecycle Management (ILM)
  • Implement proper naming conventions
  • Regular cleanup of old indices
  • Monitor shard sizes

Security

  • Enable authentication and authorization
  • Use TLS encryption
  • Regular security updates
  • Implement network segmentation

Monitoring

  • Set up comprehensive monitoring
  • Create alerting rules
  • Regular performance reviews
  • Capacity planning

Conclusion

The Elastic Stack provides powerful capabilities for log analytics, search, and observability. This guide covered the complete journey from purchasing subscriptions to building production-ready deployments with hands-on labs.

Key takeaways:

  • Choose the right subscription tier for your needs
  • Follow security best practices from the start
  • Implement proper monitoring and alerting
  • Plan for scalability and performance
  • Regular maintenance and updates are crucial

With these foundations, you’re ready to implement robust log analytics and search solutions using the Elastic Stack. Remember to continuously monitor performance, update your configurations based on changing requirements, and stay current with Elastic’s evolving feature set.

For additional resources, consult the official Elastic documentation and community forums for the latest updates and advanced configuration techniques.

Share: