Logging with Spring Boot,Elasticsearch, Logstash, Kibana

Introduction

In this tutorial, we will explore how to set up logging in a Spring Boot application using the Elastic Stack, which consists of Elasticsearch for storage, Logstash for processing, and Kibana for visualization. This powerful combination allows for centralized logging, making it easier to monitor and analyze logs from multiple microservices.

Prerequisites

• Java Development Kit (JDK) installed
• Spring Boot project set up
• Docker installed (for running Elastic Stack)

Step 1: Set Up a Spring Boot Project

If you don’t already have a Spring Boot project, you can create one using the Spring Initializr: Spring Initializr

Make sure to include the “Spring Web” and “Lombok” dependencies.

Step 2: Add Dependencies for Logging

Open your pom.xml file and add the following dependencies for logging:

<!-- Logback (default Spring Boot logging framework) -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter</artifactId>
</dependency>

<!-- Logstash Logback Encoder -->
<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>6.6</version>
</dependency>

These dependencies will enable us to format logs in a way that is compatible with Logstash.

Step 3: Configure Logback for Logstash

Create a logback-spring.xml file in the src/main/resources folder:

<configuration>
    <appender name="logstash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>localhost:5000</destination>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
    </appender>

    <root level="INFO">
        <appender-ref ref="logstash" />
    </root>
</configuration>

This configuration sends logs to Logstash via TCP on localhost:5000.

Step 4: Set Up the Elastic Stack (Elasticsearch, Logstash, Kibana)

Docker Compose Configuration

Create a docker-compose.yml file in your project’s root directory:

version: '2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.2
    environment:
      - discovery.type=single-node
    ports:
      - "9200:9200"

  logstash:
    image: docker.elastic.co/logstash/logstash:7.10.2
    volumes:
      - ./logstash/config:/usr/share/logstash/config
    ports:
      - "5000:5000"
    depends_on:
      - elasticsearch

  kibana:
    image: docker.elastic.co/kibana/kibana:7.10.2
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

Logstash Configuration

Create a logstash/config/logstash.conf file with the following content:

input {
  tcp {
    port => 5000
    codec => json_lines
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "spring-boot-logs-%{+YYYY.MM.dd}"
  }
}

This Logstash configuration specifies the input (TCP on port 5000) and the output (Elasticsearch).

Step 5: Run the Elastic Stack

Open a terminal, navigate to your project’s root directory, and run:

docker-compose up

This command will start Elasticsearch, Logstash, and Kibana.

Step 6: Verify Logging in Spring Boot

Run your Spring Boot application, and it will start sending logs to Logstash.

Access Kibana at http://localhost:5601, and set up an index pattern for spring-boot-logs-*. You can now explore and visualize your logs.

Conclusion

Congratulations! You have successfully set up logging in a Spring Boot application using the Elastic Stack. This centralized logging solution allows for better monitoring and troubleshooting across multiple microservices. Explore further customization options, such as adding additional fields to logs or configuring specific log levels.