kubernetes, efk, monitoring, logs

Get Kubernetes Logs with EFK Stack in 5 Minutes

Last week I wrote about getting Kubernetes cluster metrics with Prometheus. Metrics are only one part of the story. Logs are important and luckily we have a great set of tools that will help you to create simple and easy logging solution. In this post, I will show you how to start monitoring Kubernetes logs in 5 minutes with EFK stack (Elasticsearch, Fluent Bit, and Kibana) deployed with Helm and operators.

Previous post

Fluentd vs Fluent Bit

EFK stack usually refers to Elasticsearch, Fluentd and Kibana. But, I decided to go with Fluent Bit, which is much lighter and it has built-in Kubernetes support. Fluent Bit can read Kubernetes or Docker log files from the file system or through Systemd journal, enrich logs with Kubernetes metadata, deliver logs to third-party storage services like Elasticsearch, InfluxDB, HTTP, etc.

Fluent Bit is a Lightweight Data Forwarder for Fluentd. Fluent Bit is specifically designed for forwarding the data from the edge to Fluentd aggregators.

Using Fluent Bit doesn't mean that you cannot put Fluentd in the mix also. For the sake of simplicity, I excluded it from this post. Here are some major differences between Fluentd and Fluent Bit from official FAQ page:

- Fluentd Fluent Bit
Scope Servers Containers / Servers / Embedded Systems
Language C & Ruby C
Memory ~40MB ~450KB
Dependencies Built as a Ruby Gem, it requires a certain number of gems and Ruby interpreter Zero dependencies
Plugins More than 650 plugins available Around 30 plugins available
License Apache License v2.0 Apache License v2.0

Elasticsearch Operator

For logs storage, you will need Elasticsearch cluster. The best way to deploy stateful applications on Kubernetes is using operators. I mentioned them in almost every post related to stateful applications. Operators are the hot topic and you should start using them. Thanks to CoreOS team we have some great operators like Prometheus, Etcd and now Vault.

You can check my blog post about Elasticsearch operator for more details. My preferred way to deploy applications on Kubernetes is using Helm. For this post, I created my own Helm charts repository where I can put some charts when needed and to not wait for a pull request to get merged into official charts repository.

So, let's add the new Helm repository and install the Elasticsearch operator:

⚡ helm repo add akomljen-charts \
    https://raw.githubusercontent.com/komljen/helm-charts/master/charts/

⚡ helm install --name es-operator \
    --namespace logging \
    --set rbac.create=false \
    akomljen-charts/elasticsearch-operator
  
⚡ kubectl get pods -n logging
NAME                                                  READY     STATUS    RESTARTS   AGE
es-operator-elasticsearch-operator-568667668b-b6z2v   1/1       Running   0          33s

When you install the Elasticsearch operator you will get the new Custom Resource Definition or CRD. You can check that with this command:

⚡ kubectl get CustomResourceDefinition
NAME                                         AGE
elasticsearchclusters.enterprises.upmc.com   2m

Glue Everything Together

So, how to glue all components together? I created an umbrella Helm chart which combines multiple charts into one installation unit. This means that we have only one property file and one Helm package to install and run the whole stack. You can check for default values file and adjust values for your environment. Let's install EFK chart:

⚡ helm install --name efk \
    --namespace logging \
    akomljen-charts/efk
    
⚡ kubectl get pods -n logging
NAME                                                  READY     STATUS    RESTARTS   AGE
efk-kibana-6cf88598b6-xlkv2                           1/1       Running   0          5m
es-client-efk-cluster-545d4ddb9-mp8d6                 1/1       Running   0          5m
es-data-efk-cluster-default-0                         1/1       Running   0          5m
es-master-efk-cluster-default-0                       1/1       Running   0          5m
es-operator-elasticsearch-operator-568667668b-b6z2v   1/1       Running   0          6m
fluent-bit-pglgq                                      1/1       Running   0          5m

After a few minutes, all services should be up and running. To search for the logs I will use kubectl port-forward command on Kibana pod to access it. Exposing Kibana with Ingress is a potential security risk because Kibana doesn't have any authentication method by default. Use Ingress only on test cluster or deploy oauth2-proxy for other authentication providers like GitHub or Google.

⚡ kubectl port-forward efk-kibana-6cf88598b6-xlkv2 5601 -n logging

Open your web browser at http://localhost:5601 and you should see Kibana dashboard:

kibana

Then, go to Discover menu item, configure the index to kubernetes_cluster*, choose a timestamp and Kibana is ready. You should see all logs from all namespaces on your Kubernetes cluster. Rich Kubernetes metadata will be there also, for example:

kibana_app_logs

There you go, all Kubernetes logs in one place!

Summary

It is really nice to see that metrics and logs monitoring is easy with Kubernetes. Of course, each solution will need some tweaks, but getting started was never so easy. Stay tuned for the next one.