To have a Kubernetes cluster up and running is pretty easy these days. However, when you start to use it and deploy some applications, you might expect some issues over time. Kubernetes being a distributed system is not easy to troubleshoot. You need a proper monitoring solution, and because the Prometheus is CNCF project as Kubernetes, it is probably the best fit. In this post, I will show you how to get the Prometheus running and start monitoring your Kubernetes cluster in 5 minutes.
CoreOS introduced operators as business logic in the first place. I wrote about Elasticsearch operator and how operator works a few months ago so you might check it out. In my opinion, operators are the best way to deploy stateful applications on Kubernetes.
CoreOS team also created Prometheus operator for deploying Prometheus on top of Kubernetes. Here is the official operator workflow and relationships view:
From the picture above you can see that you can create a
ServiceMonitor resource which will scrape the Prometheus metrics from the defined set of pods. It instructs Prometheus to watch on a new target. For example, if you have a frontend app which exposes Prometheus metrics on port
web, you can create a service monitor which will configure the Prometheus server automatically:
apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: frontend-app labels: app: frontend-app release: prom spec: namespaceSelector: any: true selector: matchLabels: app: frontend-app endpoints: - port: web interval: 10s
release: prom is the Helm release name that I used to install the Prometheus operator below!
For Prometheus installation use the official
prometheus-operator Helm chart. This chart has many options, so I encourage you to take a look at default values file and override some values if needed. Among other services, this chart installs Grafana and exporters ready to monitor your cluster.
Let's enable persistent storage for all Prometheus components and also expose Grafana with ingress. I prepared a custom values file to use for installation with Helm:
⚡ cat > custom-values.yaml <<EOF # Depending on which DNS solution you have installed in your cluster enable the right exporter coreDns: enabled: false kubeDns: enabled: true alertmanager: alertmanagerSpec: storage: volumeClaimTemplate: spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 10Gi prometheus: prometheusSpec: storageSpec: volumeClaimTemplate: spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 10Gi grafana: adminPassword: "YourPass123#" ingress: enabled: true annotations: kubernetes.io/ingress.class: nginx kubernetes.io/tls-acme: "true" hosts: - grafana.test.akomljen.com tls: - secretName: grafana-tls hosts: - grafana.test.akomljen.com persistence: enabled: true accessModes: ["ReadWriteOnce"] size: 10Gi EOF
Installing Prometheus operator and Prometheus with all dependencies is just one command now:
⚡ helm install \ --name prom \ --namespace monitoring \ -f custom-values.yaml \ stable/prometheus-operator
NOTE: Kubernetes 1.10+ with Beta APIs and Helm 2.10+ are required!
When you install the Prometheus operator, the new Custom Resource Definition or CRD gets created. You can check for existing CRDs with this command:
⚡ kubectl get crd NAME AGE alertmanagers.monitoring.coreos.com 10m prometheuses.monitoring.coreos.com 10m prometheusrules.monitoring.coreos.com 10m servicemonitors.monitoring.coreos.com 10m
Also, to see what each of those does check official design doc. The Prometheus operator manages all of them.
Wait a few minutes and the whole stack should be up and running. Check for all pods in monitoring namespace:
⚡ kubectl get pods -n monitoring NAME READY STATUS RESTARTS AGE alertmanager-prom-prometheus-operator-alertmanager-0 2/2 Running 0 10m prom-grafana-6c7c9cf8fc-szkpv 3/3 Running 0 10m prom-kube-state-metrics-5648859669-lwxmr 1/1 Running 0 10m prom-prometheus-node-exporter-2896z 1/1 Running 0 10m prom-prometheus-node-exporter-2dwcv 1/1 Running 0 10m prom-prometheus-node-exporter-6fk24 1/1 Running 0 10m prom-prometheus-node-exporter-9kwvh 1/1 Running 0 10m prom-prometheus-node-exporter-9w6bc 1/1 Running 0 10m prom-prometheus-node-exporter-g9v4v 1/1 Running 0 10m prom-prometheus-node-exporter-jqpjk 1/1 Running 0 10m prom-prometheus-node-exporter-ph2xz 1/1 Running 0 10m prom-prometheus-node-exporter-q46bn 1/1 Running 0 10m prom-prometheus-node-exporter-tfdzs 1/1 Running 0 10m prom-prometheus-operator-operator-6d645ff4d6-lkc98 1/1 Running 0 10m prometheus-prom-prometheus-operator-prometheus-0 3/3 Running 1 10m ⚡ kubectl get ingress -n monitoring NAME HOSTS ADDRESS PORTS AGE prom-grafana grafana.test.akomljen.com 80, 443 10m
Grafana default dashboards are present when you log in. Of course, you can always update them, or create a completely new dashboard if you need to. In the example below you can see how the node view looks like:
If you want to access other services you can forward the port to localhost, for example:
# Alert manager ⚡ kubectl port-forward -n monitoring alertmanager-prom-prometheus-operator-alertmanager-0 9093 # Prometheus server ⚡ kubectl port-forward -n monitoring prometheus-prom-prometheus-operator-prometheus-0 9090
When you expose Prometheus server to your localhost, you can also check for alerts at
http://localhost:9090/alerts. You could also use ingress to expose those services, but, they don't have authentication so you would need something like OAuth Proxy in front.
It is almost impossible not to experience any issues with Kubernetes cluster once you start to use it. This monitoring setup will help you along the way. Of course, this is only one part of monitoring, and it's mostly cluster related. Many cloud-native applications have Prometheus support out of the box, so getting application metrics should be the next step. I will cover this in some future blog post. Stay tuned for the next one.