Wednesday, September 5, 2018

Kubernetes Log forwarding to VMware Log Intelligence using Fluentd

In this blog I will walk you through how to use fluentd to forward logs from Kubernetes deployed pods(containers)

Pre-requisites


  • A running Kubernetes Cluster. My setup has with Kubernetes 1.11.1 on CentOS VMs on vSphere. Click Here for detailed instructions 
  • Admin access to the Cluster as we will be deploying fluentd in kube-system name space
  • Application writes to "stdout" and "stderr" streams 
  • An understanding of VMware Log Intelligence. Click Here to read through if you haven't done it yet 


Getting Started


Before to get started, make sure you understand or have a basic idea about the following concepts from Kubernetes (Click Here :

Node            


A node is a worker machine in Kubernetes, previously known as a minion. A node may be a VM or physical machine, depending on the cluster. Each node has the services necessary to run pods and is managed by the master components…

Pod


A pod is a group of one or more containers (such as Docker containers), the shared storage for those containers, and options about how to run the containers. Pods are always co-located and co-scheduled, and run in a shared context…

DaemonSet


A DaemonSet ensures that all (or some) nodes run a copy of a pod. As nodes are added to the cluster, pods are added to them. As nodes are removed from the cluster, those pods are garbage collected. Deleting a DaemonSet will clean up the pods it created. 

We will be using fluentd kubernetes daemon set (Click Here) which will be reading log files for the containers and sending it to VMware Log Intelligence


Options 


There are 2 ways we can send kubernetes logs to VMware Log Intelligence.

Option 1 - Using Syslog fluentd daemon set via Data Collector using syslog protocol


This is the most simplest and the quickest way. You will need to have Data Collector deployed. Click Here to see how to deploy Data Collector
















Steps 


Copy the yaml file for syslog (Click Here) and save it locally
Modify the yaml file and to add the Data Collector IP which accepts connections on syslog port (514)
Create fluentd daemon set by executing following command 

kubectl create -f fluentd-syslog.yaml
















Once deployed and configured successfully you will see logs from fluentd pod in VMware Log Intelligence














Option 2 - Using Custom daemon set directly to VMware Log Intelligence via HTTPS protocol


You will need to modify the daemon set and install http-out-ext fluentd plugin which can forward logs directly to VMware Log Intelligence using API keys

















Steps


First Step is to create your own docker image 

I will be using the debian image as its recommended for Production environments. https://github.com/fluent/fluentd-kubernetes-daemonset/tree/master/docker-image/v1.2/debian-syslog
From the documentation - The following repository expose images based on Alpine Linux and Debian. For production environments, we strongly suggest to use Debian images.

Clone the github repo. Install git (if you don't have it already) and execute following command









Navigate to "fluentd-kubernetes-daemonset/docker-image/v1.2/debian-syslog/conf"






Modify the fluent.conf file to following 


Before













After




















You will notice that we have added is two environment variables. It is similar to what we have for the standalone fluentd setup. Click Here to view the standalone fluentd installation

Next we will need to modify the docker file to install the http-out plugin "fluentd-kubernetes-daemonset/docker-image/v1.2/debian-syslog/Dockerfile"

&& gem install fluent-plugin-out-http-ext -v 0.1.10 \


Before 





















After






















docker build -t docker.io/mydockerhubusername/reponame:v1 ./

It should take a couple of mins 

If you want to push it registry or you can use local image

You can modify the existing yaml file for syslog with following yaml file. Please ensure to generate API Key from VMware Log Intelligence UI























Execute following command. In my case name of the file is mm-fluentd.yaml

kubectl create -f mm-fluentd.yaml








Once deployed and configured successfully you will see logs from fluentd pod in VMware Log Intelligence














Happy Logging !!!


References