vRealize Log Insight Cloud - Kubernetes Log Collection using Fluentd

In this blog, I will be showing the procedure on how to forward application logs deployed in Kubernetes Clusters to vRealize Log Insight Cloud a.k.a Log Intelligence using VMware’s fluentd plugin

Last year I had written a blog how to leverage various open-source plugin (Click Here)

What is Kubernetes?

Kubernetes is a portable, extensible an open-source platform for managing containerized workloads and services

What is Fluentd ?

Fluentd is an open-source data collector, which lets you unify data collection and consumption for better use and understanding of data.

Pre-requisites

·       A running Kubernetes Cluster

·       Admin access to the Cluster as we will be deploying fluentd in kube-system namespace

·       Application writes to "stdout" and "stderr" streams 

·       An understanding of VMware vRealize Log InsightCloud

Procedure  

There are 2 approaches that you can use to forward logs. The following section provides procedure for VMware Fluentd vRealize Log Insight Cloud plugin' as Daemon set and as Sidecar container

Approach 1 – Daemon Set

Steps

1.   Generate vRealize Log Insight Cloud API Key from here

2.   SSH to the Kubernetes Master Node

·      Procedure assumes you are doing it on Master. If you want to use kubeconfig file you can do that as well

3.   Create a configMap in Kubernetes for fluent.conf

kubectl -n kube-system create configmap --from-file=fluent.conf

4.   Update the fluent.conf file. You can refer my conf file as a sample

Fluent_DS.Conf

5.   Create Daemonset YAML file and name file as lint-fluent.yml with following configuration

apiVersion: apps/v1

kind: DaemonSet

metadata:

  name: fluentd-lint-logging

  namespace: kube-system

  labels:

    k8s-app: fluentd-lint-logging

    app: fluentd-lint-logging

    version: v1

    kubernetes.io/cluster-service: "true"

spec:

 selector:

   matchLabels:

     name: fluentd-lint-logging

 template:

   metadata:

     labels:

       name: fluentd-lint-logging

       app: fluentd-lint-logging

       version: v1

       kubernetes.io/cluster-service: "true"

   spec:

     serviceAccount: fluentd-lint-logging

     serviceAccountName: fluentd-lint-logging

     tolerations:

     - key: node-role.kubernetes.io/master

       effect: NoSchedule

     containers:

     - name: fluentd-lint

       image: docker.io/vmware/log-intelligence-fluentd

       command: ["fluentd"]

       env:

       - name: FLUENTD_ARGS

         value: --no-supervisor -q

       resources:

         limits:

           memory: 500Mi

         requests:

           cpu: 100m

           memory: 200Mi

       volumeMounts:

       - name: varlog

         mountPath: /var/log

       - name: varlogcontainers

         mountPath: /var/log/containers

       - name: varlibdockercontainers

         mountPath: /var/lib/docker/containers

         readOnly: true

       - name: config-volume

         mountPath: /etc/fluent

     volumes:

     - name: varlog

       hostPath:

         path: /var/log

     - name: varlogcontainers

       hostPath:

         path: /var/log/containers

     - name: varlibdockercontainers

       hostPath:

         path: /var/lib/docker/containers

     - name: config-volume

       configMap:

         name: lint-fluent-config

     - name: lint-fluent-volume

       emptyDir: {}

     - name: var-logs

       emptyDir: {}

6.   Last step is to apply the configuration to the cluster which will create new daemonset 

kubectl apply -f lint-fluent.yml

7.   If everything goes well, you can see fluentd pods running in the kube-system namespace. You can log in to vRealize Log Insight Cloud and navigate to Explore Log and search for the Kubernetes logs by using filter log_type contains kubernetes

Approach 2 – Side Car

Steps

1.   Generate vRealize Log Insight Cloud API Key from here

2.   SSH to the Kubernetes Master Node

·      Procedure assumes you are doing it on Master. If you want to use kubeconfig file you can do that as well

3.   Create a configMap in Kubernetes for fluent.conf

kubectl create configmap --from-file=fluent.conf

4.   Update the fluent.conf file with given configuration

Fluent_Sidecar.Conf

5.   Add a sidecar fluentd container in deployment.yaml for your application

·             Replace the following place holders with respective values

·             Application Name, Service Name, Product Name, Container Name, Volume Name

apiVersion: apps/v1

kind: Deployment

metadata:

 name: test

 namespace: default

 labels:

   app: lintdemo-nginx

spec:

 replicas: 2

 revisionHistoryLimit: 2

 strategy:

   type: RollingUpdate

   rollingUpdate:

     maxUnavailable: 40%

 selector:

   matchLabels:

     app: lintdemo-nginx

 template:

   metadata:

     labels:

       app: lintdemo-nginx

   spec:

     containers:

     - name: fluentd-lint

       image: docker.io/vmware/log-intelligence-fluentd

       command: ["fluentd"]

       env:

       - name: FLUENTD_ARGS

         value: --no-supervisor -q

       resources:

         limits:

           memory: 500Mi

         requests:

           cpu: 100m

           memory: 200Mi

       volumeMounts:

       - name: varlog

         mountPath: /var/log

       - name: varlogcontainers

         mountPath: /var/log/containers

       - name: varlibdockercontainers

         mountPath: /var/lib/docker/containers

         readOnly: true

       - name: config-volume

         mountPath: /etc/fluent

     - name: lintdemo-nginx

       image: nginx

       ports:

       - containerPort: 80

         name: nginx

       volumeMounts:

       - mountPath: /var/log

         name: var-logs

       imagePullPolicy: Always

     volumes:

     - name: varlog

       hostPath:

         path: /var/log

     - name: varlogcontainers

       hostPath:

         path: /var/log/containers

     - name: varlibdockercontainers

       hostPath:

         path: /var/lib/docker/containers

     - name: config-volume

       configMap:

         name: lint-fluent-config

     - name: var-logs

       emptyDir: {}

6.   Last step is to apply the configuration to deploy the application along with fluentd side car container

kubectl apply -f deployment.yml

7.   If everything goes well, you will see logs from the newly created pod You can login to vRealize Log Insight Cloud and navigate to Explore Log and search for the Kubernetes logs by using filter kubernetes_pod_name contains lintdemo

Conclusion

VMware vRealize Log Insight Cloud makes it quite easy to forward Kubernetes logs which allows admins/application owners to troubleshoot/application monitoring

Getting Started

For a free trial, you can click here or reach out to your account team

To learn more about vRealize Log Insight Cloud please visit here