Wednesday, December 25, 2019

vRealize Log Insight Cloud - Kubernetes Log Collection using Fluentd




In this blog, I will be showing the procedure on how to forward application logs deployed in Kubernetes Clusters to vRealize Log Insight Cloud a.k.a Log Intelligence using VMware’s fluentd plugin

Last year I had written a blog how to leverage various open-source plugin (Click Here)

What is Kubernetes?

Kubernetes is a portable, extensible an open-source platform for managing containerized workloads and services



What is Fluentd ?

Fluentd is an open-source data collector, which lets you unify data collection and consumption for better use and understanding of data.



Pre-requisites


·       A running Kubernetes Cluster
·       Admin access to the Cluster as we will be deploying fluentd in kube-system namespace
·       Application writes to "stdout" and "stderr" streams 
·       An understanding of VMware vRealize Log InsightCloud

Procedure  

There are 2 approaches that you can use to forward logs. The following section provides procedure for VMware Fluentd vRealize Log Insight Cloud plugin' as Daemon set and as Sidecar container

Approach 1 – Daemon Set




Steps

1.   Generate vRealize Log Insight Cloud API Key from here
2.   SSH to the Kubernetes Master Node
·      Procedure assumes you are doing it on Master. If you want to use kubeconfig file you can do that as well
3.   Create a configMap in Kubernetes for fluent.conf

kubectl -n kube-system create configmap --from-file=fluent.conf

4.   Update the fluent.conf file. You can refer my conf file as a sample

5.   Create Daemonset YAML file and name file as lint-fluent.yml with following configuration

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-lint-logging
  namespace: kube-system
  labels:
    k8s-app: fluentd-lint-logging
    app: fluentd-lint-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
 selector:
   matchLabels:
     name: fluentd-lint-logging
 template:
   metadata:
     labels:
       name: fluentd-lint-logging
       app: fluentd-lint-logging
       version: v1
       kubernetes.io/cluster-service: "true"
   spec:
     serviceAccount: fluentd-lint-logging
     serviceAccountName: fluentd-lint-logging
     tolerations:
     - key: node-role.kubernetes.io/master
       effect: NoSchedule
     containers:
     - name: fluentd-lint
       image: docker.io/vmware/log-intelligence-fluentd
       command: ["fluentd"]
       env:
       - name: FLUENTD_ARGS
         value: --no-supervisor -q
       resources:
         limits:
           memory: 500Mi
         requests:
           cpu: 100m
           memory: 200Mi
       volumeMounts:
       - name: varlog
         mountPath: /var/log
       - name: varlogcontainers
         mountPath: /var/log/containers
       - name: varlibdockercontainers
         mountPath: /var/lib/docker/containers
         readOnly: true
       - name: config-volume
         mountPath: /etc/fluent
     volumes:
     - name: varlog
       hostPath:
         path: /var/log
     - name: varlogcontainers
       hostPath:
         path: /var/log/containers
     - name: varlibdockercontainers
       hostPath:
         path: /var/lib/docker/containers
     - name: config-volume
       configMap:
         name: lint-fluent-config
     - name: lint-fluent-volume
       emptyDir: {}
     - name: var-logs
       emptyDir: {}

6.   Last step is to apply the configuration to the cluster which will create new daemonset 

kubectl apply -f lint-fluent.yml










7.   If everything goes well, you can see fluentd pods running in the kube-system namespace. You can log in to vRealize Log Insight Cloud and navigate to Explore Log and search for the Kubernetes logs by using filter log_type contains kubernetes

















Approach 2 – Side Car




Steps

1.   Generate vRealize Log Insight Cloud API Key from here
2.   SSH to the Kubernetes Master Node
·      Procedure assumes you are doing it on Master. If you want to use kubeconfig file you can do that as well
3.   Create a configMap in Kubernetes for fluent.conf

kubectl create configmap --from-file=fluent.conf

4.   Update the fluent.conf file with given configuration

5.   Add a sidecar fluentd container in deployment.yaml for your application
·             Replace the following place holders with respective values
·             Application Name, Service Name, Product Name, Container Name, Volume Name

apiVersion: apps/v1
kind: Deployment
metadata:
 name: test
 namespace: default
 labels:
   app: lintdemo-nginx
spec:
 replicas: 2
 revisionHistoryLimit: 2
 strategy:
   type: RollingUpdate
   rollingUpdate:
     maxUnavailable: 40%
 selector:
   matchLabels:
     app: lintdemo-nginx
 template:
   metadata:
     labels:
       app: lintdemo-nginx
   spec:
     containers:
     - name: fluentd-lint
       image: docker.io/vmware/log-intelligence-fluentd
       command: ["fluentd"]
       env:
       - name: FLUENTD_ARGS
         value: --no-supervisor -q
       resources:
         limits:
           memory: 500Mi
         requests:
           cpu: 100m
           memory: 200Mi
       volumeMounts:
       - name: varlog
         mountPath: /var/log
       - name: varlogcontainers
         mountPath: /var/log/containers
       - name: varlibdockercontainers
         mountPath: /var/lib/docker/containers
         readOnly: true
       - name: config-volume
         mountPath: /etc/fluent
     - name: lintdemo-nginx
       image: nginx
       ports:
       - containerPort: 80
         name: nginx
       volumeMounts:
       - mountPath: /var/log
         name: var-logs
       imagePullPolicy: Always
     volumes:
     - name: varlog
       hostPath:
         path: /var/log
     - name: varlogcontainers
       hostPath:
         path: /var/log/containers
     - name: varlibdockercontainers
       hostPath:
         path: /var/lib/docker/containers
     - name: config-volume
       configMap:
         name: lint-fluent-config
     - name: var-logs
       emptyDir: {}

6.   Last step is to apply the configuration to deploy the application along with fluentd side car container

kubectl apply -f deployment.yml









7.   If everything goes well, you will see logs from the newly created pod You can login to vRealize Log Insight Cloud and navigate to Explore Log and search for the Kubernetes logs by using filter kubernetes_pod_name contains lintdemo















Conclusion


VMware vRealize Log Insight Cloud makes it quite easy to forward Kubernetes logs which allows admins/application owners to troubleshoot/application monitoring

Getting Started


For a free trial, you can click here or reach out to your account team

To learn more about vRealize Log Insight Cloud please visit here