Tuesday, January 12, 2021

Deploy AWS Lambda to forward AWS Service logs to vRealize Log Insight Cloud

Recently vRealize Log Insight Cloud announced Multi-Cloud support which allows you to forward logs from AWS and Azure. For more details, you can refer to the blog post  

In this blog, I would demonstrate how to deploy AWS Lambda to forward the following AWS service logs to vRealize Log Insight Cloud using Terraform AWS provider

  • S3 Bucket Logs
  • S3 Bucket Events
  • CloudWatch Logs

Once you execute the Terraform scripts it will create 3 AWS Lambda which will get the logs flowing to vRealize Log Insight Cloud.





















What is Terraform


Terraform provides Infrastructure as a Code to provision and manage any cloud, infrastructure, or service

Pre-requisites


Following are the pre-requisites
  • Access to VMware vRealize Log Insight Cloud 
  • API Key for VMware vRealize Log Insight Cloud
  • AWS Access Key and ID
    • PowerUser role should be enough for creating the required resources.
  • AWS IAM role which has Lambda Service
    • It needs to exist before executing the script. 
  • S3 Bucket for which you want logs.
    • It needs to exist before executing the script.
  • Cloud Watch Log group for which you want logs.
    • Log group needs to exist before executing the script. 

Download the Lambda Package 


Download Lambda code by executing the below command from a terminal
wget https://github.com/vmware/vmware-log-collectors-for-public-cloud/releases/download/v1.0.3/Lambda.zip

Download Terraform Scripts 


Clone following Terraform Repo from my Github Terraform Repo

git clone https://github.com/munishpalmakhija/Terraform.git








Following are the directories for vRLI Cloud 
  • Deploy-vRLICloud-AWS-Lambda-S3BucketLogs
    • It creates AWS Lambda with S3 Trigger when a new object created event happens in the bucket.
  • Deploy-vRLICloud-AWS-Lambda-S3Events
    • It creates AWS Lambda with S3 Trigger when a new object created event happens in the bucket.
  • Deploy-vRLICloud-AWS-Lambda-CloudWatchLogs
    • It creates AWS Lambda with Cloudwatch Trigger (Subscription) 

Deploy AWS Lambda



Update the   terraform.tfvars for all 3 scripts


Navigate to the folder Deploy-vRLICloud-AWS-Lambda-S3BucketLogs & update the terraform.tfvars with your environment details


Navigate to the folder Deploy-vRLICloud-AWS-Lambda-S3Events & update the terraform.tfvars with your environment details


Navigate to the folder Deploy-vRLICloud-AWS-Lambda-CloudWatchLogs & update the terraform.tfvars with your environment details

Initialize Terraform

 

terraform init


















Create an Execution Plan 

terraform plan 

























Apply the config to the environment

terraform apply -auto-approve











Once it is successfully executed you will see AWS Lambda Function created as below. 















Similarly, you can execute the other 2 scripts for S3 Bucket Events and Cloudwatch Logs















You should be able to view logs by using the following filter

log_type starts with aws













1 comment: