How to create S3 notification events using SQS via Terraform

S3 (Simple Storage Service) makes it possible to notify through events when an action occurs within a Bucket or in a specific folder. In other words, it works as a listener. Therefore, any action that takes place on a source, an event notification will be sent to a destination.

What would those actions be?

Any actions that takes place within a S3 Bucket such as creating objects, folders, removing files, restoring files and more.

Destinations

For each event notification configuration, there must be a destination. For this destination, information about each action will be sent, for example:

A new file has been created in a specific folder, so information about the file will be sent, such as the creation date, file size, event type, file name, and more. Remembering that in this process, the content of the file is not sent, okay?

There are 3 types of destinations:

1. Lambda

2. SNS

3. SQS

Understanding how it works

In this post we are going to create an event notification settings in an S3 Bucket, simulating an action and understanding the final behavior.

We could create this setting via console but for good practice reasons, we'll use Terraform as IaC tool. For those who aren't very familiar with Terraform, follow this tutorial on Getting Started using Terraform on AWS.

In the next step, we will create a flow simulating the image below. We'll set in S3 Bucket for every file created within files/ folder, a notification event will be sent to a SQS queue.

Creating Terraform files

Create a folder called terraform/ in your project and from now on, all .tf files will be created inside it.

Now, create a file called vars.tf where we're going to store the variables that will be used and paste the content below to this file.

variable "region" {
  default = "us-east-1"
  type = string
}

variable "bucket" {
  type = string
}

Create a file called provider.tf , where we will add the provider settings, which will be AWS. This means, Terraform will use AWS as the cloud to create the resources and will download the required plugins on startup. Copy the code below to the file.

provider "aws" {
  region  = "${var.region}"
}

Create a file called s3.tf , where we'll add the settings for creating a new S3 Bucket that will be used for this tutorial.

resource "aws_s3_bucket" "s3_bucket_notification" {
  bucket = "${var.bucket}"
}

Now, create a file called sqs.tf , where we'll add the settings for creating an SQS queue and some permissions according to the code below:

resource "aws_sqs_queue" "s3-notifications-sqs" {

  name = "s3-notifications-sqs"
  policy = <<POLICY
  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": "*",
        "Action": "sqs:SendMessage",
        "Resource": "arn:aws:sqs:*:*:s3-notifications-sqs",
        "Condition": {
          "ArnEquals": {
           "aws:SourceArn":     
              "${aws_s3_bucket.s3_bucket_notification.arn}" 
           }
        }
      }
    ]
  }
  POLICY
}

Understanding code above

In the code above, we're creating an SQS and adding some policy settings, see more details:

1. SQS name will be s3-notifications-sqs, detailed value in name field

2. In the policy field, we define a policy that allows S3 sending messages notification to SQS. Notice that we're referencing the Bucket S3 via ARN in the snippet ${aws_s3_bucket.s3_bucket_notification.arn}

For the last file, let's create the settings that allows sending event notifications from S3 Bucket to an SQS. Therefore, create s3_notification.tf file and add the code below:

resource "aws_s3_bucket_notification" "s3_notification" {
  bucket = aws_s3_bucket.s3_bucket_notification.id

  queue {
    events    = ["s3:ObjectCreated:*"]
    queue_arn = aws_sqs_queue.s3-notifications-sqs.arn
    filter_prefix = "files/"
  }
}

Understanding code above

1. In the code above, we are creating a resource called aws_s3_bucket_notification which will be responsible for enabling notifications from an S3 Bucket.

2. In the bucket field, we are referring to the S3 bucket setting located on s3.tf file.

3. The block queue contains some settings such as:

• events: Is the event type of the notification. In this case, for ObjectCreated type events. Notifications will be sent only for created objects. For deleted objects, there will be no notifications. It helps to restrict some types of events.

• queue_arn: Refers to the SQS defined in the sqs.tf file.

• filter_prefix: This field defines the folder where we want notifications to be triggered. In the code, we set the folder files/ to be the trigger location when the files are created. Summarizing, for all files created within folder files/ , a notification will be sent to the SQS defined in the queue_arn field.

Running Terraform

Init Terraform

terraform init

Running Plan

The plan makes it possible to verify which resources will be produced, in this case it is necessary to pass the value of the bucket variable for its creation in S3.

terraform plan -var bucket = 'type the bucket name'

Running Apply command

In this step, the creation of resources will be applied. Remember to pass the name of the bucket you want to create into the bucket variable, and the bucket name must be unique.

terraform apply -var bucket = 'type the bucket name'

Simulating an event notification

After running the previous steps and creating the resources, we will manually upload a file in the files/ folder to the bucket that was created.

Via console, access the Bucket created in S3 and create a folder called files. Inside it, load any file.

Uploading file

After loading the file in the files/ folder, access the created SQS. You'll see some available messages. Usually 3 messages will be available in queue because after creating S3 notification events settings, a test message is sent. The second one happens when we create a folder and for the last, the message related to the file upload.

It's done, we have an event notification created!.

References:

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_notification

https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications.html

Coffee and tips Github repository: https://github.com/coffeeandtips/s3-bucket-notification-with-sqs-and-terraform

---

Books to study and read

If you want to learn more about and reach a high level of knowledge, I strongly recommend reading the following book(s):

Terraform: Up & Running: Writing Infrastrucutre as Code is a book focused on how to use Terraform and its benefits. The author sought to make comparisons with several other IAC (Infrastructure as code) tools such as Ansible and Cloudformation (IAC native to AWS) and especially how to create and provision different resources for multiple cloud services. Currently, Terraform is the most used tool in software projects for creating and managing resources in cloud services such as AWS, Azure, Google Cloud and many others. If you want to be a complete engineer, I strongly recommend learning about it.

AWS Cookbook is a practical guide containing 70 familiar recipes about AWS resources and how to solve different challenges. It's a well-written, easy-to-understand book covering key AWS services through practical examples. AWS or Amazon Web Services is the most widely used cloud service in the world today, if you want to understand more about the subject to be well positioned in the market, I strongly recommend the study.

Well that’s it, I hope you enjoyed it!