Tutorial that expands on this previous post demonstrating how to take data in to an AWS Lambda function and write the data in a consistent file-naming format to AWS Simple Storage Service (S3), demonstrating somewhat of an “archiving” functionality.
Background
This post is an expansion of the previous AWS Lambda Post describing how to secure sensitive information in your AWS Lambda project/environment. This post expands further and demonstrates how an AWS Lambda function, when sent data along with a request, can write the data to an S3 bucket using a naming convention in line with the date/time the request was received, demonstrating somewhat of an archiving functionality. Although there are ingest services in AWS that can do this more natively using AWS services, this is a good proof of concept to demonstrate how to link an AWS Lambda function to S3.
As previously, all instructions within are assumed for an Ubuntu-16.04 installation. While the commands may also work on various other operating systems of the Unix type, your mileage may vary.
Like the previous tutorial, you will also need an AWS account to deploy the functionality into and test using AWS S3 (and the previous KMS functionality). As a note, S3 is very inexpensive for file storage.
Prerequisites
As before, this posts assume you follow the previous post steps for setting up the environment and having a basic code base. Please follow the previous post in its entirety before proceeding with this post.
Exploring S3
Fist and foremost, we’ll detail some commands that you can use to explore your S3 environment. Assuming you already have the AWS CLI configured (per the Prerequisites section above), you can run S3 commands to explore how to interact with the AWS S3 service:
Coding in Serverless
Now that we have some experience interacting with AWS S3 via the command line, let’s code our application to also interact with the service. We are going to update the code base so that whenever the Lambda function receives a request, if the request contains data, we record the data to a file in an S3 bucket, with a filename corresponding to the date/time the request was made.
We are going to assume that the bucket we wish to use already exists, so let’s go ahead and create one from the command line like we did before:
Next, we need to set up permissions for the Lambda function to be able to write to the S3 bucket, specify a custom parameter containing the bucket name to reduce code duplication, and add a handler that will ultimately write the data to S3. Note that the ARN for the S3 bucket is constructed using standard notation and the custom bucket name (which is one of the reasons why bucket names must be unique):
We’ll install a library that will make it easier to format dates and times:
Require the library at the top of your handler.js
file:
We’ll now add some code that will check for incoming data and, if present, write the data to
the S3 bucket using a file name that includes the date and time the data was received. Add the
following function declaration somewhere in the handler.js
file:
Then, update the hello
function to reflect the following Promise chain and functionality:
The Lambda function now inspects the incoming data element (event
) to ensure it has data (if not,
it throws an exception given that the intent of this exercise is to store data in S3). Once it verifies
data exists, it writes the data to a file named with the date/time that the event occurred and returns
the data location in S3 to the requestor. Let’s run the Lambda function locally again and inspect the
output:
It appears the data has been successfully stored - let’s check the S3 bucket via the command line tools to ensure we can see the file created:
Now, let’s run a final test to make sure the function fails as expected when no data is provided:
It looks like things are working as expected - let’s deploy the function to AWS and test it/verify that it works and stores the files as expected:
Everything is working great! Let’s perform some clean-up to ensure we aren’t billed for the usage moving forward:
Credit
The above tutorial was pieced together with some information from the following sites/resources: