Lambda Trigger on S3 File Upload Using Node Js and Write to Dyam Db

• Language Support
• Node.js
• Straight to S3 File Uploads in Node.js

This article was contributed by Volition Webberley

Will is a figurer scientist and is enthused by nigh all aspects of the engineering science domain. He is specifically interested in mobile and social computing and is currently a researcher in this area at Cardiff University.

Direct to S3 File Uploads in Node.js

English — 日本語に切り替える

Final updated March 09, 2022

Table of Contents

• Uploading straight to S3
• Overview
• Prerequisites
• Initial setup
• Directly uploading
• Running the app
• Summary

Web applications often crave the ability to allow users to upload files such as images, movies and athenaeum. Amazon S3 is a popular and reliable storage option for these files.

This article demonstrates how to create a Node.js application that uploads files directly to S3 instead of via a web awarding, utilising S3'southward Cross-Origin Resources Sharing (CORS) support. The Express spider web framework is used to facilitate request-handling in the examples below, but the procedure should be almost identical in any Node.js application.

Uploading direct to S3

A complete example of the code discussed in this commodity is available for direct utilise in this GitHub repository.

The main advantage of direct uploading is that the load on your application's dynos would be considerably reduced. Using app-side processes for receiving files and transferring to S3 can needlessly tie up your dynos and will mean that they will not be able to answer to simultaneous web requests every bit efficiently.

The awarding uses customer-side and app-side JavaScript for signing the requests. The bodily upload is carried out asynchronously so that you can decide how to handle your application's menses after the upload has completed (for case, you may wish to redirect users to some other page upon successful upload rather than a full folio refresh).

An case elementary account-editing scenario is used as a guide for completing the diverse steps required to accomplish the direct upload and to relate the application of this to a wider range of use-cases. More than information on this scenario is provided after.

Overview

S3 is comprised of a gear up of buckets, each with a globally unique name, in which individual files (known equally objects) and directories, tin be stored.

For uploading files to S3, y'all will demand an Admission Primal ID and a Hush-hush Access Key, which human activity as a username and password. The admission fundamental account will demand to accept sufficient access privileges to the target saucepan in society for the upload to be successful.

Delight come across the S3 Article for more data on this, creating buckets and handling your authentication keys.

In general, the method described in this article follows these uncomplicated steps:

• A file is selected for upload past the user in their web browser;
• The user's browser makes a request to your web application on Heroku, which produces a temporary signature with which to sign the upload asking;
• The temporary signed request is returned to the browser in JSON format;
• The browser then uploads the file directly to Amazon S3 using the signed request supplied by your Node.js application.

This guide includes data on how to implement the client-side and app-side code to course the complete organisation. After following the guide, you should have a working barebones system, assuasive your users to upload files to S3. All the same, it is ordinarily worth adding extra functionality to help improve the security of the system and to tailor information technology for your own particular uses. Pointers for this are mentioned in the advisable parts of the guide.

The signature generation on the server uses AWS'southward official SDK, as explained later on. Please see their documentation for data on the features of this SDK.

Prerequisites

• The Heroku CLI has been installed;
• Node.js has been installed;
• A Heroku application has been created for the current project;
• An AWS S3 bucket has been created;
• You take AWS hallmark keys that have write access to the bucket.

Initial setup

S3 setup

Y'all will now need to edit some of the permissions backdrop of the target S3 bucket and then that the last request has sufficient privileges to write to the saucepan. In a web-browser, sign in to the AWS console and select the S3 section. Select the appropriate bucket and click the Permissions tab. A few options are now provided on this folio (including Cake public access, Admission Control List, Bucket Policy, and CORS configuration).

Firstly, ensure that "Block all public admission" is turned off, and in particular turn off "Block public access to buckets and objects granted through new access command lists" and "Cake public access to buckets and objects granted through whatever access command lists" for the purposes of this project. Setting up the bucket in this mode allows us to read its contents without signed URLs, merely this may not be suitable for services running in production.

Side by side, you will demand to configure the bucket's CORS (Cantankerous-Origin Resource Sharing) settings, which will permit your awarding to access content in the S3 saucepan. Each dominion should specify a fix of domains from which admission to the bucket is granted and too the methods and headers permitted from those domains.

For this to piece of work in your application, click 'Edit' and enter the following JSON for the bucket's CORS settings:

            [     {         "AllowedHeaders": [             "*"         ],         "AllowedMethods": [             "GET",             "HEAD",             "POST",             "PUT"         ],         "AllowedOrigins": [             "*"         ],         "ExposeHeaders": []     } ]                      

Click 'Save changes' and shut the editor.

This tells S3 to allow any domain access to the bucket and that requests can contain any headers, which is more often than not fine for testing. When deploying, y'all should alter the 'AllowedOrigin' to but take requests from your domain.

If you wish to use S3 credentials specifically for this application, so more keys tin can be generated in the AWS business relationship pages. This provides farther security, since you tin can designate a very specific prepare of requests that this fix of keys are able to perform. If this is preferable to you, then y'all will need to configure your IAM users in the Edit bucket policy option in your S3 bucket. At that place are diverse guides on AWS's web pages detailing how this can be achieved.

App setup

If your app hasn't even so been setup, and so it is useful to do and then at this stage. To go started, create a directory somewhere on your local machine:

            $ mkdir NodeDirectUploader                      

Now create ii farther subdirectories of NodeDirectUploader/ to respectively contain your HTML pages and support files:

            $ cd NodeDirectUploader $ mkdir views $ mkdir public                      

Node's packet manager, npm, should have been installed by default along with Node and can be used to handle the installation and updates of the required packages for your app. To begin this, run Node's interactive package setup tool in the root of your app directory:

            $ npm init                      

The tool will ask some questions near your app, including its name, description, licensing, and version-control, and create a file called package.json in the app's root. This file uses your responses to maintain information about your app, which you tin can edit freehand equally you develop farther.

The same file tin be used to easily declare your app'southward dependencies, which will facilitate the deployment and share-ability of your app. To do so, edit package.json and add together a "dependencies" JSON object to contain the following bundle dependencies:

            {   "name": "NodeDirectUploader",   "version": "0.0.1",    ...    "dependencies": {     "aws-sdk": "2.ten",     "ejs": "two.ten",     "express": "four.x"   } }                      

These dependencies can then exist installed using npm:

            $ npm install                      

Use of these packages will become clear later, and installation of them in this manner allows for greater command of your per-app dependencies equally your apps grow.

Heroku setup

In order for your application to access the AWS credentials for signing upload requests, they volition need to be added equally configuration variables in Heroku:

            $ heroku config:set AWS_ACCESS_KEY_ID=30 AWS_SECRET_ACCESS_KEY=yyy Adding config vars and restarting app... done, v21     AWS_ACCESS_KEY_ID => xxx     AWS_SECRET_ACCESS_KEY => yyy                      

In addition to the AWS access credentials, set your target S3 bucket's name:

            $ heroku config:set S3_BUCKET=zzz Adding config vars and restarting app... done, v21     S3_BUCKET     => zzz                      

Using config vars is preferable over configuration files for security reasons. Avoid placing passwords and access keys directly in your application's code or in configuration files. Delight meet the article Configuration and Config Vars for more data.

Setting up local environment variables for your app is useful for running and testing your app locally. For more information, see the Set your local environs variables section of the Heroku Local commodity. Information on launching your app locally is provided afterward in this article.

Remember to add the .env file to your .gitignore, since this file should only be used for local testing.

Straight uploading

The processes and steps required to accomplish a straight upload to S3 will exist demonstrated through the utilise of a uncomplicated contour-editing scenario for the purposes of this article. This example will involve the user existence permitted to select an avatar image to upload and enter some basic information to exist stored as office of their account.

In this scenario, the following procedure will take place:

• The user is presented with a web page, containing elements encouraging the user to choose an image to upload every bit their avatar and to enter a username and their own name.
• An element is responsible for maintaining a preview of the chosen epitome by the user. By default, and if no paradigm is chosen for upload, a default avatar prototype is used instead (making the paradigm-upload effectively optional to the user in this scenario).
• When a user selects an image to be uploaded, the upload to S3 is handled automatically and asynchronously with the process described earlier in this article. The epitome preview is then updated with the selected image once the upload is complete and successful.
• The user is and so free to move on to filling in the rest of the information.
• The user and so clicks the "submit" button, which posts the username, name and the URL of the uploaded image to the Node application to be checked and/or stored. If no prototype was uploaded by the user earlier the default avatar image URL is posted instead.

Setting up the client-side code

No 3rd-party lawmaking is required to complete the implementation on the customer-side.

The HTML and JavaScript tin can now exist created to handle the file selection, obtain the asking and signature from your Node application, and so finally brand the upload request.

Firstly, create a file chosen account.html in your application's views/ directory and populate the caput and other necessary HTML tags appropriately for your application. In the trunk of this HTML file, include a file input and an chemical element that will contain status updates on the upload progress. In addition to this, create a course to allow the user to enter their username and full proper noun and a hidden input element to concord the URL of the called avatar image:

To see the completed HTML file, delight see the appropriate code in the companion repository.

            <input type="file" id="file-input"> <p id="status">Delight select a file</p> <img id="preview" src="/images/default.png">  <form method="Post" action="/relieve-details">   <input type="hidden" id="avatar-url" proper noun="avatar-url" value="/images/default.png">   <input type="text" name="username" placeholder="Username"><br>   <input type="text" proper name="full-name" placeholder="Full name"><br><br>   <input type="submit" value="Update profile"> </form>                      

The #preview element initially holds a default avatar image (which would become the user's avatar if a new image is not chosen), and the #avatar-url input maintains the current URL of the user's chosen avatar image. Both of these are updated by the JavaScript, discussed below, when the user selects a new avatar.

Thus when the user finally clicks the submit button, the URL of the avatar is submitted, along with the username and total name of the user, to your desired endpoint for server-side handling.

The client-side code is responsible for achieving two things:

1. Call up a signed request from the app with which the paradigm can exist PUT to S3
2. Actually PUT the image to S3 using the signed request

JavaScript's XMLHttpRequest objects can be created and used for making asynchronous HTTP requests.

To accomplish this, kickoff create a <script> block and write some code that listens for changes in the file input, in one case the document has loaded, and starts the upload process.

            (() => {   document.getElementById("file-input").onchange = () => {     const files = document.getElementById('file-input').files;     const file = files[0];     if(file == nil){       return alert('No file selected.');     }     getSignedRequest(file);   }; })();                      

The lawmaking as well determines the file object itself to be uploaded. If one has been selected properly, it proceeds to call a function to obtain a signed PUT request for the file. Next, therefore, write a function that accepts the file object and retrieves an appropriate signed asking for it from the app.

            office getSignedRequest(file){   const xhr = new XMLHttpRequest();   xhr.open('Go', `/sign-s3?file-name=${file.proper name}&file-type=${file.blazon}`);   xhr.onreadystatechange = () => {     if(xhr.readyState === 4){       if(xhr.status === 200){         const response = JSON.parse(xhr.responseText);         uploadFile(file, response.signedRequest, response.url);       }       else{         alert('Could not get signed URL.');       }     }   };   xhr.send(); }                      

If the proper noun (file.name) and/or mime type (file.type) of the file you upload contains special characters (such as spaces), then they should be encoded start (eastward.k. encodeURIComponent(file.name)).

The to a higher place function passes the file's name and mime blazon as parameters to the Get request since these are needed in the construction of the signed request, as volition be covered later in this article. If the retrieval of the signed request was successful, the function continues past calling a function to upload the actual file:

            function uploadFile(file, signedRequest, url){   const xhr = new XMLHttpRequest();   xhr.open('PUT', signedRequest);   xhr.onreadystatechange = () => {     if(xhr.readyState === four){       if(xhr.status === 200){         document.getElementById('preview').src = url;         document.getElementById('avatar-url').value = url;       }       else{         alert('Could not upload file.');       }     }   };   xhr.transport(file); }                      

This role accepts the file to be uploaded, the signed request, and generated URL representing the eventual retrieval URL of the avatar epitome. The latter two arguments will be returned every bit part of the response from the app. The function, if the request to S3 is successful, and so updates the preview element to the new avatar image and stores the URL in the hidden input so that it tin can be submitted for storage in the app.

At present, once the user has completed the rest of the form and clicked submit, the proper name, username, and avatar epitome can all exist posted to the same endpoint.

If you find that the page isn't working as you intend subsequently implementing the organization, then consider using console.log() to record any errors that are revealed inside the onreadystatechange function and use your browser'due south fault console to help diagnose the problem.

Information technology is good do to inform the user of any prolonged activity in whatever form of application (web- or device-based) and to brandish updates on changes. Therefore a loading indicator could be displayed between selecting a file and the upload being completed. Without this sort of information, users may doubtable that the page has crashed, and could try to refresh the page or otherwise disrupt the upload process.

Setting up the app-side Node lawmaking

This department discusses the use of Node.js for generating a temporary signature with which the upload asking can be signed. This temporary signature uses AWS authentication credentials (the access fundamental and secret key) as a ground for the signature, merely users will not have direct access to this information. After the signature has expired, and so upload requests with the same signature will not be successful.

To meet the completed Node file, please run into the appropriate code in the companion repository.

Showtime by creating your principal awarding file, app.js, in the root of your application directory and set up your skeleton application appropriately:

            const limited = require('limited'); const aws = require('aws-sdk');  const app = express(); app.set('views', './views'); app.use(express.static('./public')); app.engine('html', crave('ejs').renderFile); app.listen(process.env.PORT || 3000);  const S3_BUCKET = process.env.S3_BUCKET;                      

In some scenarios, information technology may be necessary to bank check that the environs'southward PORT var is a number by using Number(process.env.PORT).

The packages installed with npm are imported at the elevation of the application. The Express app is then set-upward and finally the bucket proper noun is loaded from the environment.

You should now configure your AWS region. To practise so, update the imported aws object. For example:

            aws.config.region = 'eu-west-one';                      

Think to use the region that your target bucket resides in. If you need information technology, apply this page to observe your region.

Next, in the aforementioned file, you will demand to create the views responsible for returning the correct information back to the user's browser when requests are made to various URLs. Inside the app.js file, define the view for requests to /business relationship to return the page account.html, which contains the grade for the user to complete:

            app.get('/account', (req, res) => res.return('account.html'));                      

Now create the view, in the same JavaScript file, that is responsible for generating and returning the signature with which the customer-side JavaScript can upload the image. This is the start request made past the customer before attempting an upload to S3. This view responds with requests to /sign-s3:

            app.become('/sign-s3', (req, res) => {   const s3 = new aws.S3();   const fileName = req.query['file-name'];   const fileType = req.query['file-type'];   const s3Params = {     Bucket: S3_BUCKET,     Key: fileName,     Expires: threescore,     ContentType: fileType,     ACL: 'public-read'   };    s3.getSignedUrl('putObject', s3Params, (err, information) => {     if(err){       console.log(err);       return res.end();     }     const returnData = {       signedRequest: data,       url: `https://${S3_BUCKET}.s3.amazonaws.com/${fileName}`     };     res.write(JSON.stringify(returnData));     res.cease();   }); });                      

This code uses the aws-sdk module to create a signed URL that the browser can use to make a PUT request to S3. In addition, the prospective URL of the object to be uploaded is produced as a combination of the S3 bucket proper noun and the object name. This URL and the signed request are then returned to the browser in JSON format.

The Expires parameter describes the number of seconds for which the signed URL will exist valid for. In some circumstances, such as when uploading large files, a larger value may be more appropriate in order to extend the validity of the signed URL.

Initialising the s3 object automatically loads the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY variables that were set into the environment earlier.

Y'all may wish to assign another, customised name to the object instead of using the one that the file is already named with, which is useful for preventing accidental overwrites in the S3 bucket. This name could be related to the ID of the user'southward business relationship, for example. If non, you should provide some method for properly quoting the name in case there are spaces or other awkward characters present. In addition, this is the stage at which you lot could provide checks on the uploaded file in order to restrict admission to certain file types. For instance, a simple cheque could be implemented to allow only .png files to proceed beyond this point.

Finally, in app.js, create the view responsible for receiving the account information after the user has uploaded an avatar, filled in the class, and clicked submit:

            app.post('/salve-details', (req, res) => {   // TODO: Read POSTed form data and do something useful });                      

This function is currently but a stub that you'll demand to complete in social club to allow the app to read and store the submitted contour information and to correctly acquaintance it with the rest of the user's account details.

Running the app

Everything should now be in place to perform the direct uploads to S3. To test the upload, relieve any changes and use heroku local to start the application:

You will need a Procfile for this to be successful. See Getting Started with Node.js on Heroku for more information. Also remember to correctly set your environment variables on your own machine before running the awarding locally.

            $ heroku local fifteen:44:36 web.1  | started with pid 12417                      

Press Ctrl+C to return to the prompt. If your application is returning 500 errors (or other server-based issues), and then start your server in debug mode and view the output in the Terminal emulator to assist gear up your problem:

            $ DEBUG=express:* node app.js                      

Summary

This commodity covers uploading to Amazon S3 directly from the browser using Node.js to temporarily sign the upload request. Although the guide and companion code focuses on the Limited framework, the idea should easily acquit over to other Node applications.

moberlythoofte.blogspot.com

Source: https://devcenter.heroku.com/articles/s3-upload-node

Share this post