Direct, SecureFile Uploads to AWS S3 from Web and Mobile

If you're developing an app which allows users to upload user-generated content such as photos, video or any other kind of document, you'll need to spend some time building out some infrastructure to handle file uploads.  

Typically these uploads are stored in a CDN or other storage provider such as Amazon S3. While the most straightforward solution is to upload files first to your backend server and then to the storage provider, this approach can cause performance issues and other problems.

In this post, we'll learn how to build secure, direct file uploads from web and mobile clients to AWS S3, skipping the step of sending the data to the server.  Though the code examples are presented in JavaScript both the server and client calls, this approach can work with any server and client technologies, including native mobile client apps.

Sever-Proxied Uploads vs Direct Client Uploads

The most straightforward approach to file uploads is to build out a backend API to process and deliver the file content to the storage provider.  This approach works okay, but it's a bit inefficient. Instead of delivering the file upload to its ultimate destination, you're transferring the file twice: first to your server, and then to your storage provider.  This means your upload is slow, and it consumes unnecessary server resources.

Furthermore, while the configuration of course varies by server software and hosting provider, you'll often find your app bumps up against file-size limits when you try to upload large files.  Just as an example, the popular nginx server, used to "front" many web services, has a default upload limit of 1MB.  This can be increased, but may be out of your control as an application developer, and even a larger setting may be too small for your application.

Direct uploads, in contrast, upload your content directly to the storage provider, by passing the need to upload the content to your own server.  This sounds simple, but is a bit tricky in practice!  To do a direct upload in a secure fashion, there are a couple of additional considerations.

Security Considerations of Direct Client Uploads

Direct client uploads offer a way to improve on the user experience and resource use problems associated with server-proxied uploads.  But implementing direct client uploads must be done with caution: if implemented incorrectly, it can lead to security issues like incorrect permissions or leaking of credentials.

The challenge is giving the client permission to upload the file in a secure fashion.  If we were to try to use the same credentials we use on the backend, it means they pass through the client and can be intercepted by an attacker.  Even if we create separate keys with limited access for the client, an attacker could still upload arbitrary content allowing them to fill up storage, perform DOS attacks, etc.  The only secure solution is to let a client upload only the content that we authorize.

Essential what we want is a one-time key that allows a direct upload of a file that we allow.  As it turns out, S3 (and other storage providers) allow us to generate something like this in the form of a pre-signed URL.  This is a URL referencing our S3 object, with a signature parameter appended.  This special URL allows us to write to the referenced object and only the referenced object.

Implementing Direct Client Uploads

Our approach to direct client uploads will leverage pre-signed URLs to let the client upload directly to our storage provider.  Of course, generating the pre-signed URL requires service provided credentials with sufficient credentials, so the signing cannot be done on the client.  Instead, we'll make a server request to validate the upload metadata (checking user authorization, file type, size, etc), generate a signed URL, and return it to the client.

The image below shows the difference in flow between a server-proxied upload and  a direct client upload.

Here are the basic steps to building a secure, direct client upload with a pre-signed URL:

• Implement a file input handler in client to receive a file from the user
• Send the file metadata to the server to validate the request
• Generate a pre-signed upload URL on the server and return it to the client
• Make a PUT request from the client to upload the file to S3

Server-Side Signed URL Generation

In the typical upload approach, our server holds AWS credentials which allow it to upload files.  In the direct upload approach, the client won't have direct access to these credentials–it would be highly insecure to put them in the client!

So in order to give the client permission to upload a file directly, we need to generate a pre-signed upload URL on the server, and send it back to the client.  The pre-signed URL is a URL to an S3 object which contains a signature that grants write permission to that resource (and of course, only that resource).

We'll implement a service on our backend to fetch the upload URL using the AWS SDK:

import AWS from "aws-sdk"

const s3 = new AWS.S3()

const getUploadURL = async (bucket: string, path: string, contentType: string) => {
  let putURL = await s3.getSignedUrlPromise('putObject', {
    Bucket: bucket,
    Key: path,
    Expires: 120,
    ContentType: contentType
  })

  return putURL
}

This function to generates our signed URL–next we'll write a quick service to expose the functionality.  For example, using the Express web server, we could write a service as follows:

app.post("/upload_url", async (req: Request, res: Response) => {
  let url = await uploadURL(userUploadBucket, req.body["path"], req.body["content_type"])
  res.send({ put_url: url })
})

Client-Side Upload Implementation

Once we've implemented the backend service, we'll use it to fetch the upload destination URL, then deliver our file content to the URL via a PUT request.

We'll typically implement this in the onChange handler of our file input.

const handler = (e: ChangeEvent<HTMLInputElement>) => {
  const files = e?.target?.files;

  if (files && files.length > 0) {
    let extension = files[0].name.split(".").pop()

    if (extension != null) {
        let response = await fetch(`${serverURL}/upload_url`, {
        	method: 'POST',
	        headers: { 'Content-Type': 'application/json' },
    	    body: JSON.stringify({ directory, extension, content_type: contentType })
       	})
            
        let json = await response.json()
    
        let uploadResponse = await fetch(json.url, {
          method: "PUT",
          body: content
        })
        
        // handle completion, update state to display uploaded file, etc
    }
  }
}

Tada!  With this code in place, we can perform uploads direct to S3 without sending them to a backend server first.

One Last Catch!  Displaying Uploaded Files

Though our upload implementation may be complete, the feature is probably not done yet!  In most web apps, after uploading the content, the uploaded file gets displayed back to the user.  The simplest way to do this is to use the newly uploaded URL.  The catch is that the upload URL we've generated doesn't actually allow us to view the object!  Instead, we need to generate another URL that will be used to view the object after it's uploaded.  Don't worry, this is simple!  We just have our original service return two URLs:

const uploadURL = async (bucket: string, path: string, contentType: string) => {
    var getURL = s3.getSignedUrl('getObject', {
        Bucket: bucket,
        Expires: 60 * 60 * 24 * 7,
        Key: path
    });

    let putURL = await s3.getSignedUrlPromise('putObject', {
        Bucket: bucket,
        Key: path,
        Expires: 120,
        ContentType: contentType
    })

    return { getURL, putURL }
}

Recap – Faster Uploads for Web & Mobile Apps 🎉

Direct client uploads to S3 are a great technique for building more responsive frontend applications.  By uploading your content directly to S3, you avoid the performance and resource consumption issues associated with sending large files through your backend server.