Easy Video Encoding with AWS

The following CloudFormation template creates resources that automate the encoding of video. It creates a stack that provides the following workflow:

  1. User uploads video file to the /originals directory of a S3 bucket.
  2. A lambda function is notified of the newly uploaded file and starts an AWS MediaConvert job to encode the video in both Dash and HLS.
  3. The resulting encoded files are stored in the /assets directory of the same S3 bucket.

More detailed explanation following the template.

I was calling this setup "transcodekit" as I worked on it. Feel free to ignore that label wherever you see it. :)

AWSTemplateFormatVersion: 2010-09-09

Parameters:
  BucketName:
    Type: String
    Default: 'transcodekit'

Resources:
  TranscodekitLambdaPolicy:
    Type: 'AWS::IAM::ManagedPolicy'
    Properties:
      Description: Provides necessary access to MediaConvert and CloudWatch logs
      ManagedPolicyName: !Join
        - '-'
        - - !Ref 'AWS::Region'
          - TranscodekitLambdaExecutor
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - mediaconvert:CreateJob
              - mediaconvert:DescribeEndpoints
            Resource:
              - '*'
          - Effect: Allow
            Action:
              - 'logs:CreateLogGroup'
              - 'logs:CreateLogStream'
              - 'logs:PutLogEvents'
            Resource:
              - '*'
          - Effect: Allow
            Action:
              - 'iam:PassRole'
            Resource:
              - !GetAtt MediaConvertRole.Arn

  LambdaExecutionRole:
    Type: 'AWS::IAM::Role'
    Properties:
      RoleName: LambdaExecution
      Description: Allows Transcodekit lambda function to start MediaConvert job
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
      ManagedPolicyArns:
        - !Ref TranscodekitLambdaPolicy

  TranscodeVideoFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: TranscodeVideo
      Description: Sends uploaded S3 object to MediaConvert for transcoding
      Handler: index.handler
      Role: !GetAtt LambdaExecutionRole.Arn
      Environment:
        Variables:
          ROLE: !GetAtt MediaConvertRole.Arn
      Code:
        ZipFile: |
          const path = require('path')
          const MediaConvert = require('aws-sdk/clients/mediaconvert')

          exports.handler = async function(event, context, cb) {
            const mediaConvert = new MediaConvert({
              apiVersion: '2017-08-29'
            })
            const s3Record = event.Records[0].s3
            const { base: fileName, name: title } = path.parse(s3Record.object.key)

            try {
              const { Endpoints: [{ Url: endpoint }]} = await mediaConvert.describeEndpoints().promise()
              mediaConvert.endpoint = endpoint

              const hlsResponse = await mediaConvert.createJob({
                Role: process.env.ROLE,
                JobTemplate: 'System-Ott_Hls_Ts_Avc_Aac',
                Settings: {
                  Inputs: [{
                    FileInput: `s3://${s3Record.bucket.name}/${s3Record.object.key}`,
                    AudioSelectors: {
                      'Audio Selector 1': {
                        Offset: 0
                      }
                    },
                  }],
                  OutputGroups: [{
                    OutputGroupSettings: {
                      Type: 'HLS_GROUP_SETTINGS',
                      HlsGroupSettings: {
                        Destination: `s3://${s3Record.bucket.name}/assets/${title}/hls/`
                      }
                    }
                  }]
                },
              }).promise()

              const dashResponse = await mediaConvert.createJob({
                Role: process.env.ROLE,
                JobTemplate: 'System-Ott_Dash_Mp4_Avc_Aac',
                Settings: {
                  Inputs: [{
                    FileInput: `s3://${s3Record.bucket.name}/${s3Record.object.key}`,
                    AudioSelectors: {
                      'Audio Selector 1': {
                        Offset: 0
                      }
                    },
                  }],
                  OutputGroups: [{
                    OutputGroupSettings: {
                      Type: 'DASH_ISO_GROUP_SETTINGS',
                      DashIsoGroupSettings: {
                        Destination: `s3://${s3Record.bucket.name}/assets/${title}/dash/`
                      }
                    }
                  }]
                },
              }).promise()

              cb(null, [hlsResponse, dashResponse])
            } catch (e) {
              cb(e.message)
            }
          }
      Runtime: nodejs12.x

  S3Bucket:
    Type: 'AWS::S3::Bucket'
    DependsOn:
      - S3ExecutionPermission
    Properties:
      BucketName: !Ref BucketName
      NotificationConfiguration:
        LambdaConfigurations:
          - Event: 's3:ObjectCreated:*'
            Function: !GetAtt TranscodeVideoFunction.Arn
            Filter:
              S3Key:
                Rules:
                  - Name: prefix
                    Value: 'originals/'

  S3ExecutionPermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !GetAtt TranscodeVideoFunction.Arn
      Action: lambda:InvokeFunction
      Principal: s3.amazonaws.com
      SourceAccount: !Ref 'AWS::AccountId'
      SourceArn: !Sub 'arn:aws:s3:::${BucketName}'

  TranscodekitMediaConvertPolicy:
    Type: 'AWS::IAM::ManagedPolicy'
    Properties:
      ManagedPolicyName: !Join
        - '-'
        - - !Ref 'AWS::Region'
          - TranscodekitMediaConverter
      Description: Provides access to S3 for MediaConvert transcode jobs
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - 's3:PutObject'
            Resource:
              - !Sub 'arn:aws:s3:::${BucketName}/assets/*'
          - Effect: Allow
            Action:
              - 's3:GetObject'
            Resource:
              - !Sub 'arn:aws:s3:::${BucketName}/originals/*'

  MediaConvertRole:
    Type: 'AWS::IAM::Role'
    Properties:
      RoleName: MediaConvertExecution
      Description: Allows MediaConvert to gain access to S3
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - mediaconvert.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
      ManagedPolicyArns:
        - !Ref TranscodekitMediaConvertPolicy

Getting the video from S3 to MediaConvert

At the time of writing, S3 does not have a notification that will send an uploaded file directly to MediaConvert. So, we'll have to use a Lambda function. S3 will notify our Lambda function of the file and Lambda will kick off the MediaConvert encoding jobs.

This general setup is relatively simple. The tricky part was the required IAM roles and managed policies required to give the different resources permission to interact. The Lambda function has its own role, but must pass a different role to the MediaConvert jobs it initiates. This passed role gives the MediaConvert jobs permission to do what they need to do: read from and write to S3.

Encoding the Video

We use the System-Ott_Hls_Ts_Avc_Aac and System-Ott_Dash_Mp4_Avc_Aac job templates to simplify configuring the encoding settings. These templates produce HLS and Dash encoded videos respectively in a range of resolutions and bitrates.

The jobs themselves are instructed where to read the original files in S3 and where to write the encoded assets when they are done.

Cost

For a 5 minute video, encoding in both HLS and Dash in the variety of resolutions and bitrates provided by the aforementioned MediaConvert templates costs $2-$3.

Happy encoding!