Easy Video Encoding with AWS
Use S3, Lambda, and MediaConvert to create an automatic video encoding pipeline.
The following CloudFormation template creates resources that automate the encoding of video. It creates a stack that provides the following workflow:
- User uploads video file to the /originals directory of a S3 bucket.
- A lambda function is notified of the newly uploaded file and starts an AWS MediaConvert job to encode the video in both Dash and HLS.
- The resulting encoded files are stored in the /assets directory of the same S3 bucket.
More detailed explanation following the template.
I was calling this setup "transcodekit" as I worked on it. Feel free to ignore that label wherever you see it. :)
AWSTemplateFormatVersion: 2010-09-09
Parameters:
BucketName:
Type: String
Default: 'transcodekit'
Resources:
TranscodekitLambdaPolicy:
Type: 'AWS::IAM::ManagedPolicy'
Properties:
Description: Provides necessary access to MediaConvert and CloudWatch logs
ManagedPolicyName: !Join
- '-'
- - !Ref 'AWS::Region'
- TranscodekitLambdaExecutor
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- mediaconvert:CreateJob
- mediaconvert:DescribeEndpoints
Resource:
- '*'
- Effect: Allow
Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource:
- '*'
- Effect: Allow
Action:
- 'iam:PassRole'
Resource:
- !GetAtt MediaConvertRole.Arn
LambdaExecutionRole:
Type: 'AWS::IAM::Role'
Properties:
RoleName: LambdaExecution
Description: Allows Transcodekit lambda function to start MediaConvert job
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
ManagedPolicyArns:
- !Ref TranscodekitLambdaPolicy
TranscodeVideoFunction:
Type: AWS::Lambda::Function
Properties:
FunctionName: TranscodeVideo
Description: Sends uploaded S3 object to MediaConvert for transcoding
Handler: index.handler
Role: !GetAtt LambdaExecutionRole.Arn
Environment:
Variables:
ROLE: !GetAtt MediaConvertRole.Arn
Code:
ZipFile: |
const path = require('path')
const MediaConvert = require('aws-sdk/clients/mediaconvert')
exports.handler = async function(event, context, cb) {
const mediaConvert = new MediaConvert({
apiVersion: '2017-08-29'
})
const s3Record = event.Records[0].s3
const { base: fileName, name: title } = path.parse(s3Record.object.key)
try {
const { Endpoints: [{ Url: endpoint }]} = await mediaConvert.describeEndpoints().promise()
mediaConvert.endpoint = endpoint
const hlsResponse = await mediaConvert.createJob({
Role: process.env.ROLE,
JobTemplate: 'System-Ott_Hls_Ts_Avc_Aac',
Settings: {
Inputs: [{
FileInput: `s3://${s3Record.bucket.name}/${s3Record.object.key}`,
AudioSelectors: {
'Audio Selector 1': {
Offset: 0
}
},
}],
OutputGroups: [{
OutputGroupSettings: {
Type: 'HLS_GROUP_SETTINGS',
HlsGroupSettings: {
Destination: `s3://${s3Record.bucket.name}/assets/${title}/hls/`
}
}
}]
},
}).promise()
const dashResponse = await mediaConvert.createJob({
Role: process.env.ROLE,
JobTemplate: 'System-Ott_Dash_Mp4_Avc_Aac',
Settings: {
Inputs: [{
FileInput: `s3://${s3Record.bucket.name}/${s3Record.object.key}`,
AudioSelectors: {
'Audio Selector 1': {
Offset: 0
}
},
}],
OutputGroups: [{
OutputGroupSettings: {
Type: 'DASH_ISO_GROUP_SETTINGS',
DashIsoGroupSettings: {
Destination: `s3://${s3Record.bucket.name}/assets/${title}/dash/`
}
}
}]
},
}).promise()
cb(null, [hlsResponse, dashResponse])
} catch (e) {
cb(e.message)
}
}
Runtime: nodejs12.x
S3Bucket:
Type: 'AWS::S3::Bucket'
DependsOn:
- S3ExecutionPermission
Properties:
BucketName: !Ref BucketName
NotificationConfiguration:
LambdaConfigurations:
- Event: 's3:ObjectCreated:*'
Function: !GetAtt TranscodeVideoFunction.Arn
Filter:
S3Key:
Rules:
- Name: prefix
Value: 'originals/'
S3ExecutionPermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !GetAtt TranscodeVideoFunction.Arn
Action: lambda:InvokeFunction
Principal: s3.amazonaws.com
SourceAccount: !Ref 'AWS::AccountId'
SourceArn: !Sub 'arn:aws:s3:::${BucketName}'
TranscodekitMediaConvertPolicy:
Type: 'AWS::IAM::ManagedPolicy'
Properties:
ManagedPolicyName: !Join
- '-'
- - !Ref 'AWS::Region'
- TranscodekitMediaConverter
Description: Provides access to S3 for MediaConvert transcode jobs
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 's3:PutObject'
Resource:
- !Sub 'arn:aws:s3:::${BucketName}/assets/*'
- Effect: Allow
Action:
- 's3:GetObject'
Resource:
- !Sub 'arn:aws:s3:::${BucketName}/originals/*'
MediaConvertRole:
Type: 'AWS::IAM::Role'
Properties:
RoleName: MediaConvertExecution
Description: Allows MediaConvert to gain access to S3
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- mediaconvert.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
ManagedPolicyArns:
- !Ref TranscodekitMediaConvertPolicy
Getting the video from S3 to MediaConvert
At the time of writing, S3 does not have a notification that will send an uploaded file directly to MediaConvert. So, we'll have to use a Lambda function. S3 will notify our Lambda function of the file and Lambda will kick off the MediaConvert encoding jobs.
This general setup is relatively simple. The tricky part was the required IAM roles and managed policies required to give the different resources permission to interact. The Lambda function has its own role, but must pass a different role to the MediaConvert jobs it initiates. This passed role gives the MediaConvert jobs permission to do what they need to do: read from and write to S3.
Encoding the Video
We use the System-Ott_Hls_Ts_Avc_Aac and System-Ott_Dash_Mp4_Avc_Aac job templates to simplify configuring the encoding settings. These templates produce HLS and Dash encoded videos respectively in a range of resolutions and bitrates.
The jobs themselves are instructed where to read the original files in S3 and where to write the encoded assets when they are done.
Cost
For a 5 minute video, encoding in both HLS and Dash in the variety of resolutions and bitrates provided by the aforementioned MediaConvert templates costs $2-$3.
Happy encoding!