Fantastic transcoder is a video transcoder which utilizes massively parallel compute to achieve ludicrous conversion speeds.

This is an orchestrated collection of Lambda tasks. We use DynamoDB, SQS, and S3 for data structure, job tracking, and object storage respectively.

The steps are as follows:

Runs every minute (triggered by cloudwatch cron)

• Polls SQS Queue ft_videoconvert_queue
• If a job is found, check to see if we've been here before with this ConversionID DynamoDB.FT_VideoConversions
• If ConversionID is not present, write to DynamoDB.FT_VideoConversions (ConversionID, created)
• If ConversionID is present, increment retry
• If retry count > some number, hard fail - Move to deadletter queue - Alert
• Update SQS Status Queue with status of "Waiting for encoder"

Triggered by DynamoDB.FT_VideoConversions

• Update SQS Status Queue with status of "Downloading"
• Grabs video from S3
• Update SQS Status Queue with status of "Ready to process"
• Breaks out Audio
• Segment Video
• Upload audio file and each video segment to s3.VideoConversionsNG/Segmented/ConversionID/
• Log number of segments to DynamoDB.VideoConversions
• Write relevant data to DynamoDB.FT_SegmentState (ConversionID, SegmentID, created, ConversionFormat.each)
• Update SQS Status Queue with status of "Processing"

Triggered by DynamoDB.FT_SegmentState

• If multiple formats are required, one row should be written per segment per ConversionFormat
• Downloads Video
• Runs ffprobe / greps for the correct max rectangle value to check dimensions of video
• Converts video to mp4 / dimensions depending on settings in convert.py
• Transcodes video to transport stream
• Uploads video to s3.VideoConversions/ConvertedSegments/ConversionID/Format{1,2,3}/
• Updates dynamoDB.FT_SegmentState with (ConversionID, SegmentID, created, complete, ConversionFormat.each)
• Checks if each Segment has been converted
• if all segments have been converted, trigger concat step by writing to DynamoDB.FT_VideoConversions

Triggered by DynamoDB.FT_ConversionState

• Update SQS Status Queue with status of "Saving"
• One lambda triggered per format
• Downloads all converted transcode streams from s3.VideoConversions/ConvertedSegments/ConversionID/Format{1,2,3}/
• Makes sure all files are unique
• Checks DynamoDB to ensure it has the right / right number of segments. If not, Retry download
• Concatenates all segments into a single file
• Downloads and munges audio
• Places final transcoded file into s3 bucket
• Deletes message from SQS queue
• writes to DynamoDB.FT_VideoConversions.Complete
• Update SQS Status Queue with status of "Finished"

There are three tables within DynamoDB. ConversionID is the shared key between the tables. It's a unique identifier for each individual video to be converted.

• FT_VideoConversions triggers Conversions and contains basic data about the video file, such as the unique identifier and the number of retries. It also includes identifiers for the SQS messages
• FT_SegmentState tracks the conversion status of each segment, and is responsible for triggering FT_Convert
• FT_ConversionState tracks state data about the conversion process and is responsible for triggering FT_Concat

• Any lambda, if ffmpeg exits with status other than 1, trigger failed in dynamoDB & update SQS queue message visibility to 1
• If retries > 5 when accepting from SQS queue, send to deadletter queue instead. Configure cloudwatch alert on deadletter queue

fileName : video.mp4
bucket : bucket_name
uploadID : THIS15ANUPL0ADID
path : path/to/your/thing/
sizeFormat : {'1080p': ['MP4'], 'original': ['MP4'], '480p': ['MP4']}
s3_url : https://bucket_name.s3.amazonaws.com/path/to/your/thing/video.mp4

• get upload endpoint up and dumping to correct bucket
• Concat step functional testing & triggering off of dynamoDB.
• encoding parameters logic - allow people to specify destination codec and desired formats in SQS message.
• make bundler fully operational - should properly install our ffmpeg binary
• Add installation/implementation guide
• 100% code coverage
• Jenkinsfile (or travis.yml) for automated deployment to multiple environments

• set up expiration time on s3 buckets
• developer documentation that includes how to alter the ffmpeg commands
• update bundler script, build full tutorial on how to integrate FC
• Add frontend for example purposes?
• add support for converting audio files?
• add support for status queue:
• hopefully deprecate polling function in favor of triggering from SQS - cmon amazon!
• hopefully add autoscaling for dynamoDB instead of specifying static values in main.tf - cmon hashicorp!
• Make it easier for contributors to add support for alternate IIASPs (google and azure functions)
• Add support for subtitles - break them out with the audio during segmentation
• FAQ

Current Maintainers:

• Gabe Abinante (ClearSlide) - https://github.com/gabinante

• Jonathan Tan (ClearSlide) - https://github.com/jonathanjtan
• Barak Zhou (ClearSlide) - https://github.com/zhoubw

Copyright 2017 ClearSlide, inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License in this repository, or at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.