Object storage integration about codem-transcode HOT 6 CLOSED

Hi Brad,

At the moment we are considering a fairly big rewrite of the transcoder, and this is one of the things that has crossed our minds. What's holding me back at the moment is the many different object storages we would have to support, each with it's one backing library and API plus the added complexity of having to deal with network transfers.

As an alternative at this moment, you might want to consider mounting your object storage directly on your system. Many providers have some form of FUSE support or similar. This removes the need for the transcoder to concern itself with these transfers and the file system driver can focus on transfers and storage.

from codem-transcode.

@tieleman Thanks for the super-quick reply! And thanks for the work on codem, it's great. I was actually going to go the FUSE route and I have a proof-of-concept working, however I'm looking at deploying on a cloud provider that doesn't provide sufficient privilege control over the container to allow mounting the device.

I actually do have access to such an environment but given that transcoding is so elastic in its resource needs, I'm looking to deploy in an environment that charges by actual usage, rather than having to spin up nodes that mostly sit idle between jobs. (I'm looking at IBM Bluemix, specifically.)

Regarding the various types of file systems to support, that pkgcloud library looks rather promising as it has support for many of the common providers, which is why it caught my eye. I'll follow-up here if I end up going that route. Regardless, support for either object stores natively or perhaps just a pre/post job hook API would be awesome and get closer to a true microservices type approach.

from codem-transcode.

👍

I'll be sure to look at pkgcloud, thanks for the pointer. There is a callback url though that you can specify to the transcoder job that allows you to receive a notification when a job finishes, maybe that's of use to you. You'd need a small HTTP service that accepts those callbacks. You could use that for a post hook.

from codem-transcode.

FYI @tieleman I'm working on this here - not sure if you'd be looking for a PR since you're considering a major refactor but thought I'd mention it here in case anyone else was interested.

from codem-transcode.

@bradjones1: Thanks for the PR and the heads up. At the moment I'm going to leave it dangling, but I'll keep your code in mind when we initiate the rewrite. Will your fork remain public? One of the things that still worries me is the fact that FFmpeg can have so many different input/output configurations ("plain" files, segmented files, multiple output formats, etc.) that it will be hard to support such a thing completely.

from codem-transcode.

@tieleman Yes I'll keep it public and thanks for the response. I attacked the input/output question by tar/gzipping output directories on segment/thumbnail jobs, but you're right, it's a complicated question. Regardless, I imagine others might be interested which is why I put it on here. Cheers.

from codem-transcode.