jdupl / lancoder Goto Github PK

ClusterProtocol and Status should both use Enums for type safe validation

User should be able to override default 5min task length

Detect and convert all audio tracks to the output file.
Theoretical support for multiple video tracks.

Master gets the ip of the client from what the client sent.
Master should get ip address from the remoteaddress of the socket

x265 will be experimental for now

ffmpeg concat protocol and x265 not compatible ?

If a directory is given as input, create jobs for every file in the directory.

Hi, I noticed that when launched with -w -vvvvv or -w -vvvvvv, lancoder crashed, it only happens with the worker.

Here is the log :

e:\#Download>java -jar lancoder-0.4.0-jar-with-dependencies.jar -w -vvvvv
Sat May 21 14:55:25 CEST 2016 INFO: Runnning lancoder 0.4.0
Sat May 21 14:55:25 CEST 2016 FINE: Loaded config from disk.
Sat May 21 14:55:25 CEST 2016 FINE: Detected 8 threads available.
Sat May 21 14:55:25 CEST 2016 FINER: [C:\Program Files (x86)\ffmpeg\bin\ffmpeg.exe, -encoders]
Exception in thread "main" java.lang.NullPointerException
        at org.lancoder.worker.Worker.getStatus(Worker.java:258)
        at org.lancoder.worker.logging.LogSenderPool.isOnline(LogSenderPool.java:27)
        at org.lancoder.common.logging.ClusterLogStagingHandler.flush(ClusterLogStagingHandler.java:29)
        at org.lancoder.common.logging.ClusterLogStagingHandler.publish(ClusterLogStagingHandler.java:24)
        at java.util.logging.Logger.log(Unknown Source)
        at java.util.logging.Logger.doLog(Unknown Source)
        at java.util.logging.Logger.log(Unknown Source)
        at java.util.logging.Logger.finer(Unknown Source)
        at org.lancoder.ffmpeg.FFmpegReader.read(FFmpegReader.java:65)
        at org.lancoder.ffmpeg.FFmpegReader.read(FFmpegReader.java:37)
        at org.lancoder.ffmpeg.probers.CodecProber.getNodeCapabilities(CodecProber.java:29)
        at org.lancoder.ffmpeg.FFmpegWrapper.getAvailableCodecs(FFmpegWrapper.java:19)
        at org.lancoder.worker.Worker.bootstrap(Worker.java:83)
        at org.lancoder.Main.run(Main.java:122)
        at org.lancoder.Main.main(Main.java:38)
Exception in thread "Thread-1" java.lang.NullPointerException
        at org.lancoder.common.network.cluster.Server.stop(Server.java:28)
        at org.lancoder.common.Container.stopServices(Container.java:98)
        at org.lancoder.common.Container.stop(Container.java:107)
        at org.lancoder.worker.Worker.shutdown(Worker.java:126)
        at org.lancoder.Main$1.run(Main.java:115)

Web api should be able to shutdown nodes (including master itself).

Detect current OS and change null output file from /dev/null to NUL if windows.

Used in:

• WorkThread - first pass x264 output

Prevent refusal-proposition loop if an unknown (for now) exception occurs.

• No ffmpeg
• no more local disk space
• Network disconnected
• libav<->ffmpeg problem

The double assignment protection (concurrency) is not affected by this.

Solution is to never assign n times the same job (not task) to a node.

• Perhaps retry from time to time (network outage)
• Try to detect if problem is temporary
• Crash report object ?

Extract first audio stream from input file
Encode to predefined audio codec and bitrate
Mux into final mkv file

Protect the node key with secure tunnel.
For later.

Allow user to provide his own command line encoder options.
Check-box for --slow-first-pass.

Add filter of status/task in node and job list

Pools should be able to say to listener that their list of todo items is getting too big. The listener can then slow the requests made to the pool or deny incoming requests.

Rename the output folder of the job to the input's name (handle extensions)

Pause jobs and nodes.

Restful like.

Use of java http servlet with jetty server.

At muxing step, extract selected audio from input file and encode into specific format and mux to final container.

Should support:

• Codec (vorbis, ac3, aac)
• Channel (original, stereo, mono)
• Bitrate

Asynchronous node checking so node checker can look up multiple nodes without having to wait for timeouts.
Timeouts could then be increased from 2 seconds to 5 seconds.

Replace the shared folder name 'job_name - filename' to 'job_name/optional_folder/original_filename'

It's almost time to allow starting with --master to run as master server and with --worker as a node.

This will finally allow running on multiple machines without any hassle.

Master should handle missing encoder and other fatal exceptions.

Detect which operations can be achieved on a worker before sending tasks.

This will ensure each node can actually encode the desired codec.

Encoding libraries can be parsed from FFmpeg with ffmpeg -encoders.

Mux all tasks of a job with mkvtoolnix to a single file.

Copy audio from source file.

Some encoders might not be multi threaded.
Nodes should be able to have multiple jobs (one per thread).

Hi, I tried to build lancoder from Windows (with modification to the hardcoded paths in the tests) and Linux and it fails most of the time (95%) from tests in PoolTest :
Running org.lancoder.common.pool.PoolTest Exception in thread "Thread-5" java.lang.NullPointerException at org.lancoder.common.pool.PoolWorker.run(PoolWorker.java:171) at java.lang.Thread.run(Thread.java:745) Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.056 sec <<< FAILURE! - in org.lancoder.common.pool.PoolTest testDispatchingSingleWorker(org.lancoder.common.pool.PoolTest) Time elapsed: 0.403 sec <<< FAILURE! java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertFalse(Assert.java:64) at org.junit.Assert.assertFalse(Assert.java:74) at org.lancoder.common.pool.PoolTest.testDispatchingSingleWorker(PoolTest.java:98)

And even when I am lucky enough for it to pass the tests, I always get the NullPointerException.

Encoding 2 pass CRF is not possible.

Hi,

i tried to encode on 3 servers and one died with aSegfault on ffmpeg.
So Part 9 is now missing and lancoder is not restarting it.

Should this be implemented or how should can i restart the missing part?

thanks

Provide progress tracking for audio tasks.

Provide an estimate for each pass of the tasks (only for multi passes video tasks) instead of a global task estimate.

refactor to TaskReport and TaskReportMessage

Subtask: refactor Message to interface to allow TaskReportMessage to extend TaskReport and implement Message

Refactor method dealing with events to implement listeners in Master and Worker main thread

lancoder is really cool

• [ x] FFmpeg seems to fail concatening 2 pass encoded x265.
• Confirm if 1 pass x265 encodings are affected

Cluster was a Linux machine with FFmpeg 2.4.2 and a Windows machine.

Some encodings exited with code 0 and only first part of the file is playable but show full length.

Some encodings simply failed and if done manually, same problem occurs.

MkvMerge is able to concatenate these parts...

Create a user interactive configuration maker if a valid config file not found.

Handle mutli pass encoding with libx264

Embed web resources to simplify deployment of the program.
Jetty should serve static files from the jar itself.

Stop work on all nodes handling the canceled job.

Service should only tag cleanables as to be cleaned then call the clean to avoid comodification errors