jdupl / lancoder Goto Github PK
View Code? Open in Web Editor NEWA cross os cluster for distributed video encoding
License: GNU General Public License v3.0
A cross os cluster for distributed video encoding
License: GNU General Public License v3.0
ClusterProtocol and Status should both use Enums for type safe validation
User should be able to override default 5min task length
Detect and convert all audio tracks to the output file.
Theoretical support for multiple video tracks.
Master gets the ip of the client from what the client sent.
Master should get ip address from the remoteaddress of the socket
x265 will be experimental for now
ffmpeg concat protocol and x265 not compatible ?
If a directory is given as input, create jobs for every file in the directory.
Hi, I noticed that when launched with -w -vvvvv
or -w -vvvvvv
, lancoder crashed, it only happens with the worker.
Here is the log :
e:\#Download>java -jar lancoder-0.4.0-jar-with-dependencies.jar -w -vvvvv
Sat May 21 14:55:25 CEST 2016 INFO: Runnning lancoder 0.4.0
Sat May 21 14:55:25 CEST 2016 FINE: Loaded config from disk.
Sat May 21 14:55:25 CEST 2016 FINE: Detected 8 threads available.
Sat May 21 14:55:25 CEST 2016 FINER: [C:\Program Files (x86)\ffmpeg\bin\ffmpeg.exe, -encoders]
Exception in thread "main" java.lang.NullPointerException
at org.lancoder.worker.Worker.getStatus(Worker.java:258)
at org.lancoder.worker.logging.LogSenderPool.isOnline(LogSenderPool.java:27)
at org.lancoder.common.logging.ClusterLogStagingHandler.flush(ClusterLogStagingHandler.java:29)
at org.lancoder.common.logging.ClusterLogStagingHandler.publish(ClusterLogStagingHandler.java:24)
at java.util.logging.Logger.log(Unknown Source)
at java.util.logging.Logger.doLog(Unknown Source)
at java.util.logging.Logger.log(Unknown Source)
at java.util.logging.Logger.finer(Unknown Source)
at org.lancoder.ffmpeg.FFmpegReader.read(FFmpegReader.java:65)
at org.lancoder.ffmpeg.FFmpegReader.read(FFmpegReader.java:37)
at org.lancoder.ffmpeg.probers.CodecProber.getNodeCapabilities(CodecProber.java:29)
at org.lancoder.ffmpeg.FFmpegWrapper.getAvailableCodecs(FFmpegWrapper.java:19)
at org.lancoder.worker.Worker.bootstrap(Worker.java:83)
at org.lancoder.Main.run(Main.java:122)
at org.lancoder.Main.main(Main.java:38)
Exception in thread "Thread-1" java.lang.NullPointerException
at org.lancoder.common.network.cluster.Server.stop(Server.java:28)
at org.lancoder.common.Container.stopServices(Container.java:98)
at org.lancoder.common.Container.stop(Container.java:107)
at org.lancoder.worker.Worker.shutdown(Worker.java:126)
at org.lancoder.Main$1.run(Main.java:115)
Web api should be able to shutdown nodes (including master itself).
Detect current OS and change null output file from /dev/null to NUL if windows.
Used in:
Prevent refusal-proposition loop if an unknown (for now) exception occurs.
The double assignment protection (concurrency) is not affected by this.
Solution is to never assign n times the same job (not task) to a node.
Extract first audio stream from input file
Encode to predefined audio codec and bitrate
Mux into final mkv file
Protect the node key with secure tunnel.
For later.
Allow user to provide his own command line encoder options.
Check-box for --slow-first-pass.
Add filter of status/task in node and job list
Pools should be able to say to listener that their list of todo items is getting too big. The listener can then slow the requests made to the pool or deny incoming requests.
Rename the output folder of the job to the input's name (handle extensions)
Pause jobs and nodes.
Restful like.
Use of java http servlet with jetty server.
At muxing step, extract selected audio from input file and encode into specific format and mux to final container.
Should support:
Asynchronous node checking so node checker can look up multiple nodes without having to wait for timeouts.
Timeouts could then be increased from 2 seconds to 5 seconds.
Replace the shared folder name 'job_name - filename' to 'job_name/optional_folder/original_filename'
It's almost time to allow starting with --master
to run as master server and with --worker
as a node.
This will finally allow running on multiple machines without any hassle.
Master should handle missing encoder and other fatal exceptions.
Detect which operations can be achieved on a worker before sending tasks.
This will ensure each node can actually encode the desired codec.
Encoding libraries can be parsed from FFmpeg with ffmpeg -encoders
.
Mux all tasks of a job with mkvtoolnix to a single file.
Copy audio from source file.
Some encoders might not be multi threaded.
Nodes should be able to have multiple jobs (one per thread).
Hi, I tried to build lancoder from Windows (with modification to the hardcoded paths in the tests) and Linux and it fails most of the time (95%) from tests in PoolTest :
Running org.lancoder.common.pool.PoolTest Exception in thread "Thread-5" java.lang.NullPointerException at org.lancoder.common.pool.PoolWorker.run(PoolWorker.java:171) at java.lang.Thread.run(Thread.java:745) Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.056 sec <<< FAILURE! - in org.lancoder.common.pool.PoolTest testDispatchingSingleWorker(org.lancoder.common.pool.PoolTest) Time elapsed: 0.403 sec <<< FAILURE! java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertFalse(Assert.java:64) at org.junit.Assert.assertFalse(Assert.java:74) at org.lancoder.common.pool.PoolTest.testDispatchingSingleWorker(PoolTest.java:98)
And even when I am lucky enough for it to pass the tests, I always get the NullPointerException.
Encoding 2 pass CRF is not possible.
Hi,
i tried to encode on 3 servers and one died with aSegfault on ffmpeg.
So Part 9 is now missing and lancoder is not restarting it.
Should this be implemented or how should can i restart the missing part?
thanks
Provide progress tracking for audio tasks.
Provide an estimate for each pass of the tasks (only for multi passes video tasks) instead of a global task estimate.
refactor to TaskReport and TaskReportMessage
Subtask: refactor Message to interface to allow TaskReportMessage to extend TaskReport and implement Message
Refactor method dealing with events to implement listeners in Master and Worker main thread
lancoder is really cool
Cluster was a Linux machine with FFmpeg 2.4.2 and a Windows machine.
Some encodings exited with code 0 and only first part of the file is playable but show full length.
Some encodings simply failed and if done manually, same problem occurs.
MkvMerge is able to concatenate these parts...
Create a user interactive configuration maker if a valid config file not found.
Handle mutli pass encoding with libx264
Embed web resources to simplify deployment of the program.
Jetty should serve static files from the jar itself.
Stop work on all nodes handling the canceled job.
Service should only tag cleanables as to be cleaned then call the clean to avoid comodification errors
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.