Git Product home page Git Product logo

djmarinara's Introduction

djmarinara

djmarinara is an ffmpeg-based live streaming music queue manager!

  • It pulls a user-defined URL which is expected to be full of more URLs, representing the playlist choices.
  • While djmarinara is running, random URLs are chosen from the playlist URL and automatically converted into FLV videos suitable for YouTube and other streaming services.
  • djmarinara also produces an endless series of playlist files. These are meant to be used by ffmpeg's ffconcat demuxer to act as an infinite playlist.

Note that you can see it live and in action here: https://youtube.com/gorzek/live

Believe it or not, that stream runs entirely off of a cluster of Raspberry Pi 4s!

But you probably don't care about the details. What you really want to know is...

How do I use this?

First, pick which run method you want to use.

Simple, Command-Line

  1. Clone this repository.
  2. Ensure you have Python3 installed.
  3. Ensure you have ffmpeg and ffprobe in your PATH.
  4. Update src/djmarinara/main.py with appropriate parameters. (Parameters will be described further down.)
  5. Run main.py!
  6. In another command line session, run ffmpeg like so: ffmpeg -stats -re -f concat -i /media/playlist0.txt -progress - -nostats -c:v copy -c:a copy -f flv [YOUR_STREAMING_URL_HERE]"
  7. Monitor your streaming URL (YouTube works best, for now) and your startup video will play until some additional files have been pre-rendered.
  8. Enjoy the show!

Kubernetes Cluster (Raspberry Pi)

What?! You can run this in Kubernetes?!

Yes, djmarinara was initially built for a cluster of Raspberry Pi 4s.

Check out src/ffmpeg-streamer/ffmpeg-streamer-deployment.yaml for an example of how to run on Kubernetes.

Please note that you'll need to build your own Docker image for djmarinara with updated parameters. (Parameters will be described further down.)

Podman (Raspberry Pi)

If you don't want to run a full Kubernetes cluster, consider podman play kube. You can run the manifest under src/ffmpeg-streamer/ffmpeg-streamer-deployment.yaml locally with podman!

Please note that the same caveats apply in terms of building your own Docker image for djmarinara with custom parameters. (Parameters will be described further down.)

Configuration

Regardless of run method, you'll need to edit the djmarinara script to fit your needs.

For src/djmarinara/main.py, you can edit the following in the noted configuration section:

  • extensions - A list of file extensions that can be downloaded and played. Only change these to add formats that ffmpeg supports! Note that archive types other than zip will be supported in the future.
  • temppath - Files are stored here temporarily during processing. You shouldn't have to change this.
  • mediapath - This is where your rendered videos and playlists are stored. If you change this, make sure when you run ffmpeg you point to the proper location for playlist0.txt.
  • playlisturl - This needs to be a list of URLs of songs to play, one per line. zip files are fine, too, so long as they contain one or more songs. (A song is chosen at random from zip files containing more than one.)
  • fonturl - URL to a font to use for rendering text on the videos. Fixed-width is preferred if playing music modules, as many contain ASCII art in their comments. Any of the fonts from this site that support code page 437 are highly recommended: https://int10h.org/oldschool-pc-fonts/
  • startupvideo - This is a URL to a video that will be played until there are some songs queued up. Pick something a minute or two long that is pleasant to listen to and look at. :)
  • gastanklimit - You probably won't have to change this. It defines, in seconds, how much djmarinara should have pre-rendered before taking a break. It's not perfect and will be replaced with a more adaptive method in the future.
  • targetspeed - Target pre-rendering speed. 2.0 is a safe default, but set it higher if you have CPU to spare!

Quirks and Limitations

This is a hobby project, so it's bound to have some peculiarities to it!

  • It might crash. In Kubernetes, this is fine--it'll just restart. If you run it outside a cluster, consider running both the main.py script and ffmpeg in endless loops (in separate terminal sessions).
  • When djmarinara crashes, it'll pick up where it left off when you start it again.
  • The queue of pre-rendered videos is managed in a few different ways all at once. First, the gastanklimit defines how much play time to pre-render. Second, any video rendered more than 90 minutes ago is deleted. Third, if the disk used by mediapath goes above 80% full, pre-rendered songs will be deleted, starting with the oldest, until disk usage falls below 80% again. Finally, any files in the working directory where djmarinara is run will be deleted regularly (except for files critical to djmarinara itself.) So, don't go storing important files there! (This will be adjusted in the future, but consider it fair warning for now!)
  • This app was built for Linux, and the Docker images are for 64-bit Raspberry Pi 4 systems, but you can probably run it on Windows or anything else that runs both Python3 and ffmpeg.
  • Currently, only 1080p 30fps FLV videos are produced for streaming, with some hard-coded settings. Visualization is from ffmpeg's showcqt filter. These are all meant to be adjustable in the future, but they're set in stone for now. (You can, of course, edit the djmarinara.py module! Contributions welcome.)

Roadmap

There are plenty of features and fixes planned. In no particular order:

  • Parameterization of most ffmpeg options, allowing for more use cases than 1080p 30fps streaming of music files put through the showcqt filter. :)
  • Proper health probes for the Kubernetes version.
  • Parameterizing the ffmpeg-bootstrap image to pull its assets from another repo instead of being hard-coded.
  • Parameterizing djmarinara's Docker image so it doesn't have to be rebuilt to use different settings.
  • More intelligent switching from standby video to the actual playlist.
  • Option to play a playlist in order instead of just randomly.
  • Fix some quirks with the scrolling speed of info text when there is a lot of text.
  • Efficiency improvements in conversion of music modules. Some use a vast amount of CPU and can have sub-real-time rendering speeds on low-power hardware.
  • Add visualizer presets and make them user-configurable, while letting djmarinara choose them randomly when rendering.
  • Add option to convert and play videos instead of just songs.

Copyright Warning

Streaming copyrighted material, including music, is a fraught endeavor. Be sure that anything you stream with this tool won't get you into legal hot water. Stay safe!

djmarinara's People

Contributors

gorzek avatar maseca avatar

Watchers

 avatar

Forkers

maseca

djmarinara's Issues

Avoid storing temporary data in djmarinara's working directory

Currently, djmarinara pulls files down into its working directory when preparing to render them. This has led to a complex cleanup system requiring a manifest file to ensure djmarinara cleans up after itself even if it has crashed previously.

Instead of this mess, ensure that files are only ever downloaded to a path under /tmp and that when djmarinara starts up, it cleans up anything sitting around from its previous run.

Suggest path: /tmp/djmarinara

Create TOML-based configuration system for djmarinara

djmarinara should be accompanied by an example TOML configuration file covering all of the current startup parameters, as represented in the djmarinara module's constructor.

Additional configuration should be added to this file going forward.

Propose config.toml as the filename.

Add support for additional archive formats

djmarinara can currently find songs to play if they are contained in ZIP files. If the file chosen from the playlist found at the playlist URL has a .zip extension, the archive is pulled down, extracted, and any candidate songs are identified and one is chosen at random to play.

This functionality should be enhanced to also support:

  • .tar.gz/.tgz archives (tarballs)
  • .7z (7-zip)
  • .lha
  • .ace
  • .rar
  • .bzip/.bz2

The availability of this functionality is contingent on Python support or ease of algorithmic implementation in Python. Let's not rewrite any of these from scratch if it would require a lot of work.

Allow djmarinara to play a list of URLs in order

Currently, the playlist URL djmarinara pulls down is expected to consist of URLs that it will play at random.

Add a TOML configuration option for djmarinara to allow playing the playlist URLs in order.

Speed up music module conversion

Music modules (mod, it, xm, s3m, etc.) don't convert to AAC very quickly. It's likely there is a good native PCM format they convert best to initially, and that should be used first to reduce CPU load/conversion time and to prioritize conversion to AAC for the video rendering step.

Logging fixes

A couple small issues with the new logging changes:

  • Console display only occurs for WARNING and higher. It should default to INFO, or at least have an option in main.py for that.
  • The location of the log file is not clear. This may be more of a documentation issue, or it might be useful to actually log the location explicitly when the script starts, and ensure that is displayed in the console, too.

Implement censoring filter

Since djmarinara is conducive to using massive playlists of songs that may not have been fully vetted ahead of time, it's desirable to put some basic constraints on what may be contained in song metadata.

  1. Choose a reasonably complete list of slurs and/or profane words.
  2. Allow configuration for whether the censor is enabled and whether such words should be censored, or whether a song whose metadata contains any of them should be discarded completely.
  3. Implement text replacement for censored words.

Track disk space used per second of video

I'm still roughing out what I want this for, but it comes down to effective cache management.

Every time a song is rendered, a running average should be calculated for how much disk space is being used per second of video. This should be kept around as a djmarinara object member.

I have a few use cases in mind for it, just need the number to exist first! Expected calculation is bytes per second. An integer is fine.

Allow djmarinara to recover from crashes

Whenever it starts up, djmarinara should go through any existing songs in the media path and add them to the gas tank, so gas tank calculations will make sense.

(djmarinara doesn't crash very often, but this is still useful to account for.)

Make text scrolling speed constant

The formula used to handle text scrolling in the rendered videos works pretty well, except that songs with a lot of text (hundreds of lines) tend to scroll very quickly. It would be ideal to rework this formula to scroll text at a constant rate over time, regardless of frame rate or text length.

Additional checks to ensure videos always render with text

There are occasions where videos get rendered that have no text displayed on them. I assume something is amiss with the text encoding or there is some other issue related to the source text. Better validation is needed here, and to either fix the malfunctioning text or discard songs whose text data can't be salvaged.

Use only /tmp for scratch space for rendering

Currently, djmarinara pulls files down into its working directory when preparing to render them. This has led to a complex cleanup system requiring a manifest file to ensure djmarinara cleans up after itself even if it has crashed previously.

Instead of this mess, ensure that files are only ever downloaded to a path under /tmp and that when djmarinara starts up, it cleans up anything sitting around from its previous run.

Suggest path: /tmp/djmarinara

Figure out how to tell which playlist file ffmpeg is currently playing

djmarinara and ffmpeg have limited means of communicating with one another. djmarinara queues up an infinite playlist of pre-rendered videos, while also trying to keep the cache of said videos to a reasonable size. However, if ffmpeg laps the existing playlist set, it crashes and will restart with whatever the oldest video still available in the cache is.

Example:

  • the queue has playlist0.txt, playlist6.txt, playlist7.txt, and playlist8.txt
  • playlist8.txt points to playlist9.txt, which djmarinara has not finished rendering
  • ffmpeg crashes when it tries to access playlist9.txt, so it restarts on playlist0.txt, which follows up with playlist6.txt, and onward

It would be better to know exactly what ffmpeg has already played so that the cache and queue management done by djmarinara can simply clean up already-played videos and their accompanying playlists, as well as to configure playlist0.txt to pick up exactly where ffmpeg left off before the last time it crashed.

This issue requires some research, hence the lengthy explanation.

Rework the ffmpeg-bootstrap system

The current ffmpeg-bootstrap setup is probably not needed anymore. It should be evaluated for one of the following outcomes:

  • Remove it altogether
  • Handle adding the desired files programmatically via djmarinara itself (already handled via the sanity check but make sure it works in a pod configuration)
  • Handle adding the desired files via configmap/secret instead, eliminating the need for an init container at all

Add more visualization presets

djmarinara currently has a single visualization profile, which uses the ffmpeg showcqt filter, horizontally flipped, with song metadata as scrolling text from bottom of the screen to top, along with a fade from/to black at the beginning/end of a song.

ffmpeg offers numerous visualization options that haven't been tried out with djmarinara yet. The objective for this issue would be to parameterize the visualization options for ffmpeg pre-rendering and offer 10 different presets.

A file format for the presets will have to be established and those presets will have to be included with djmarinara and built into its Dockerfile.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.