Git Product home page Git Product logo

bubuntoid / ethereal Goto Github PK

View Code? Open in Web Editor NEW
8.0 2.0 4.0 11.79 MB

🎵 Split youtube video chapters into mp3 files using just url and (optionally) timecodes

Home Page: http://81.177.135.200

License: GNU General Public License v3.0

Shell 0.08% C# 80.69% JavaScript 1.03% TypeScript 10.30% CSS 1.51% HTML 6.39%
youtube mp3 mp3-files ffmpeg csharp netcore5 aspnetcorewebapi yt-dlp youtube-chapters

ethereal's Introduction

Ethereal

Ethereal is a service that splitting youtube video chapters into mp3 files using just url and (optionally) timecodes. For most of processing operations it uses FFMPEG and yt-dlp.

Working preview web application: http://81.177.135.200

Syntax

There is no strict rule, if youtube recognize chapters from your video then we can parse it without any other additional actions. How do youtube chapters work: https://www.tubics.com/blog/youtube-chapters/

Correct (https://youtu.be/UZ7oOhhPEWU):

1. (0:00) KOPI / KOBASOLO – Swayed in Spring Reiniscence
2. (04:11) Sakanaction – Shin Takarajima
3. (09:13) Memai Siren - Nisemono no Utage

Correct as well (https://youtu.be/CU_ruPKWJpc):

0:00 Lazerhawk - So Far Away 
https://soundcloud.com/lazerhawk/so-f...
https://lazerhawk.bandcamp.com/

4:38  Zane Alexander - Elementary
https://soundcloud.com/thezanealexand...
https://zanealexander.bandcamp.com/al...

8:55 oDDling - Reverie
https://soundcloud.com/oddlingmusic/r...
https://oddling.bandcamp.com/releases

If your video has no timecodes in description, but you got one (often situation when people leave time codes in comments) or writted it by yourself, then you can explicit custom description with timecodes with youtube like syntax. Ensure that you have new line after each time code.

Stack / Dependencies

  • PostgreSQL 13
  • FFMPEG 4.2.4-1ubuntu0.1
  • .NET 6.0 (ASP.NET Core Runtime 6.0.16)
  • yt-dlp 2023.03.04

Backend

working app: http://81.177.135.200
swagger: http://81.177.135.200/swagger/index.html
hangfire dashboard: http://81.177.135.200/hangfire
healthcheck: http://81.177.135.200/api/hc

$ chmod +x scripts/run.sh
$ ./scripts/run.sh

Path executables

Download and install fmpeg (https://ffmpeg.org/)
Download and install yt-dlp (https://www.videohelp.com/software/yt-dlp)

Specify path to FFMPEG/yt-dlp executables in appsetings.json:

Linux:

 "System": {
    "TempPath" : "{current}\\bin\\temp",
    "FfmpegExecutablesPath" : "ffmpeg",
    "YtdlpExecutablesPath" : "yt-dlp",
    "VideoDurationLimit" : "05:00:00",
    "DownloadingTimeout": "00:01:00",
    "YouTubeProvider": "yt-dlp"
  },

Windows:

"System": {
    "TempPath" : "{current}\\bin\\temp",
    "FfmpegExecutablesPath" : "C:\\path\\to\\ffmpeg.exe",
    "YtdlpExecutablesPath" :  "C:\\path\\to\\yt-dlp.exe",
    "VideoDurationLimit" : "05:00:00",
    "DownloadingTimeout": "00:01:00",
    "YouTubeProvider": "yt-dlp"
  },

Development

Updating frontend version

  1. do frontend changes in angular project
  2. cd frontend
  3. export NODE_OPTIONS=--openssl-legacy-provider
  4. ng build --watch
  5. rm ../src/Ethereal.WebAPI/Frontend/*
  6. cp -r "./dist/ethereal/." "../src/Ethereal.WebAPI/Frontend"

Linux issues

If video downloading stage is skipping and job`s outcome results is not valid mp4/mp3 files - update yt-dlp by following this command:

yt-dlp -U

ethereal's People

Contributors

bubuntoid avatar emrd95 avatar webgtx avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

ethereal's Issues

Use autofac

remove

 services.AddScoped<ExceptionFilter>();
            
            services.AddScoped<FfmpegSettings>();
            services.AddScoped<IVideoSplitterService>(s =>
            {
                var settings = s.GetService<FfmpegSettings>();

                return new VideoSplitterService(
                    tempPath: settings.TempPath,
                    executablesPath: string.IsNullOrEmpty(settings.ExecutablesPath) ? null : settings.ExecutablesPath);
            });

initialize/initializeWithoutChapters bug

An exception has been raised that is likely due to a transient failure. at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func4 operation, Func4 verifySucceeded, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken) at Ethereal.Application.Commands.InitializeProcessingJobCommand.ExecuteAsync(String url, String description) in /root/ethereal/ethereal/src/Ethereal.Application/Commands/InitializeProcessingJobCommand.cs:line 77 at Ethereal.WebAPI.Controllers.JobsController.InitializeWithoutChapters(InitializeJobRequestDto dto) in /root/ethereal/ethereal/src/Ethereal.WebAPI/Controllers/JobsController.cs:line 50 at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

When logging to entity use new instance of dbcontext, not from di

Unhandled exception. System.ObjectDisposedException: Cannot access a disposed context instance. A common cause of this error is disposing a context instance that was resolved from dependency injection and then later trying to use the same context instance elsewhere in your application. This may occur if you are calling 'Dispose' on the context instance, or wrapping it in a using statement. If you are using dependency injection, you should let the dependency injection container take care of disposing context instances.
Object name: 'EtherealDbContext'.
at Microsoft.EntityFrameworkCore.DbContext.CheckDisposed()
at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
at Ethereal.Application.Commands.FetchYoutubeVideoCommand.<>c__DisplayClass5_0.<b__0>d.MoveNext() in /root/ethereal/ethereal/src/Ethereal.Application/Commands/FetchYoutubeVideoCommand.cs:line 56
--- End of stack trace from previous location ---
at System.Threading.Tasks.Task.<>c.b__140_1(Object state)
at System.Threading.QueueUserWorkItemCallback.<>c.<.cctor>b__6_0(QueueUserWorkItemCallback quwi)
at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
at System.Threading.QueueUserWorkItemCallback.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()

Split this github repository into banch of anothers

Минусы того, что у нас будет все в одном репозитории:

  • Если делать git clone -> будет захватывать миллион ненужных сурсов (возможно некоторым нужен только фронт/десктоп/сервер)
  • CI/CD
  • При билде будут скачиваться много вероятно ненужных зависимостей

Для разработки не на web application, где будет использоваться исключительно логика YOVPS.Core проекта - ide вероятно будет билдить все существующие проекты, это как минимум:

  • YOVPS.WebAPI,
  • YOVPS.Desktop,
  • YOVPS.CLI
  • Tests

Плюс ко всему возможно добавиться еще чо нибудь, а у каждого проекта свои зависимости (например для билда YOVPS.Desktop придется electron скачивать) - в общем грязь лютая получается.

Идея такая:
Пускай этот репозиторий будет с идеологией некого независимого ядра, в котором из коробки будет идти ТОЛЬКО WebAPI, а кто хочет - используйте либо этот репозиторий как сабмодуль для доступа к самому ядру (например тот же YOVPS.Desktop), или же просто юзайте его в качестве сервера

Итоги:
bubuntoid/YOVPS - ядро/бек:

  • YOVPS.Core
  • YOVPS.Core.UnitTests
  • YOVPS.WebAPI
  • YOVPS.WebAPI.AutoTests

bubuntoid/YOVPS.CLI - command line interface:
todo

webgtx/YOVPS.Desktop (если хочешь):
todo

webgtx/YOVPS.Web (ну или с названием сам там разберешься):
переносим сюда Web UI часть

New folder structure

т.к. планировали в будущем использовать electron для десктопного приложения, предлагаю разделить структуру сурсов:

c:

└───src
    ├───BE
    │ 
    └───FE

на:

└───src
    ├───YOVPS.Core
    ├───YOVPS.Core.UnitTests
    ├───YOVPS.Core.Desktop (новый проект с электроном)
    ├───YOVPS.WebAPI
    └───YOVPS.WebApplication (старый FE)

но вообще чота так каша получается, имхо правильнее было бы тогда использовать разные репозитории -

  • YOVPS
  • YOVPS.Web
  • YOVPS.Desktop
  • YOVPS.Mobile
  • etc

Remove DownloadingTimeout logic

https://github.com/bubuntoid/Ethereal/blob/main/src/Ethereal.Application/Commands/FetchYoutubeVideoCommand.cs

var cts = new CancellationTokenSource();
           var timeoutDate = DateTime.UtcNow.Add(settings.DownloadingTimeout);
           var thread = new Thread(async () =>
           {
               while (true)
               {
                   // Console.WriteLine(timeoutDate.ToLongTimeString() + " || " + DateTime.UtcNow.ToLongTimeString());
                   
                   if (DateTime.UtcNow > timeoutDate)
                   {
                       job.Status = ProcessingJobStatus.Failed;
                       // ReSharper disable once MethodSupportsCancellation
                       await dbContext.SaveChangesAsync();
                       await job.LogAsync(dbContext, "Could not fetch video from youtube. It happens sometimes. Try again.");

                       cts.Cancel();
                       // throw new InternalErrorException(
                       //     "Could not fetch video from youtube. It happens sometimes. Try again.");
                       return;
                   }

                   if (isDownloaded)
                       return;

                   // ReSharper disable once MethodSupportsCancellation
                   await Task.Delay(10);
               }
           });
           thread.Start();

Fix build warnings

Ethereal.Application.csproj: [NU1701] Package 'YoutubeExtractor 0.10.11' was restored using '.NETFramework,Version=v4.6.1, .NETFramework,Version=v4.6.2, .NETFramework,Version=v4.7, .NETFramework,Version=v4.7.1, .NETFramework,Version=v4.7.2, .NETFramework,Version=v4.8' instead of the project target framework 'net5.0'. This package may not be fully compatible with your project.

image

Bug on Server side

If you give a link to the input without timecodes in the description, the server will return an error

old backend todos

Todo / Goals

v1 / mvp:

  • DownloadMp3Async optimization
    • Fork https://github.com/Tyrrrz/YoutubeExplode and reimplement client.Videos.Streams.GetAsync(info)
      add param with specific stream position and length

      смысл в том, что нужно качать не целый видос, а начиная с какого нибудь position(int если seconds/frames, timespan - если в виде "00:00:00") и докачивать не оставшееся время видоса - а какой нибудь length (так же в секундах или фреймы)
  • YOVPS.WebAPI auto tests
  • VideoSplitterService unit tests
  • StringExtensions unit tests
  • TimeSpanExtensions unit tests
  • Add logs
  • dry VideoSplitterService
  • db usage with processing status
    • MariaDb / MySQL / PostgreSQL
  • total optimization
  • thumbnail for each chapter
  • native ffmpeg usage / own client (?)
  • choose bitrate quality
  • fluent validator
  • github ci/cd

fe 29.07.2021 remarks

image
Может сделаем вью таким (речь про зеленый квадратик со скрина) - {name} ({startTimespan}) ?
пример: Side A: (00:00:00)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.