Make it fast about video2dataset HOT 4 CLOSED

1. Make yt downloading fast. If you want to test out efficiency that isn't bottlenecked by yt_dlp you can use mp4 links (subset of WebVid parquet for example).

from video2dataset.

Next step here is introducing some dataset example and then using for benchmark

from video2dataset.

Under what conditions is this complete? Let's specify some of these points so we can actually finish v0

from video2dataset.

Ideal speed is bottleneck of network.
So for example on a 10Gbps connection that means 1.25GB/s
So to measure how close we are from this we need to measure, during download, the size of videos
That way we can have a total bandwidth metric and a per worker metric

If we can reach 60% of the maximum speed in average I'd say it's good enough.

Another interesting metric is video/s
Yet another is video hour/s/core

We want to maximize those. This can be a different criterion.
One hour of video should be about 1GB, so it could take one second at best.

If we can process one hour of video every 10 seconds on one core it would be good

I am not sure if video/s metric is truly usable

So yeah let's:

• Add metrics video hour/s
• Add metric byte/s

Then it should be clear how close we are to getting this issue done

from video2dataset.