leandromoreira / digital_video_introduction Goto Github PK

In the Chrome subsampling part, it wasn't specified what the individual numbers exactly mean. In the 4:2:0 example, why were the Cb and Cr occupying only horizontal half of what the Y part was covering? Specifically, what do the 4, 2 and 0 even mean?

Including the slide 48 from the linked Google Sheets would maybe help understand everything better. This website goes into much more (maybe too much) detail, so maybe link as a side note in case people don't immediately understand?

Using Win10 Home on my desktop, so no Hyper-V :(

Any chance of using another Virtual Machine with this? If so, any tips?

Regardless, seems an excellent tutorial/intro that I intend to dig into repeatedly as time permits. Thank you so much for this!

docker for mac version 17.06.0-ce-mac19 (18663)
I have ./setup.sh successfully
while try ./s/start_jupyter.sh docker returns error:
docker: Error response from daemon: ContainerInspect failed.
ERRO[0000] error waiting for container: context canceled

How to setup docker properly?
Thanks~

In README.md, there is a source of graph shows how the first and second coefficient is calculated, using weights which are unique for each index.
But the Source : https://www.iem.thm.de/telekom-labor/zinke/mk/mpeg2beg/whatisit.htm return 404.

Hi, I got this error when trigger the ./setup.sh, and my OS is MacOS 10.15.6.

Unable to find image 'jrottenberg/ffmpeg:3.3' locally
3.3: Pulling from jrottenberg/ffmpeg
35b42117c431: Pull complete
ad9c569a8d98: Pull complete
293b44f45162: Pull complete
0c175077525d: Pull complete
2fcff9069a9f: Pull complete
248d2888375c: Pull complete
33f41cdaf34e: Pull complete
Digest: sha256:b41f44a20238f36797f44f26e93b01e3b1d86adb3d40aa32a2cb845ef5d1eb64
Status: Downloaded newer image for jrottenberg/ffmpeg:3.3
ffmpeg version 3.3.9 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.11) 20160609
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libass --enable-libfreetype --enable-libvidstab --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-libkvazaar --enable-postproc --enable-small --enable-version3 --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib --extra-libs=-ldl --prefix=/opt/ffmpeg
  libavutil      55. 58.100 / 55. 58.100
  libavcodec     57. 89.100 / 57. 89.100
  libavformat    57. 71.100 / 57. 71.100
  libavdevice    57.  6.100 / 57.  6.100
  libavfilter     6. 82.100 /  6. 82.100
  libavresample   3.  5.  0 /  3.  5.  0
  libswscale      4.  6.100 /  4.  6.100
  libswresample   2.  7.100 /  2.  7.100
  libpostproc    54.  5.100 / 54.  5.100
/files/v/bunny_1080p_60fps.mp4: Operation not permitted
ffmpeg version 3.3.9 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.11) 20160609
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libass --enable-libfreetype --enable-libvidstab --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-libkvazaar --enable-postproc --enable-small --enable-version3 --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib --extra-libs=-ldl --prefix=/opt/ffmpeg
  libavutil      55. 58.100 / 55. 58.100
  libavcodec     57. 89.100 / 57. 89.100
  libavformat    57. 71.100 / 57. 71.100
  libavdevice    57.  6.100 / 57.  6.100
  libavfilter     6. 82.100 /  6. 82.100
  libavresample   3.  5.  0 /  3.  5.  0
  libswscale      4.  6.100 /  4.  6.100
  libswresample   2.  7.100 /  2.  7.100
  libpostproc    54.  5.100 / 54.  5.100
/files/v/bunny_1080p_30fps.mp4: Operation not permitted

The video files that the setup script is trying to download does not seem to be available anymore?

Awesome document 💯. Could I translate this document into simplified Chinese and share with more Chinese developers, please? We have an organization, and we have translated more than 560 articles. We are looking forward to hearing from you.

About our organization: github.com/xitu/gold-miner/

Converting between YCbCr and RGB

Some may argue, how can we produce all the colors without using the green?

To answer this question we'll walk through a conversion from RGB to YCbCr. We'll use the coefficients from the standard BT.601 that was recommended by the group ITU-R* . The first step is to calculate the luma, we'll use the constants suggested by ITU and replace the RGB values.

Y = 0.299R + 0.587G + 0.114B

Once we had the luma, we can split the colors (chroma blue and red):

Cb = 0.564(B - Y)
Cr = 0.713(R - Y)

And we can also convert it back and even get the green by using YCbCr.

R = Y + 1.402Cr
B = Y + 1.772Cb
G = Y - 0.344Cb - 0.714Cr

is this done for each pixel ?

and i didn't understand how did you use the diagram to get the Y = R + G + B values.. and where did you get the constants from
can you give an example ?

In the 6th step of "A generic codec" you have the following line:

The second byte (binary=01100100, hex=0x64, dec=100) of an SPS NAL is the field profile_idc which shows the profile that the encoder has used, in this case, we used the constrained high-profile, it's a high profile without the support of B (bi-predictive) slices.

Maybe it should be just "high profile"? Because the next byte is 0x00 and this next byte determines the actual type of High Profile (constrained or progressive). And if this byte is 0x00 then I guess it is just High Profile.

Please fix me if I'm wrong.

chroma subsampling, yuv420, our eye "weakness", frame (picture) types (I B P), temporal redundancy, motion estimation, spatial redundancy,

2nd step - predictions contains these links

ccd, cmos, Bayer Demosaicing,

Hello.

Does this tutorial cover MAS - Minimalist Adaptive Streamming?

I was trying to execute the hands-on example below,
https://github.com/leandromoreira/digital_video_introduction#hands-on-play-around-with-image-and-color

Following is the output that i got,

Executing the command: jupyter notebook
[I 15:14:55.718 NotebookApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
[I 15:14:56.435 NotebookApp] JupyterLab extension loaded from /opt/conda/lib/python3.8/site-packages/jupyterlab
[I 15:14:56.436 NotebookApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 15:14:56.438 NotebookApp] Serving notebooks from local directory: /home/jovyan
[I 15:14:56.438 NotebookApp] Jupyter Notebook 6.1.3 is running at:
[I 15:14:56.439 NotebookApp] http://da8379b349f0:8888/?token=fd62f89f08afd54b0b9f00a83288009de18d0d4e8775221d
[I 15:14:56.439 NotebookApp] or http://127.0.0.1:8888/?token=fd62f89f08afd54b0b9f00a83288009de18d0d4e8775221d
[I 15:14:56.439 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 15:14:56.442 NotebookApp]

To access the notebook, open this file in a browser:
file:///home/jovyan/.local/share/jupyter/runtime/nbserver-6-open.html
Or copy and paste one of these URLs:
http://da8379b349f0:8888/?token=fd62f89f08afd54b0b9f00a83288009de18d0d4e8775221d
or http://127.0.0.1:8888/?token=fd62f89f08afd54b0b9f00a83288009de18d0d4e8775221d

Of the above, only the URL http://127.0.0.1:8888/?token=fd62f89f08afd54b0b9f00a83288009de18d0d4e8775221d was working for me.

Below is the snapshot of the URL that is loaded from my browser,

I am unable to proceed further as to how to execute the 'ipython' script for the hands-on task ( do not find the ipython script loaded in the URL tab). Any further inputs on how to run a sample jupyter notebook hands-on exercise would be helpful. (Yes, I did read the section on "How to use jupyter")

Thank you

Hi,
Thanks for wonderful documentation.

I'm having some doubt on YCbCr 422 vs YCbCr 420 file size. As per my understanding in case of 4:2:0, the file size reduction is more than 50% compared to 4:4:4. In case of 1280x720 the alternate lines will not take Cb or Cr. Am I right?
If yes then 'Y' pixels = 1280x720x8 bits
CbCr pixels = 640x360x8 bits
Total Bytes = 1152000
Size = 115GBytes.

For 4:2:2, the alternate lines do take chroma samples and hence
Y pixels = 1280x720x8 bits
CbCr pixels = 640x720x8
Total Bytes = 139GBytes

Kindly correct me.

we have a color depth of 24 (8 * 3) bits and we can also infer that we could use 2 to the power of 24 different colors.
Can you give me an example?

I'm triying to run the example provided in debug video Example to generate a debug video with macro blocks. But I get the next error:

[NULL @ 0x9219c0] Unable to parse option value "vis_mb_type"
[NULL @ 0x9219c0] Error setting option debug to value vis_mb_type.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x91ff80] Failed to open codec in avformat_find_stream_info
[NULL @ 0x922d40] [Eval @ 0x7ffd11e97290] Undefined constant or missing '(' in 'vis_mb_type'
[NULL @ 0x922d40] Unable to parse option value "vis_mb_type"
[NULL @ 0x922d40] Error setting option debug to value vis_mb_type.

The image that docker runs from jrottenberg/ffmpeg contanis:

ffmpeg version 4.1 Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.11) 20160609

This doesnt really matter that much, but this says "Luigi" instead of "green" lol

https://github.com/leandromoreira/digital_video_introduction/blob/master/i/rgb_channels_intensity.png

Now the majority of the videos are done in BT 709, which is used by HD television, a comparison with the standard for SD BT.601 should be good addition.

Hello guys.
I've come across your repository while looking for more info on Minimalist Adaptive Streamming. Does anyone have information that could share?

Many thanks.

In P-Frame section, it is mentioned "We can rebuild frame 1, only using the difference and referencing to the previous frame". Shouldn't it be "frame 2"?

Kindly correct me.

Just wanted to leave a thank you note that I've been meaning to learn about codecs & related video
technologies for so long, but always felt intimidated by the terminology and complexity.

Your project on Github made it really approachable and to the point! Waiting for the sections on Online Streaming - That's what i really want to focus on as my background is in Web Development.

Thank you once again for putting together such great material!

Hi guys just wanted to give people a heads up that I've started a Russian translation of the document. Anyone feeling like helping out is more than welcome to do so.

Cheers! B

Hi scientists, i want make the U and V image, what should i do? i have try:
y, u, v <- r, g, b
r, g, b <- 0, u, 0
and i failed

The color contrast between green and white makes the images really hard to read. I'd be happy to update the images to WCAG contrast standards.

container vs codec, history, current situation av1 hevc, picture partitioning, predictions, transform, quantization, entropy coding, bitstream format (h264 example), comparing why and how h265 is getting better results than h264

(specifically h264 tunning) CBR vs VBR, profiles, frame types, inspect what the big players are doing (yt, netflix, ..), entropy,

I executed the following command in oreder to generate images from video.
But images were not generated.
What should I do ?

https://github.com/leandromoreira/digital_video_introduction/blob/master/encoding_pratical_examples.md#generate-images-from-video

• ffmepg: 3.3.9

vagrant@example:~/Documents/digital_video_introduction (master)$ ./s/ffmpeg -y -i /files/v/small_bunny_1080p_30fps.mp4 -ss 00:01:24 -t 00:00:01  /files/v/smallest_bunny_1080p_30fps_%3d.jpg
ffmpeg version 3.3.9 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.11) 20160609
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libass --enable-libfreetype --enable-libvidstab --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-libkvazaar --enable-postproc --enable-small --enable-version3 --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib --extra-libs=-ldl --prefix=/opt/ffmpeg
  libavutil      55. 58.100 / 55. 58.100
  libavcodec     57. 89.100 / 57. 89.100
  libavformat    57. 71.100 / 57. 71.100
  libavdevice    57.  6.100 / 57.  6.100
  libavfilter     6. 82.100 /  6. 82.100
  libavresample   3.  5.  0 /  3.  5.  0
  libswscale      4.  6.100 /  4.  6.100
  libswresample   2.  7.100 /  2.  7.100
  libpostproc    54.  5.100 / 54.  5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/files/v/small_bunny_1080p_30fps.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    title           : Big Buck Bunny, Sunflower version
    artist          : Blender Foundation 2008, Janus Bager Kristensen 2013
    composer        : Sacha Goedegebure
    encoder         : Lavf57.56.100
    comment         : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
    genre           : Animation
  Duration: 00:00:10.02, start: 0.000000, bitrate: 2274 kb/s
    Stream #0:0(und): Video: h264 (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 1927 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (mp4a / 0x6134706D), 48000 Hz, 5.1, fltp, 340 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
[swscaler @ 0x204c360] deprecated pixel format used, make sure you did set range correctly
Output #0, image2, to '/files/v/smallest_bunny_1080p_30fps_%3d.jpg':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    title           : Big Buck Bunny, Sunflower version
    artist          : Blender Foundation 2008, Janus Bager Kristensen 2013
    composer        : Sacha Goedegebure
    genre           : Animation
    comment         : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
    encoder         : Lavf57.71.100
    Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 30 fps, 30 tbn, 30 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc57.89.100 mjpeg
    Side data:
      cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: -1
frame=    0 fps=0.0 q=0.0 Lsize=N/A time=00:00:00.00 bitrate=N/A speed=   0x    
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)

setup.sh calls s/cut_smaller_video.sh
while the file uri in s/cut_smaller_video.sh is strange, it starts with '/file'

Should the End of stream's NAL type id is 11, not 12?

I've found the following images which need to also be updated to a contrast ratio of 7:1 or above:

• codec_history_timeline.png
• h264_bitstream_macro_diagram.png
• nal_units.png

Since I already did issue #130 I have the colors and templates set up so I can fix this as well. It might take a little longer as I need to find time over the next week.

I'm having trouble reading the data flow of the block diagram presented in the Review section. I did some self-study to try and understand it but so far with little success.

When trying to read it I get something like "The input video - (intra frame prediction and inter frame prediction) gets transformed, then quantize and finally do entropy encoding to get the bitstream." Is this right? And the line after intra frame prediction and inter frame prediction can either go to the block before transform or the block before loop filters. How do you know which path this flow will take? And when does the flow go to inverse transform, is it for decoding? If it is for decoding I'd expect to see some output video and input bitstream paths, is that wrong?

resolution, display aspect ratio, pixel aspect ratio, video, interlaced, progressive, bitrate, CBR, VBR, ABR,

There are a lot of links in the text that point to:
https://github.com/leandromoreira/digital_video_introduction/blob/master/... seems like the notebook links are mostly working but the other ones are not

first example interlacing image is wrong, shows running dogo with simulated division into scan lines, but does not take into account timing difference - that was one of the mayor sources of deinterlacing artifacts. Alternating fields are 1/60 second apart in time.

http://www.onlinevideo.net/2011/05/learn-the-basics-of-deinterlacing-your-online-videos/

also " it basically sends half of the screen in 1 "frame" and the other half in the next "frame"." line is wrong. Should be "sends odd numbered lines of screen 1 in 1 frame and the even numbered lines of screen 2 in the next "frame".

I think, below sentence (other is same), describes about "video coding format", not "video codec"
"The video codec H.261 was born in 1990 (technically 1988), and it was designed to work with data rates of 64 kbit/s. It already uses ideas such as chroma subsampling, macro block, etc. In the year of 1995, the H.263 video codec standard was published and continued to be extended until 2001."

Refer:
https://en.wikipedia.org/wiki/Video_codec
https://en.wikipedia.org/wiki/Video_coding_format

It should have been called green instead of luigi.

Hi, @leandromoreira, thanks for your detailed tutorial.

In Simplistic explanation of how our eyes work section, you mentioned The rod cells are mostly responsible for brightness while the cone cells are responsible for color as why we are more sensitive to brightness. The conclusion is alright but, I think, cone cells also responsible for brightness and here is the words from wikipedia:

They(Cones) are responsible for color vision and function best in relatively bright light, as opposed to rod cells, which work better in dim light...
Cones are less sensitive to light than the rod cells in the retina... (That means cones also sense light, does not?)

I believe cone cells also responsible for light when the surrounding is bright enough. If rods also works in sunshine, why eyes need take time to adjust when we go to a dark room? I think it's because rods do not work in brightness, so there is a "warm up" process to let rods work.

Cone cell - wikipedia

I think it's worth to mention about deblocking / in-loop filter which is used to remove partitioning artifacts.

&> in setup.sh is Bash-ism. It does not work in, say, Dash.
So it might be a good idea to include Bash shabang to indicate the script expects Bash.

The corner case I ran into is that ./setup.sh works as expected.
But sudo ./setup.sh gives Error: docker command not found, you must install it before.
I probably should allow non-root use of Docker on my system instead but I imagine it might become a problem for other people too.

SYSTEM Details:
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
REVISION=4.15.0-106-generic
Docker version:docker-ce_19.03.93-0ubuntu-xenial_amd64.deb

ISSUE Details:
$ ./setup.sh
.
.
docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/create: dial unix /var/run/docker.sock: connect: permission denied.

scipy.misc.bytescale method does not exists in jupyter/scipy-notebook docker image.

https://github.com/leandromoreira/digital_video_introduction/blob/master/uniform_quantization_experience.ipynb

import numpy as np
import matplotlib.pyplot as plt
from scipy import fftpack
from scipy.misc import bytescale
import matplotlib.image as mpimg
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-17733afde104> in <module>
      2 import matplotlib.pyplot as plt
      3 from scipy import fftpack
----> 4 from scipy.misc import bytescale
      5 import matplotlib.image as mpimg
ImportError: cannot import name 'bytescale' from 'scipy.misc' (/opt/conda/lib/python3.8/site-packages/scipy/misc/__init__.py)

• python 3.8.8
• scipy 1.6.3

I suggest specifying tag of docker image jupyter/scipy-notebook.
https://github.com/leandromoreira/digital_video_introduction/blob/master/s/start_jupyter.sh

AOM, ISO/IEC, ITU-R, JVT/JCT, MPEG-LA, rec 601, rec 709, rec 2020

The text says "y is the number of changes of chroma samples between the first and seconds rows of a pixels."

But isn't it more accurate to say:

"y is the number of changes of chroma samples between the first and last rows of a pixels."

If not, I still don't understand how chroma subsampling is used!