google / mediapy Goto Github PK

View Code? Open in Web Editor NEW

427.0 10.0 19.0 1.41 MB

This Python library makes it easy to display images and videos in a notebook.

Home Page: https://pypi.org/project/mediapy/

License: Apache License 2.0

Python 15.86% Jupyter Notebook 84.08% Jinja 0.07%

image video viewer ironpython jupyter-notebook

mediapy's Introduction

Read/write/show images and videos in an IPython/Jupyter notebook.

[GitHub source] [API docs] [Colab example]

Examples:

See the notebook mediapy_examples.ipynb

Images:

    !pip install -q mediapy
    import mediapy as media
    import numpy as np

    image = media.read_image('https://github.com/hhoppe/data/raw/main/image.png')
    print(image.shape, image.dtype)  # It is a numpy array.
    media.show_image(image)

    checkerboard = np.kron([[0, 1] * 16, [1, 0] * 16] * 16, np.ones((4, 4)))
    media.show_image(checkerboard)

    media.show_image(media.color_ramp((128, 128)), height=48, title='ramp')

    images = {
        'original': image,
        'brightened': media.to_float01(image) * 1.5,
    }
    media.show_images(images)

    media.write_image('/tmp/checkerboard.png', checkerboard)

Videos:

    url = 'https://github.com/hhoppe/data/raw/main/video.mp4'
    video = media.read_video(url)
    print(video.shape, video.dtype)  # It is a numpy array.
    print(video.metadata.fps)  # The 'metadata' attribute includes framerate.
    media.show_video(video)  # Play the video using the retrieved framerate.

    media.show_images(video, height=80, columns=4)  # Show frames side-by-side.

    video = media.moving_circle((128, 128), num_images=10)
    media.show_video(video, fps=10)

    media.write_video('/tmp/video.mp4', video, fps=60)

    # Darken a video frame-by-frame:
    filename_in = '/tmp/video.mp4'
    filename_out = '/tmp/out.mp4'
    with media.VideoReader(filename_in) as r:
      print(f'shape={r.shape} fps={r.fps} bps={r.bps}')
      darken_image = lambda image: media.to_float01(image) * 0.5
      with media.VideoWriter(
          filename_out, shape=r.shape, fps=r.fps, bps=r.bps) as w:
        for image in r:
          w.add_image(darken_image(image))
    media.show_video(media.read_video(filename_out), fps=60)

Setup:

Video I/O relies on the external program ffmpeg, which must be present in the system PATH. On Unix, it can be installed using:

    apt install ffmpeg

or within a notebook using:

    !command -v ffmpeg >/dev/null || (apt update && apt install -y ffmpeg)

mediapy's People

Contributors

Stargazers

Watchers

Forkers

hhoppe iamthatiam777 salemameen fei-xia-oe jamiem0 gmzang isabella232 thexwarn awestlake87 pwhiddy python-repository-hub ghas-results l3aalteshuva aimanifest mattgpt camenduru atifs ruofeidu talairaqe

mediapy's Issues

ffmpeg with copy codecs is not guaranteed to contain number of frames in output

Noticed this when using ffmpeg installed from apt, when vcodec and acodec are copy, the frame= XXX line does not appear in the output, causing api like read_video to fail. The actual output of the ffmpeg -nostdin -i videofile.m4v -acodec copy -vcodec copy -f null - call is

ffmpeg version 6.1-3 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 13 (Debian 13.2.0-5)
  configuration: --prefix=/usr --extra-version=3 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librist --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --disable-sndio --enable-libjxl --enable-pocketsphinx --enable-librsvg --enable-libvpl --disable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libsvtav1 --enable-libx264 --enable-libplacebo --enable-librav1e --enable-shared
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'videofile.m4v':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2023-04-26T22:39:08.000000Z
  Duration: 00:00:16.50, start: 0.000000, bitrate: 5329 kb/s
  Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, progressive), 640x480, 5114 kb/s, 30.10 fps, 29.67 tbr, 90k tbn (default)
    Metadata:
      creation_time   : 2023-04-26T22:39:08.000000Z
      handler_name    : ISO Media file produced by Corp Inc. Created on: 04/26/2023.
      vendor_id       : [0][0][0][0]
    Side data:
      displaymatrix: rotation of 90.00 degrees
  Stream #0:1[0x2](und): Data: none (mett / 0x7474656D), 20 kb/s (default)
    Metadata:
      creation_time   : 2023-04-26T22:39:08.000000Z
      handler_name    : ISO Media file produced by Corp Inc. Created on: 04/26/2023.
  Stream #0:2[0x3](und): Data: none (mett / 0x7474656D), 64 kb/s (default)
    Metadata:
      creation_time   : 2023-04-26T22:39:08.000000Z
      handler_name    : ISO Media file produced by Corp Inc. Created on: 04/26/2023.
  Stream #0:3[0x4](und): Data: none (mett / 0x7474656D), 64 kb/s (default)
    Metadata:
      creation_time   : 2023-04-26T22:39:08.000000Z
      handler_name    : ISO Media file produced by Corp Inc. Created on: 04/26/2023.
Output #0, null, to 'pipe:':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    encoder         : Lavf60.16.100
  Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, progressive), 640x480, q=2-31, 5114 kb/s, 30.10 fps, 29.67 tbr, 90k tbn (default)
    Metadata:
      creation_time   : 2023-04-26T22:39:08.000000Z
      handler_name    : ISO Media file produced by Corp Inc. Created on: 04/26/2023.
      vendor_id       : [0][0][0][0]
    Side data:
      displaymatrix: rotation of 90.00 degrees
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
[out#0/null @ 0x55bbe6083100] video:10287kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
size=N/A time=00:00:16.47 bitrate=N/A speed=2.13e+03x

Incorrect video framerate

A video that is 50seconds @ 24fps appears to be 20s @ 60fps in mediapy

show_image() shows jaggies when the desktop or browser is magnified

(This affects show_image(), show_images(), show_video(), and show_videos().)

By design, when any image or video content is magnified, mediapy shows it pixelated (via a CSS style) -- each image pixel appears as a block of screen pixel values (rather than some bilinear or bicubic filtered reconstruction).
Showing this pixelated view is useful to reveal the original resolution of the image, for example in closeups.
(If one does desire a smoother interpolated version of the image, it can be achieved by first calling media.resize_image().)

With default settings (no width or height attribute), a show_*() call displays the image or video at native resolution, and in that case it should not matter.
However, this is not actually the case.
When there is either (1) desktop DPI scaling (e.g., common in Windows) or (2) browser scaling (e.g., in Chrome using control-+),
the image is displayed over more or fewer pixels than its native resolution.
And in this case, the CSS pixelated setting introduces unexpected ugly jaggies, particularly for vector graphics. See the image below.

The solution is to introduce the CSS pixelated setting only when the show_*() call has explicit height/width parameters that are larger than the image content, i.e., intended to introduce magnification. For flexibility, it's also useful to have a separate pixelated parameter.

I am readying a PR to address this.

Unrecognized option 'qp'

I run the offifial example code but go wrong.

from mediapy import *
video = moving_circle((100, 100), num_images=10)
show_video(video, fps=10)

I am not sure it is relative to torchvision or ffmpeg

when i dont conda install torchvision, the code can run well. but after i conda install torchvision (and ffmpeg also installed automatedly), mediapy can not run well, so how to solve the problem?

torchvision and ffmpeg version are as follows:

BrokenPipeError                           Traceback (most recent call last)
File [~/anaconda3/envs/MSPred/lib/python3.9/site-packages/mediapy/__init__.py:1663](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224e4b552d43562d4c41422d5562756e74753132227d.vscode-resource.vscode-cdn.net//~/anaconda3/envs/MSPred/lib/python3.9/site-packages/mediapy/__init__.py:1663), in write_video(path, images, **kwargs)
   [1662](file:///home/ubuntu/anaconda3/envs/MSPred/lib/python3.9/site-packages/mediapy/__init__.py?line=1661) for image in images:
-> [1663](file:///home/ubuntu/anaconda3/envs/MSPred/lib/python3.9/site-packages/mediapy/__init__.py?line=1662)   writer.add_image(image)

File [~/anaconda3/envs/MSPred/lib/python3.9/site-packages/mediapy/__init__.py:1572](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224e4b552d43562d4c41422d5562756e74753132227d.vscode-resource.vscode-cdn.net//~/anaconda3/envs/MSPred/lib/python3.9/site-packages/mediapy/__init__.py:1572), in VideoWriter.add_image(self, image)
   [1571](file:///home/ubuntu/anaconda3/envs/MSPred/lib/python3.9/site-packages/mediapy/__init__.py?line=1570) assert self._proc.stdin
-> [1572](file:///home/ubuntu/anaconda3/envs/MSPred/lib/python3.9/site-packages/mediapy/__init__.py?line=1571) if self._proc.stdin.write(data) != len(data):
   [1573](file:///home/ubuntu/anaconda3/envs/MSPred/lib/python3.9/site-packages/mediapy/__init__.py?line=1572)   self._proc.wait()

BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
Cell In[13], line 2
      1 video = moving_circle((100, 100), num_images=10)
----> 2 show_video(video, fps=10)

File [~/anaconda3/envs/MSPred/lib/python3.9/site-packages/mediapy/__init__.py:1772](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224e4b552d43562d4c41422d5562756e74753132227d.vscode-resource.vscode-cdn.net//~/anaconda3/envs/MSPred/lib/python3.9/site-packages/mediapy/__init__.py:1772), in show_video(images, title, **kwargs)
   [1749](file:///home/ubuntu/anaconda3/envs/MSPred/lib/python3.9/site-packages/mediapy/__init__.py?line=1748) def show_video(
   [1750](file:///home/ubuntu/anaconda3/envs/MSPred/lib/python3.9/site-packages/mediapy/__init__.py?line=1749)     images: Iterable[_NDArray], *, title: str | None = None, **kwargs: Any
   [1751](file:///home/ubuntu/anaconda3/envs/MSPred/lib/python3.9/site-packages/mediapy/__init__.py?line=1750) ) -> str | None:
   [1752](file:///home/ubuntu/anaconda3/envs/MSPred/lib/python3.9/site-packages/mediapy/__init__.py?line=1751)   """Displays a video in the IPython notebook and optionally saves it to a file.
...
   [1587](file:///home/ubuntu/anaconda3/envs/MSPred/lib/python3.9/site-packages/mediapy/__init__.py?line=1586) self._popen = None

RuntimeError: Error writing '/tmp/tmpugt6gx5m/file.mp4': Unrecognized option 'qp'.
Error splitting the argument list: Option not found

and i tried

ffmpeg pkgs/main/linux-64::ffmpeg-4.2.2-h20bf706_0 runs well

but ffmpeg pytorch/linux-64::ffmpeg-4.3-hf484d3e_0 cannot run well

mediapy `display_images()` downsamples images seemingly arbitrarily....

Issue tested with version: #41

a) If I call media.show_images() once passing it a dict of 32 images, each with shape (512, 512, 3), it stubbornly displays a row matrix of images that are (256, 256, 3) each.

b) If I call media.show_images() 32 times passing it a dict with a single image each time, each with shape (512, 512, 3), it displays the images at the correct resolution (however the images are displayed one below each other which is not what I want).

c) In case a), if I pass the argument height=512 then the images are displayed pixelated 2x: it looks like mediapy makes a 256x256 internal image buffer and displays it pixelated at 512x512.

d) Passing downsample=False doesn't fix the issue either (too easy-no points).

e) I also have a repro case where it dowsamples from 480 to 240 so this seems like a 2x downsample bug.

f) Aha!!! It looks like displaying a smaller number of images (6 vs. 32) works as intended.

Looks like it is some kind of cap on total memory it can consume?

Release 1.2.1 breaks Python 3.8

Hello!

It looks like this line:

mediapy/mediapy/__init__.py

Line 191 in c51276b

_Path = typing.Union[str, os.PathLike[str]]

breaks Python 3.8, because os.PathLike[str] isn't subscriptable.

Here's the full error:

>>> import mediapy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/brentyi/miniconda3/envs/py38/lib/python3.8/site-packages/mediapy/__init__.py", line 191, in <module>
    _Path = typing.Union[str, os.PathLike[str]]
TypeError: 'ABCMeta' object is not subscriptable

cc nerfstudio-project/nerfstudio#3189

`show_video` with a list of rgba images

Is it possible to use show_video with a list of rgba images (i.e., with a transparency channel).

Side by side image comparison slider

This library is awesome. Will like to make a feature request to add some functions that makes it easy to do side by side image comparison with a slider. There are some html/css elements that does this like https://www.npmjs.com/package/img-comparison-slider https://juxtapose.knightlab.com/ which may be useful. A easy way to do slider based image comparison in colabs/notebooks apart from side by side viewing via mediapy will be very convenient.

AttributeError: module 'PIL.Image' has no attribute 'Resampling'

I encounter the following error when reading an image on Google Colab.

%pip install mediapy

import mediapy as media

img = media.read_image('/content/images/yorkshire_terrier_8.jpg')

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

[<ipython-input-16-5d047d0e728a>](https://localhost:8080/#) in <cell line: 1>()
----> 1 import mediapy as media
      2 
      3 img = media.read_image('/content/images/yorkshire_terrier_8.jpg')

2 frames

[/usr/local/lib/python3.9/dist-packages/PIL/Image.py](https://localhost:8080/#) in __getattr__(name)
     75             name, 10, f"{old_resampling[name]} or Resampling.{old_resampling[name]}"
     76         )
---> 77         return Resampling[old_resampling[name]]
     78     msg = f"module '{__name__}' has no attribute '{name}'"
     79     raise AttributeError(msg)

AttributeError: module 'PIL.Image' has no attribute 'Resampling'

According to this StackOverflow answer, this is due to a recent version of Pillow. The version on Google Colab is 9.5.0.

%pip list

[...]
mediapy                       1.1.6
[...]
Pillow                        9.5.0
[...]

By downgrading Pillow, as suggested by the StackOverflow answer, the issue disappears.

%pip install --ignore-installed Pillow==9.0.0

mediapy load, visualize or write a video with audio

Hi, would like to ask if mediapy can load, visualize or write a video with audio in a colab notebook.

I checked the functions in mediapy, but cannot find any audio related code.

I wonder if adding audio into the function is possible or not?

Thank you.

question about write_video

Hi, thanks for open-sourcing the great library!

I am just curious in your demo notebook, when saving the video:

# Write a video to a file:
media.write_video('/tmp/video1.mp4', video1, fps=10, qp=10)

what does 'qp' here mean? I searched online but didn't find anything.

Thanks in advance!

Setting vmin to a value greater than 0 can produce unexpected results for unsigned integer types

Since to_rgb simply subtracts vmin from array, low pixels can underflow to high values, producing unexpected results.

I think it should be simple to fix: upcast array to int32 before subtracting vmin, and clamp to zero afterwards. But I haven't thought through edge cases.

Large videos

Hii, I would like to ask how can I open large videos (.avi format) using mediapy.
The problem is when I try to open large video the system freezes for a while than it is closed with no errors.
Hence I would like to know if mediapy doesn't support large files.
Thanks in advance

`codec="gif"` gives not RGB color

Hi mediapy,
This is an AWESOME tool! One quick question

When I use the codec="gif", the color is not right;

But if I remove that, the color is back:

Do I miss some parameters in order to have the rgb color?

Again, thanks for open-sourcing this wonderful tool!

Intermittent test failure

I got a test failure in https://github.com/hhoppe/mediapy/actions/runs/5408425104/jobs/9827481857 on HEAD:

=============================== warnings summary ===============================
mediapy_test.py::MediapyTest::test_to_type_extreme_value
  /home/runner/work/mediapy/mediapy/mediapy/__init__.py:421: RuntimeWarning: invalid value encountered in cast
    dst = dst.astype(dtype)

Possibly this is due to differences in precision across machines when processing uint64 arrays.
I'm working on a PR to omit dst_dtype=uint64 from the test, and to refactor using @parameterized.parameters so we can identify the precise test parameters that cause a failure.

Will support transform BGR to RGB for 3-channel images?

Hi mediapy team!

Thanks for sharing this awesome tool. It seems that the color map only support grey to RGB currently. Because opencv uses BGR by default and it's also widely used. I want to know if there are some plans to support this channel mode, and if this mode is already supported, could you give me some hints where this interface is?