How should a high-quality still stream and a fast video-stream work together? about libcamera HOT 11 CLOSED

Hi @pietermarsman, thank you for the report. To start with, what sensor are you using, and what resolution still image capture are you looking for? Is it going to be full sensor resolution, or will something like 1080p (i.e. same as the video encode) be good enough? Because of the limited throughput of the VC4 ISP, we are unfortunately not able to pass through large (i.e. > 5Mpix) resolutions through the ISP at full video rates, so there will be certain limitations to work around.

Although libcamera apps can be bent to do what you want here, it might be better to look at picamera2 for such tasks. In fact, one of our examples does pretty much this! I'm sure @davidplowman can provide more details on this script if you need.

from libcamera.

Hi @naushir, thanks for the link to the picamera2 script. I'll try that and see if it does what we need it to do.

We are using the imx219 and imx477. Code is running on a compute module 4 (=raspberry pi 4). The still capture resolution should be 1080p or higher, with hq noise reduction. The video can be lower quality but should be faster than 20 fps. My expectation was and is indeed that those two requirements cannot be combined in one configuration, and hence my question about changing the configuration dynamically.

Currently we are using picamera based on mmal but it has some stability issues when recording videos for a long time. Also we integrate it into a golang code base, so its easier to use c/c++ directly.

Also good to know, we are integrating it into our product so I have plenty of development time. In fact, I reserved the entire month of December for this 😄

from libcamera.

Any dynamic re-configuration is going to cause interruptions in the video stream, this is unavoidable. We can look to ways of minimising this if needed. However to start with, I would look at the picamera2 script, setup a video/capture resolution of 1920x1080, and evaluate the quality to see if it is acceptable.

from libcamera.

I'll definitely do that. Thanks for the help so far!

from libcamera.

I've checked out the script. I think it works like this:

• The picam2.create_video_configuration configures the camera for high framerate. E.g. it uses fast denoising.
• The original capture from the camera is half its sensor resolution, this is probably fast because it can use 2x2 binning.
• The video is recorded in 640x480. The image is captured in half the sensor resolution.

I can see that this works, and why it works. The camera is configured with just one configuration with multiple streams.

Since the overall framerate is determined by the slowest stream this does not fit my usecase. I want to run the video at high framerates 99% of the time. And very occasionally capture a high-quality image. E.g. an image that would have been captured with the picam2.create_still_configuration settings. I'm still wondering what the best way to do this is. And what you would recommend in this scenario.

(Thanks for your help so-far, I'm still reading through the docs and trying things out 😄 )

from libcamera.

The picam2.switch_mode_and_capture_file looks pretty interesting to me. From what I can see that switches to another mode and captures a high-quality image in about 1 second. So also 1 second of frame drop.

from libcamera.

Yes, unfortunately with the hardware we have there really is no way round this. You can have 12MP, but you will max out at 10fps. Or you can have 30fps but that will max out at 3MP. And switching between camera modes involves stopping the sensor, re-programming and re-starting it which is relatively slow.

from libcamera.

As @davidplowman noted, the solution comes down to a choice of either:

1. A lower resolution stills capture to avoid any glitches in the video stream.
2. A high resolution stills capture if approx. 1s of glitches in the video stream are acceptable.

from libcamera.

There is another option (3) that perhaps you might want to consider:

3. Use the camera v2 instead of the HQ camera. This allows you to use a 8MPix 20fps mode. Based on some quick calculations, it ought to be possible for the ISP to achieve the 8*20 = 160 MPix/s. Then you might get a glitch-free video encode stream, and a 8Mpix still capture periodically. You may have to try this out and tuning framerates and output resolutions to see what exactly is achievable.

from libcamera.

Is there anything more to look at here? If not, I'll close this issue down.

from libcamera.

Closing now. Feel free to re-open if you still require more information.

from libcamera.