Comments (6)
davidplowman
commented on August 17, 2024
This is an area we've quite wanted to do work on, but haven't got round to it.
It should be possible to synchronise a pair of cameras with a software control loop. The things you'd have to do would be:
-
Look at the
"SensorTimestamp"value in the metadata with each frame. By comparing these you can obviously tell how unsynchronised the cameras are. -
Adjust the
"FrameDurationLimits"control value to bring the frames together.
from picamera2.
mgineer85
commented on August 17, 2024
Thank you, that was the starter information I needed. I will check it out!
from picamera2.
mgineer85
commented on August 17, 2024
``So, this is cool, seems to work.
Finally I wanted to save stills so I started directly using the create_still_configuration which implicitely uses just one buffer. When the control loop is close to zero delta it could lead to a jump in the SensorTimestamp by 1/FPS seconds due to a dropped frame. I fixed by using a buffer, here buffer_count=3.
This is my very basic script, that just syncronizes, but nothing more yet:
#!/usr/bin/python
from picamera2 import Picamera2
def P_controller(Kp: float = 0.05, setpoint: float = 0, measurement: float = 0, output_limits=(-10000, 10000)):
e = setpoint - measurement
P = Kp * e
output_value = P
# output and limit if output_limits set
lower, upper = output_limits
if (upper is not None) and (output_value > upper):
return upper
elif (lower is not None) and (output_value < lower):
return lower
return output_value
if len(Picamera2.global_camera_info()) <= 1:
print("SKIPPED (one camera)")
quit()
# Primary (leads)
picam2a = Picamera2(0)
# need buffer_count > 1 because if a frame is skipped, there will be a jump in SensorTimestamp due to dropped frame which messes with the control
config2a = picam2a.create_still_configuration(controls={"FrameRate": 10}, buffer_count=3)
picam2a.configure(config2a)
# Secondary (follows)
picam2b = Picamera2(1)
# need buffer_count > 1 because if a frame is skipped, there will be a jump in SensorTimestamp due to dropped frame which messes with the control
config2b = picam2b.create_still_configuration(controls={"FrameRate": 10}, buffer_count=3)
picam2b.configure(config2b)
picam2a.start()
picam2b.start()
print("Press Ctrl+C to exit")
try:
while True:
metadata_picam2a = picam2a.capture_metadata()
metadata_picam2b = picam2b.capture_metadata()
timestamp_picam2a = metadata_picam2a["SensorTimestamp"] / 1000 # convert ns to µs because all other values are in µs
timestamp_picam2b = metadata_picam2b["SensorTimestamp"] / 1000 # convert ns to µs because all other values are in µs
timestamp_delta = timestamp_picam2b - timestamp_picam2a
controller_output_frameduration_delta = int(P_controller(0.05, 0, timestamp_delta, (-10000, 10000)))
control_out_frameduration = int(metadata_picam2a["FrameDuration"] + controller_output_frameduration_delta) # sync to a, so use that for ref
print("Cam A: SensorTimestamp: ", timestamp_picam2a, " FrameDuration: ", metadata_picam2a["FrameDuration"])
print("Cam B: SensorTimestamp: ", timestamp_picam2b, " FrameDuration: ", metadata_picam2b["FrameDuration"])
print("SensorTimestampDelta: ", round(timestamp_delta / 1000, 1), "ms")
print("FrameDurationDelta: ", controller_output_frameduration_delta, "new FrameDurationLimit: ", control_out_frameduration)
with picam2b.controls as ctrl:
# set new FrameDurationLimits based on P_controller output.
ctrl.FrameDurationLimits = (control_out_frameduration, control_out_frameduration)
except KeyboardInterrupt:
print("got Ctrl+C, exiting")
picam2a.stop()
picam2b.stop()
This is the output. The delta remains very stable close to zero after first start:
Cam A: SensorTimestamp: 10227724610.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10227752441.0 FrameDuration: 97300 SensorTimestampDelta: 27.8 ms
FrameDurationDelta: -1391 new FrameDurationLimit: 98591
Cam A: SensorTimestamp: 10227824592.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10227849742.0 FrameDuration: 97510 SensorTimestampDelta: 25.1 ms
FrameDurationDelta: -1257 new FrameDurationLimit: 98725
Cam A: SensorTimestamp: 10227924577.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10227947255.0 FrameDuration: 97721 SensorTimestampDelta: 22.7 ms
FrameDurationDelta: -1133 new FrameDurationLimit: 98849
Cam A: SensorTimestamp: 10228024561.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10228044979.0 FrameDuration: 97931 SensorTimestampDelta: 20.4 ms
FrameDurationDelta: -1020 new FrameDurationLimit: 98962
Cam A: SensorTimestamp: 10228124544.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10228142911.0 FrameDuration: 98115 SensorTimestampDelta: 18.4 ms
FrameDurationDelta: -918 new FrameDurationLimit: 99064
Cam A: SensorTimestamp: 10228224528.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10228241027.0 FrameDuration: 98273 SensorTimestampDelta: 16.5 ms
FrameDurationDelta: -824 new FrameDurationLimit: 99158
Cam A: SensorTimestamp: 10228324509.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10228339302.0 FrameDuration: 98430 SensorTimestampDelta: 14.8 ms
FrameDurationDelta: -739 new FrameDurationLimit: 99243
Cam A: SensorTimestamp: 10228424491.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10228437736.0 FrameDuration: 98588 SensorTimestampDelta: 13.2 ms
FrameDurationDelta: -662 new FrameDurationLimit: 99320
Cam A: SensorTimestamp: 10228524475.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10228536326.0 FrameDuration: 98720 SensorTimestampDelta: 11.9 ms
FrameDurationDelta: -592 new FrameDurationLimit: 99390
Cam A: SensorTimestamp: 10228624458.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10228635047.0 FrameDuration: 98825 SensorTimestampDelta: 10.6 ms
FrameDurationDelta: -529 new FrameDurationLimit: 99453
Cam A: SensorTimestamp: 10228724440.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10228733874.0 FrameDuration: 98956 SensorTimestampDelta: 9.4 ms
FrameDurationDelta: -471 new FrameDurationLimit: 99511
Cam A: SensorTimestamp: 10228824423.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10228832833.0 FrameDuration: 99061 SensorTimestampDelta: 8.4 ms
FrameDurationDelta: -420 new FrameDurationLimit: 99562
Cam A: SensorTimestamp: 10228924407.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10228931899.0 FrameDuration: 99140 SensorTimestampDelta: 7.5 ms
FrameDurationDelta: -374 new FrameDurationLimit: 99608
Cam A: SensorTimestamp: 10229024390.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10229031037.0 FrameDuration: 99219 SensorTimestampDelta: 6.6 ms
FrameDurationDelta: -332 new FrameDurationLimit: 99650
Cam A: SensorTimestamp: 10229124372.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10229130260.0 FrameDuration: 99298 SensorTimestampDelta: 5.9 ms
FrameDurationDelta: -294 new FrameDurationLimit: 99688
Cam A: SensorTimestamp: 10229224355.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10229229557.0 FrameDuration: 99377 SensorTimestampDelta: 5.2 ms
FrameDurationDelta: -260 new FrameDurationLimit: 99722
Cam A: SensorTimestamp: 10229324338.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10229328939.0 FrameDuration: 99430 SensorTimestampDelta: 4.6 ms
FrameDurationDelta: -230 new FrameDurationLimit: 99752
Cam A: SensorTimestamp: 10229424322.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10229428371.0 FrameDuration: 99508 SensorTimestampDelta: 4.0 ms
FrameDurationDelta: -202 new FrameDurationLimit: 99780
Cam A: SensorTimestamp: 10229524308.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10229527881.0 FrameDuration: 99561 SensorTimestampDelta: 3.6 ms
FrameDurationDelta: -178 new FrameDurationLimit: 99804
Cam A: SensorTimestamp: 10229624287.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10229627440.0 FrameDuration: 99587 SensorTimestampDelta: 3.2 ms
FrameDurationDelta: -157 new FrameDurationLimit: 99825
Cam A: SensorTimestamp: 10229724271.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10229727031.0 FrameDuration: 99640 SensorTimestampDelta: 2.8 ms
FrameDurationDelta: -138 new FrameDurationLimit: 99844
Cam A: SensorTimestamp: 10229824253.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10229826672.0 FrameDuration: 99666 SensorTimestampDelta: 2.4 ms
FrameDurationDelta: -120 new FrameDurationLimit: 99862
Cam A: SensorTimestamp: 10229924236.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10229926340.0 FrameDuration: 99719 SensorTimestampDelta: 2.1 ms
FrameDurationDelta: -105 new FrameDurationLimit: 99877
Cam A: SensorTimestamp: 10230024220.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10230026062.0 FrameDuration: 99745 SensorTimestampDelta: 1.8 ms
FrameDurationDelta: -92 new FrameDurationLimit: 99890
Cam A: SensorTimestamp: 10230124202.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10230125808.0 FrameDuration: 99771 SensorTimestampDelta: 1.6 ms
FrameDurationDelta: -80 new FrameDurationLimit: 99902
Cam A: SensorTimestamp: 10230224185.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10230225581.0 FrameDuration: 99798 SensorTimestampDelta: 1.4 ms
FrameDurationDelta: -69 new FrameDurationLimit: 99913
Cam A: SensorTimestamp: 10230324169.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10230325381.0 FrameDuration: 99824 SensorTimestampDelta: 1.2 ms
FrameDurationDelta: -60 new FrameDurationLimit: 99922
Cam A: SensorTimestamp: 10230424151.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10230425207.0 FrameDuration: 99824 SensorTimestampDelta: 1.1 ms
FrameDurationDelta: -52 new FrameDurationLimit: 99930
Cam A: SensorTimestamp: 10230524135.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10230525032.0 FrameDuration: 99850 SensorTimestampDelta: 0.9 ms
FrameDurationDelta: -44 new FrameDurationLimit: 99938
Cam A: SensorTimestamp: 10230624117.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10230624885.0 FrameDuration: 99876 SensorTimestampDelta: 0.8 ms
FrameDurationDelta: -38 new FrameDurationLimit: 99944
Cam A: SensorTimestamp: 10230724099.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10230724763.0 FrameDuration: 99876 SensorTimestampDelta: 0.7 ms
FrameDurationDelta: -33 new FrameDurationLimit: 99949
Cam A: SensorTimestamp: 10230824083.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10230824641.0 FrameDuration: 99876 SensorTimestampDelta: 0.6 ms
FrameDurationDelta: -27 new FrameDurationLimit: 99955
Cam A: SensorTimestamp: 10230924066.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10230924525.0 FrameDuration: 99903 SensorTimestampDelta: 0.5 ms
FrameDurationDelta: -22 new FrameDurationLimit: 99960
Cam A: SensorTimestamp: 10231024049.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10231024430.0 FrameDuration: 99903 SensorTimestampDelta: 0.4 ms
FrameDurationDelta: -19 new FrameDurationLimit: 99963
Cam A: SensorTimestamp: 10231124032.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10231124335.0 FrameDuration: 99929 SensorTimestampDelta: 0.3 ms
FrameDurationDelta: -15 new FrameDurationLimit: 99967
Cam A: SensorTimestamp: 10231224016.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10231224265.0 FrameDuration: 99929 SensorTimestampDelta: 0.2 ms
FrameDurationDelta: -12 new FrameDurationLimit: 99970
Cam A: SensorTimestamp: 10231323998.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10231324197.0 FrameDuration: 99929 SensorTimestampDelta: 0.2 ms
FrameDurationDelta: -9 new FrameDurationLimit: 99973
Cam A: SensorTimestamp: 10231423981.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10231424128.0 FrameDuration: 99929 SensorTimestampDelta: 0.1 ms
FrameDurationDelta: -7 new FrameDurationLimit: 99975
Cam A: SensorTimestamp: 10231523964.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10231524058.0 FrameDuration: 99929 SensorTimestampDelta: 0.1 ms
FrameDurationDelta: -4 new FrameDurationLimit: 99978
Cam A: SensorTimestamp: 10231623948.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10231623985.0 FrameDuration: 99955 SensorTimestampDelta: 0.0 ms
FrameDurationDelta: -1 new FrameDurationLimit: 99981
Cam A: SensorTimestamp: 10231723930.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10231723949.0 FrameDuration: 99955 SensorTimestampDelta: 0.0 ms
FrameDurationDelta: 0 new FrameDurationLimit: 99982
Cam A: SensorTimestamp: 10231823927.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10231823907.0 FrameDuration: 99955 SensorTimestampDelta: -0.0 ms
FrameDurationDelta: 1 new FrameDurationLimit: 99983
Cam A: SensorTimestamp: 10231923889.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10231923865.0 FrameDuration: 99955 SensorTimestampDelta: -0.0 ms
FrameDurationDelta: 1 new FrameDurationLimit: 99983
Cam A: SensorTimestamp: 10232023872.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10232023823.0 FrameDuration: 99955 SensorTimestampDelta: -0.0 ms
FrameDurationDelta: 2 new FrameDurationLimit: 99984
Cam A: SensorTimestamp: 10232123855.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10232123780.0 FrameDuration: 99955 SensorTimestampDelta: -0.1 ms
FrameDurationDelta: 3 new FrameDurationLimit: 99985
Cam A: SensorTimestamp: 10232223843.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10232223737.0 FrameDuration: 99955 SensorTimestampDelta: -0.1 ms
FrameDurationDelta: 5 new FrameDurationLimit: 99987
Cam A: SensorTimestamp: 10232323825.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10232323693.0 FrameDuration: 99955 SensorTimestampDelta: -0.1 ms
FrameDurationDelta: 6 new FrameDurationLimit: 99988
Cam A: SensorTimestamp: 10232423809.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10232423652.0 FrameDuration: 99955 SensorTimestampDelta: -0.2 ms
FrameDurationDelta: 7 new FrameDurationLimit: 99989
Cam A: SensorTimestamp: 10232523791.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10232523607.0 FrameDuration: 99982 SensorTimestampDelta: -0.2 ms
FrameDurationDelta: 9 new FrameDurationLimit: 99991
Cam A: SensorTimestamp: 10232623774.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10232623592.0 FrameDuration: 99982 SensorTimestampDelta: -0.2 ms
FrameDurationDelta: 9 new FrameDurationLimit: 99991
Cam A: SensorTimestamp: 10232723757.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10232723574.0 FrameDuration: 99982 SensorTimestampDelta: -0.2 ms
FrameDurationDelta: 9 new FrameDurationLimit: 99991
Cam A: SensorTimestamp: 10232823740.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10232823558.0 FrameDuration: 99982 SensorTimestampDelta: -0.2 ms
FrameDurationDelta: 9 new FrameDurationLimit: 99991
Cam A: SensorTimestamp: 10232923723.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10232923543.0 FrameDuration: 99982 SensorTimestampDelta: -0.2 ms
FrameDurationDelta: 9 new FrameDurationLimit: 99991
Cam A: SensorTimestamp: 10233023706.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10233023525.0 FrameDuration: 99982 SensorTimestampDelta: -0.2 ms
FrameDurationDelta: 9 new FrameDurationLimit: 99991
Cam A: SensorTimestamp: 10233123689.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10233123509.0 FrameDuration: 99982 SensorTimestampDelta: -0.2 ms
FrameDurationDelta: 9 new FrameDurationLimit: 99991
Cam A: SensorTimestamp: 10233223674.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10233223493.0 FrameDuration: 99982 SensorTimestampDelta: -0.2 ms
FrameDurationDelta: 9 new FrameDurationLimit: 99991
Cam A: SensorTimestamp: 10233323655.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10233323476.0 FrameDuration: 99982 SensorTimestampDelta: -0.2 ms
FrameDurationDelta: 8 new FrameDurationLimit: 99990
Cam A: SensorTimestamp: 10233423638.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10233423461.0 FrameDuration: 99982 SensorTimestampDelta: -0.2 ms
FrameDurationDelta: 8 new FrameDurationLimit: 99990
Cam A: SensorTimestamp: 10233523621.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10233523444.0 FrameDuration: 99982 SensorTimestampDelta: -0.2 ms
FrameDurationDelta: 8 new FrameDurationLimit: 99990
Cam A: SensorTimestamp: 10233623604.0 FrameDuration: 99982
Cam B: SensorTimestamp: 10233623428.0 FrameDuration: 99982 SensorTimestampDelta: -0.2 ms
FrameDurationDelta: 8 new FrameDurationLimit: 99990
^Cgot Ctrl+C, exiting
from picamera2.
davidplowman
commented on August 17, 2024
Very cool, thanks for the update!
from picamera2.
mgineer85
commented on August 17, 2024
I added also sync capture triggered by gpio on pi5. Pictures came out synced perfectly and got no hiccups after photos were taken in the synchronization.
You mind if I send a minimal example via pullrequest?
from picamera2.
davidplowman
commented on August 17, 2024
Yes, please do send new examples. If you can check the contribution guidelines that would be helpful - I think it boils down mostly to running flake8, signing the commit, and submitting it to the "next" branch.
from picamera2.
Related Issues (20)
- [HOW-TO] change the exposition mode and constraint? HOT 4
- [HOW-TO] Reduce delay after switching sensor modes. HOT 4
- [SUGGESTION] HOT 1
- [OTHER] Please improve/update Documentation HOT 3
- [BUG]Program blocking due to camera disconnection HOT 1
- Unexpected warning from v4l2 HOT 1
- strange video size increase with waiting time between video captures
- [SUGGESTION] Mirror DRM/KMS camera preview, across multiple monitors HOT 6
- disable logging when taking picture HOT 5
- [HOW-TO] Improve streaming quality of Raspberry Pi Camera v3 during drone flight HOT 4
- [HOW-TO] Improve video quality of Raspberry Pi Camera v3 for detecting aruco markers in very high or low light conditions HOT 1
- [BUG] capture.array not working with imx296 and rpi-update > 6.6.31 on compute module 4 Bullseye 64bit HOT 1
- [BUG] libav_h264 encoder uses pyav function wrong HOT 6
- [OTHER] IndexError HOT 2
- [BUG] Continuous AutoFocus fails to converge HOT 11
- [HOW-TO] Camera Flash HOT 3
- [HOW-TO] Continuously stream lores, Occasionally record main HOT 2
- [BUG] No exit from function capture_request() HOT 8
- [BUG] Numpy 2.x not supported HOT 1
- [OTHER] Blocks of code overflowing in PDF documentation HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from picamera2.