hovren / crisp Goto Github PK

View Code? Open in Web Editor NEW

203.0 17.0 74.0 110 KB

Camera-to-IMU calibration and synchronization toolbox

License: GNU General Public License v3.0

Python 99.65% Batchfile 0.18% Shell 0.17%

crisp's Introduction

Camera-to-IMU calibration toolbox

This toolbox provides a python library to perform joint calibration of a rolling shutter camera-gyroscope system.

Given gyroscope and video data, this library can find the following parameters

True gyroscope rate
Time offset
Rotation between camera and gyroscope coordinate frames
Gyroscope measurement bias

If you use the package for your work, please cite the following paper

Ovrén, H and Forssén, P.-E. "Gyroscope-based video stabilisation with auto-calibration." In 2015 IEEE International Conference on Robotics and Automation (ICRA) (pp. 2090–2097). Seattle, WA

Can I use these methods for my application?

The calibration methods in this package assumes the following

Your camera is calibrated, including known readout time
The camera frame rate is constant, and known
The gyroscope frame rate is constant, and approximately known (within a few Hz, or percent)

If the video and gyroscope data are not uniformly sampled, but you have access to somewhat reliable timestamps, then you can still use the method if you resample the data to be uniform. By "reliable" we mean timestamps without drift, and no (or negligble) jitter.

Changes from 1.0

The 2.0 version of crisp features a new fully automatic calibrator. This means that there is no compelling reason to use the semi-manual methods in the previous version of crisp. Therefore the old example scripts have been removed, and the old functions are not imported into the module namespace. No old functions have been removed, so if you want to use them they are still available in submodules.

Installation

To use the package you need the following Python packages:

NumPy
SciPy
OpenCV
matplotlib

The easiest way is to install from PyPI:

$ pip install crisp

If you want to build the package from source, you also need the Cython package. To build and install the crisp module just run the following commands:

$ python setup.py build
$ python setup.py install

For a user-only installation add --user to the install command.

Usage

The gyroscope and video data are first loaded into a stream object (GyroStream, and a subclass of VideoStream respectively). To be able to understand how points are mapped from the real world to the image, the video stream also need a CameraModel (-subclass) instance.

import crisp

gyro = crisp.GyroStream.from_data(some_data_array)
camera_model = crisp.AtanCameraModel(...) # One specific choice of camera model
video = crisp.VideoStream.from_file(camera_model, video_file_path)

We then tie the streams together using a AutoCalibrator instance. Since the calibration proces need to have estimates of the time offset and relative rotation, these are first estimated using the initialize() member. This initialization only requires that you give an approximate gyroscope sample rate (in Hz).

calibrator = crisp.AutoCalibrator(video, gyro)
calibrator.initialize(guessed_gyro_rate)
result = calibrator.calibrate() # Dict of calibrated parameters

Initialization and calibration errors can be caught by handling InitializationError and CalibrationError.

Example scripts

We bundle one example script gopro_dataset_example.py which shows how to use the library with the data in our dataset (http://www.cvl.isy.liu.se/research/datasets/gopro-gyro-dataset/). This is the same dataset that was used to produce the above mentioned ICRA 2015 paper.

Feedback

For any questions regarding the method and paper, please send an e-mail to [email protected].
For issues about the code, you are welcome to either use the tools (issue reporting, etc.) provided by GitHub, or send an e-mail.

License

All code in this repository is licensed under the GPL version 3.

crisp's People

Contributors

Stargazers

Watchers

crisp's Issues

Calibration fails: "Number of calls to function has reached maxfev = 800."

Trying to calibrate camera - gyroscope pair. Camera runs at 30 fps, Gyro at 100 Hz.

The example file 'gopro_dataset_example.py', when modified to suit our dataset, runs for 2 - 3 minutes and fails with the following message:

runfile('/crisp/gopro_dataset_example.py', args='/Set3/set3.mov', wdir='/crisp')
Creating video stream from /Set3/set3.mov
Creating gyro stream from /Set3/set3_gyro.csv
Post processing L3G4200D gyroscope data to remove frequency spike noise
Creating calibrator
Estimating time offset and camera to gyroscope rotation. Guessing gyro rate = 100.00
Running calibration. This can take a few minutes.
Calibration failed. Reason "Number of calls to function has reached maxfev = 800."
/crisp/gopro_dataset_example.py:101: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  print('Calibration failed. Reason "{}"'.format(e.message))
To exit: use 'exit', 'quit', or Ctrl-D.
An exception has occurred, use %tb to see the full traceback.

SystemExit: -2

Traceback:

Traceback (most recent call last):

  File "<ipython-input-12-64f97a132da6>", line 1, in <module>
    runfile('/crisp/gopro_dataset_example.py', args='/Set3/set3.mov', wdir='/crisp')

  File "/usr/local/lib/python2.7/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 714, in runfile
    execfile(filename, namespace)

  File "/usr/local/lib/python2.7/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 81, in execfile
    builtins.execfile(filename, *where)

  File "/crisp/gopro_dataset_example.py", line 102, in <module>
    sys.exit(-2)

SystemExit: -2

In the example python file, there are quite a few parameters which are not very obvious:

Care to explain these ones:

CAMERA_DIST_CENTER
CAMERA_DIST_PARAM
CAMERA_READOUT

Also, in the *_reference.csv, please explain these column headers:

Fg,
offset
rot_x
rot_y
rot_z
gbias_x
gbias_y
gbias_z

What happens if I do not already know the gyro biases? Do we need to provide an estimate?

Fast Integrate Dependency

I was trying to install this, but it errors on build that it is missing a crisp/fastintegrate/fastintegrate.c file. I see that in the gitignore there as .c file is excluded. Where am I suppose to download this file so I can compile the project?

I am on ubuntu 14.04 and here is the error:

running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/crisp
copying crisp/cli.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/stream.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/rotations.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/calibration.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/znccpyr.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/videoslice.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/l3g4200d.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/remove_slp.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/tracking.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/__init__.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/timesync.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/imu.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/ransac.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/camera.py -> build/lib.linux-x86_64-2.7/crisp
copying crisp/pose.py -> build/lib.linux-x86_64-2.7/crisp
running build_ext
building 'crisp.fastintegrate' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/crisp
creating build/temp.linux-x86_64-2.7/crisp/fastintegrate
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c crisp/fastintegrate/fastintegrate.c -o build/temp.linux-x86_64-2.7/crisp/fastintegrate/fastintegrate.o
x86_64-linux-gnu-gcc: error: crisp/fastintegrate/fastintegrate.c: No such file or directory
x86_64-linux-gnu-gcc: fatal error: no input files
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 4

Edit: I am using the master branch.

Video Stabilization problem

Hi, I used the result of calibration toolbox to do video stabilization for the video in the paper. I found that according to the time offset in the calibration result, the stabilized video was very bad.
After scaling and offsetting the gyroscope data(as follow), I can get satisfactory results.

coeffies = [1.60190524e-03, -2.79751510e+00] # rotation.mp4
gyro_times = (gyro_times + coeffies[1]) / (1 - coeffies[0])

My question is, the video stabilization in the paper mentioned in the Readme directly use the time offset( 2.8333333333333335 as follow) of the calibration toolbox?

rotation.mp4 calibration result by the calibration toolbox:
gyro_rate = 853.660712254363
time_offset = 2.8333333333333335
gbias_x = -0.01001219994597313
gbias_y = -0.008216488048614853
gbias_z = 0.010610096301797737
rot_x = 0.022640457925138826
rot_y = -0.0068401509599219324
rot_z = -1.5426536394739456

Looking forward to your reply.

AttributeError: 'numpy.ndarray' object has no attribute 'sqrt'

Hi, when I run gopro_dataset_example.py with sample video in gopro-gyro-dataset， I got following errors in camera.py in line:

rn = np.sqrt((X[0,:] - wx)**2 + (X[1,:] - wy)**2)

AttributeError: 'numpy.ndarray' object has no attribute 'sqrt'

in function apply.

All three video: rotation.MP4, walk.MP4, rccar.MP4 will fall into this error after some successfully computation for previous calling.

I dumped the X data and found that X will be like below when reporting this error.
print(X) -->
[[array([-0.44746115])]
[array([0.02795225])]
[array([1.])]]

Could you help me found out why?
Thanks a lot!

Please add Cython to the list of dependencies.

SVD did not converge error

I'm running crisp with my own camera and gyro data, it raises an error:

/usr/lib/python2.7/dist-packages/pkg_resources/init.py:1224: UserWarning: /home/lili/.python-eggs is writable by group/others and vulnerable to attack when used with get_resource_filename. Consider a more secure location (set with .set_extraction_path or the PYTHON_EGG_CACHE environment variable).
warnings.warn(msg, UserWarning)
/usr/local/lib/python2.7/dist-packages/crisp-2.2-py2.7-linux-x86_64.egg/crisp/camera.py:181: RuntimeWarning: invalid value encountered in true_divide
Traceback (most recent call last):
File "gopro_dataset_example.py", line 75, in
calibrator.initialize(gyro_rate=GYRO_RATE_GUESS)
File "build/bdist.linux-x86_64/egg/crisp/calibration.py", line 127, in initialize
File "build/bdist.linux-x86_64/egg/crisp/calibration.py", line 256, in find_initial_rotation
File "build/bdist.linux-x86_64/egg/crisp/videoslice.py", line 52, in estimate_rotation
File "build/bdist.linux-x86_64/egg/crisp/rotations.py", line 306, in estimate_rotation_procrustes_ransac
File "build/bdist.linux-x86_64/egg/crisp/ransac.py", line 24, in RANSAC
File "build/bdist.linux-x86_64/egg/crisp/rotations.py", line 289, in
File "build/bdist.linux-x86_64/egg/crisp/rotations.py", line 72, in procrustes
File "/home/lili/.local/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 1359, in svd
u, s, vt = gufunc(a, signature=signature, extobj=extobj)
File "/home/lili/.local/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 99, in _raise_linalgerror_svd_nonconvergence
raise LinAlgError("SVD did not converge")
numpy.linalg.linalg.LinAlgError: SVD did not converge

How can I solve this?

A bug in tracking.py

At line 208, I think "final_ok = np.intersect1d(ok_track, retracked_ok)" is wrong ,it should be "final_ok=ok_track[retracked_ok]". Because ok_track is forward_track and backward_track 's index, but retracked_ok is forward_first and backward_last 's index . They are not the same index. So they can't do the "intersect1d" operate. "retracked_ok" is "ok_track" 's index.

understanding

I have several doubts, the first is how I calculate the matrix k,
the vector wc and omega,
readout is not 1/fps ¿why?
and finally where the stabilized video is exported (I can run the program perfectly but I do not see results)
I am using a gopro hero5.

Translation between camera and IMU

Hi, @hovren, I have a few questions:
Can the crisp toolbox obtain the translation between camera and IMU?
How can I obtain the readout time of a camera?
Thank you in advance!

AssertionError in calibration.py

I'm using the OpenCV camera model. After fixing several code issues and running crisp, I get the following error:

Creating video stream from /home/sergiu/Projects/test_sequences/building1.mp4
Creating gyro stream from /home/sergiu/Projects/test_sequences/building1_gyro.csv
Creating calibrator
Estimating time offset and camera to gyroscope rotation. Guessing gyro rate = 200.00
Traceback (most recent call last):
  File "parrot_ardrone_dataset.py", line 70, in <module>
    calibrator.initialize(gyro_rate=GYRO_RATE_GUESS)
  File "/home/sergiu/Projects/crisp/crisp/calibration.py", line 125, in initialize
    R = self.find_initial_rotation()
  File "/home/sergiu/Projects/crisp/crisp/calibration.py", line 290, in find_initial_rotation
    assert data.shape == (6, len(gyro_axes))
AssertionError

gyro_axes remains empty. I assume that estimating the initial rotation fails resulting in the error. Is this correct?

Package with pip

It would be useful to be able to install using a simple pip install crisp.

Video Stabilization Method

Hi, @hovren
Thank you for your excellent work. After calibration, I need to stabilize the video. But I'm not sure which method to use. I want to stabilize the video on a UAV in real time. Do you have some recommendations?
Thank you in advance.

stability of parameter estimation

Hi, I used the calibration toolbox to calibrate the data given by the paper, and the result was different every time.
In the experimental section of the paper, figure 4 shows the error relative to the set of reference parameters. The frequency error is within +/ -0.2, and the time offset error is between -0.008 and 0004. The results obtained within this range can achieve a nicely stabilised video, but the results of my calibration each time exceed the above range, resulting in the use of calibrated parameters can not generate a good stabilised video.

Here are some of the results I got(rotation.mp4):

Frequency(Hz) -1.245274 0.7984233 -1.291282
Offset(s) 0.004855215 0.004855141 0.004858229

Error: 'flow_mag' referenced before assignment

Thanks for making this library public! I was testing it out with the following sample code:

import crisp

gyro_ms = 5.05050505048712

f = 1000.06549452
im_h, im_w = 1080, 1920
cx = im_w*0.5
cy = im_h*0.5

cam_mat = [[f, 0, cx],
           [0, f, cy],
           [0, 0, 1 ]]
disto = [-0.20100871814959385, 0.048642904244843899, 0, 0, -0.0063799856646590239]

gyro = crisp.GyroStream.from_csv("gyro.csv")
camera_model = crisp.OpenCVCameraModel((im_h, im_w), 59.94, 0, cam_mat, disto)
video = crisp.VideoStream.from_file(camera_model, "vid.mp4")

calibrator = crisp.AutoCalibrator(video, gyro)
calibrator.initialize(gyro_ms)
result = calibrator.calibrate() # Dict of calibrated parameters

print(result)

But I got the following error.

Traceback (most recent call last):
  File "calib.py", line 20, in <module>
    calibrator.initialize(gyro_ms)
  File "/home/matt/.local/lib/python3.8/site-packages/crisp/calibration.py", line 124, in initialize
    time_offset = self.find_initial_offset()
  File "/home/matt/.local/lib/python3.8/site-packages/crisp/calibration.py", line 226, in find_initial_offset
    time_offset = timesync.sync_camera_gyro(flow, frame_times, self.gyro.data.T, gyro_times, levels=pyramids)
  File "/home/matt/.local/lib/python3.8/site-packages/crisp/timesync.py", line 107, in sync_camera_gyro
    ishift = znccpyr.find_shift_pyr(flow_mag, gyro_mag, levels)
UnboundLocalError: local variable 'flow_mag' referenced before assignment

I am using python3.8 and installed crisp with python3 -m pip install crisp. I did have to change one function in as matplotlib doesn't have normpdf anymore. I changed it to from scipy.stats import norm instead.

Any idea how to fix this error?

zncc: Division by zero

Hi, I have some troubles using crisp with my own samples. I took three sets of qvga@30fps video with 30Hz gyroscope and one set of 720p@60fps and 100Hz gyroscope and every one of those is giving me a Division by Zero error in the calibration.initialize(...) . I've tried to tune my setting values in your gopro_dataset_example.py, but the error is still there...

➜  examples python gopro_dataset_example.py dataset/samples/Set1/test.avi
Creating video stream from dataset/samples/Set1/test.avi
Creating gyro stream from dataset/samples/Set1/test_gyro.csv
Post processing gyroscope data to remove frequency spike noise
Creating calibrator
Estimating time offset and camera to gyroscope rotation. Guessing gyro rate = 30.00
/usr/lib/python2.7/dist-packages/numpy/core/_methods.py:55: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)
/usr/lib/python2.7/dist-packages/numpy/core/_methods.py:67: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
Traceback (most recent call last):
  File "gopro_dataset_example.py", line 96, in <module>
    calibrator.initialize(gyro_rate=GYRO_RATE_GUESS)
  File "build/bdist.linux-x86_64/egg/crisp/calibration.py", line 128, in initialize
  File "build/bdist.linux-x86_64/egg/crisp/calibration.py", line 230, in find_initial_offset
  File "build/bdist.linux-x86_64/egg/crisp/timesync.py", line 107, in sync_camera_gyro
  File "build/bdist.linux-x86_64/egg/crisp/znccpyr.py", line 273, in find_shift_pyr
  File "build/bdist.linux-x86_64/egg/crisp/znccpyr.py", line 175, in zncc
ZeroDivisionError: division by zero

The debug output is saying :

DEBUG:crisp:Number of slices: 84
DEBUG:crisp:Generating optical flow magnitude. This can take minutes depending on video length
DEBUG:root:pyramid size = 7
DEBUG:root:size of first element 2243 
DEBUG:root:size of last element 35

Do you have any idea of what I can be doing wrong?