skimslozo / 3dv Goto Github PK

According to @skimslozo there is slight error between the generated pixel positions and the actual position in the video frame. This needs to be investigated.

Generate a smooth trajectory of the ball. Reject outliers. Maybe by fitting some sort of curve.

Right now the camera is static, but all other code but the DatasetGenerator already has provisions for moving cameras.

@teamathe Could you please post a link which sequence do we already have hand-labeled data for?

As mentioned in #11 , there seems to be a discrepancy between the renedered position of the ball on the screen, and what the projected coordinates predict it to be, based on the 3D ground-truth ball position and calibration matrices.

My best guess is that there is a factor added somewhere between geometric calculations of the camera (which is the info we use to calculate pixel coordinates) and the rendering of the image, however, I have failed to find it so far.
Even though I have also checked my methods for calculating the calibration matrices quite rigorously, there is, of course, a chance I have made a mistake in calculating those, but I could not find it.

Could you guys take a look at it at some point, if you think it's worth looking into?
@teamathe @Quexor @corkillj

Show ground truth from recorded data as dots in 3d visualization.

Based on a metric of your choice (e.g. RMSE), compare the 3D reconstruction accuracy with different set-ups of the simulation:

1. Number of cameras, N = 1, ..., some upper bound?
2. Noisy 2D detections, noise type, variance and number of datapoints it is applied to TBD
3. Noisy camera extrinsics/intrinsics - robustness of reconstruction with respect to calibration errors

We would like to have a video with the football video stream on one side, and the trajectory the ball is taking in our visualizer on the other side.

• Ask about robustness feasibility
• Send the proposal to them on Friday (19/03/2021) on Thursday
• Ask about GRF implementation - have they worked with them? Advice on e.g. camera view change
• Ask about data availability

do it @skimslozo

• GRF: their plan of approach, what data woudl they like to generate
• Output of their pipeline

optional, but would be cool

We need a method to import the data from GRF into our 3D trajectoty implementation pipeline.

Maybe various types of noise, maybe only certain cameras, maybe an offset for one camera or whatever.
Also maybe inject random complete non-sense outliers

• Create Project Proposal PDF until Friday 08:00
• Send to supervisor
• Implement feedback
• Submit before 23:59

First set of data shoud be generated using Google Football Research. Data shoud include:

1. Ground-truth 3D trajectory points of the ball:

• x, y, z [m] - position of the ball in the world frame at a T [timestamp]

2. 2D projected trajectories of the ball from 2 calibrated cameras (perhaps trajectories themselves not necessary, if we have the camera info then we should be able to project the 3D trajectory). Requirements:

• u, v position of the ball [pix] at a specific timestamp T from at least 2 camera views
• Ideally, T (approximately) the same for both cameras
• In case it is not explicitly possible, we estimate the 2D trajectories ourselves by projecting the 3D path on multiple 2D image panes. Can be done either by implementing in GRF side, or on our side.

3. Camera calibration info. This includes:

• Camera pose matrix [R|T] in the world coordinates
• Camera internal calibration matrik K

For now, the data type is not relevant/tbd as the task advances. Let's keep each other updated on progress within this issue

We need a way how to make a 3D visualization of our triangulated 3D points and the spline fitted/interpolated through them.

Implement reading a settings file descirbing camera parameters and a simulation scenrio, sucht that those are the params that the sim is ran with.

• Do a literature review of the state of the art in extracting a 3D trajectory from multiple 2D trajectories (e.g. bundle adjustment). Understand the algorithms and the Math. Aspects to figure out:
• Correspondence, Triangulation, Trajectory Estimation
• What is already implemented and accessible for us?
• Computational performance?
• Robustness against disagreement of the multiple streams.
• Robustness against occlusion
• Kalman Filtering
• Learning based elements or more or less pure math
• Select Performance Metrics

increase the clipping plane parameter value such that it does not get cut off somewhere in the field

what ever data we get from real word stuff will have wrong outliers, these need to be detected and removed.

The leasts_sqares method in the nonlinear refinement method might have some options...

Hello,

I'm trying to run the following command to test my installation on Ubuntu 22.04, with python-3.7 and gcc-9.4.0:
python datagen.py

However, it produces a segmentation fault when trying to import anything from gfootball library.
I followed the installation instructions in the main branch exactly. Also tried different python and gcc versions but these did not solve the problem.

There seems to be no problem if I compile the original repo from Google Research. The simulation launches and, expectedly, it gives an error where it tries to set the camera location which the original repo does not support.

Any idea why it could give segmentation fault?

Thanks.

It would be nice to have methods already defined for the triangulation, so the higher-level architecture with some dummy variables. So this would include:

• Reading corresponding 2D points from 2 different image stream dataframe (already clean data, provided by @teamathe implementation)
• Reading the camera intrinsic and extrinsic parameters
• Triangulation method (calculate a 3D point based on n 2D correspondences)
• Iteration and minimization (of a metric still TBD) to obtain the final 3D point

@Quexor could you update with the possible metrics (e.g. reprojection error, LSQ in case of bundle adjust) that we'll use to iterate on and to obtain the "final" 3D point? Feel free to add/edit what goes in there