In the future, there will be more and more cameras around us, especially after AR glasses become popular. Using these huge amount of visual information to build a virtual parallel world is a wonderful dream. The goal of this project is to provide vision-based positioning capabilities for this parallel world.
This project is also called chamo_vps for short.
The specific goal in the near future is to establish a location system based on the map which is formed by feature points. And all data used to create map are from mobile phone level sensors (monocular camera, imu, gps). And provide a positioning service that give position with gps coordinate system. The accuracy is expected around 20 cm.
Main Content
Video data collection and LocalMap creation on the mobile phone.
Image data collection and feature point extraction. These means the system supports both sequence frame and single picture update.
After uploading the LocalMap or image feature points to the server, the server merges the data.
The merging process includes the culling of map to maintain trackable map size.
New data that is not matched by the map will be stored, waiting for matching with future data.
Status
Map visualization in APP.
Merging new sensor data is OK, but not stable. After the clean function complete, it will be more stable.
Recording sensor data with custom exposure adjustment fucntion (Fix expousre time and adjust ISO).
Align layout with localization map.
Localization with only global match.
Average GPS to get more accurate global position.
Copy out the recorded bag with itune.
There is memory leak problem, some memory could not be released.
Backlog
Sprint
Target
September 1st
Use covisibility to choose update candidators
September 8th
Clean all distached or bad data in the end
September 15th
Self-calibration on APP
September 22th
Tracking with odomtry and global match
Future Function
Resume function for interrupt in processing a bag.
Reset the change caused by the lasted bag update.
Make global match realtime on IOS.
Show keypoints on frame image during SLAM and localization.
BA with IMU function
Automatical divide the bag.
Exposure duration max limitation.
Use SIM3 to transform the whole map before se3 pose optimization.
Further reduce memory leak.
Show percentage of mapping process.
Technical Details
Compilation framework: catkin
Serialization method: custom format based on fstream read and write
Debugging visualization tool: rviz
Client end: ios
Math library: eigen
Algorithm Flow
The green box represents the input data, which can be a sequence of pictures arranged in time, or it can be a single picture. And an approximate location obtained by global positioning is required for all data, which is used to put the data into the corresponding map block.
The purple box represents the processing flow on the clinet end. The final piece of data is split into multiple LocalMaps according to the covisibilities.
The blue box represents the processing flow on the server end.
The red box represents the data stored on the server after all processing.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent