An implementation of the Position-Based Visual Servo (PBVS) in Python 3.10 with OpenCV 4.6.0. The main reference paper is Visual servo control, Part I: Basic approaches by F. Chaumette and S. Hutchinson.

The purpose of visual servoing is to have control over the end-effector relative to the goal. For this, image features of the target are extracted, and preprocessed to eventually localize the target by its position.

Fig.1. Position-based visual servo.

The target used as goal is a fiducial marker known as ArUco marker, more concise information can be found in Automatic generation and detection of highly reliable fiducial markers under occlusion by S. Garrido-Jurado, R. Muñoz-Salinas, F. J. Madrid-Cuevas, and M. J. Marín-Jiménez.

The main reason for using this fiducial marker is because of the ease with which the pose of the camera can be estimated. Also, OpenCV offers a complete submodule dedicated exclusively for detecting ArUco markers.

There exists a convenient platform to easily generate online ArUco markers. The configuration that this project uses is:

• Dictionary: 4x4 (50, 100, 250, 1000).
• Marker ID: 0 and 1.
• Marker size, mm: 100.

However, I don't really use the Marker ID parameter so that it can be any of the allowed ID's (0, 1, 2, 3...). Another recommendation is to crop around the ArUco in such a way that some white space is left, this will help for the better detection of the ArUco. Additionally, I recommend sticking it on a hard surface which will prevent that the ArUco bends, and it also helps for a better dectection. A piece of cardboard will be enough. Leaving a kind of cardboard piece at the end in order to have a better grip of the ArUco helped a lot. Otherwise, it's difficult not to occlude the ArUco.

Essentially, the Graphical User Interface (GUI) works as a guide for whoever is running the program and wants to reach the final position with the help of a minimal but sufficient text indicator. This text let the user know what are the steps to follow to reach the goal position, and they are displayed sequentially.

At the moment of showing the information, different windows are displayed. The top-left window is the current position in $x, y$ and $z$ the bottom-left window is the current orientation in the Euler angles $\phi, \theta$ and $\psi$. The top-middle window is the desired position $x_d, y_d$ and $z_d$ and the bottom-middle window is the desired orientation $\phi_d, \theta_d$ and $\psi_d$. Finally, the top-right window is the error in the position $x_e, y_e$ and $z_e$ and the bottom-right window is the error in the orientation $\phi_e, \theta_e$ and $\psi_e$.

I have decided to add some charts as a reference using the well-known matplotlib.pyplot library, this to have a tracking of the position and orientation as well as their position and orientation error of the camera with respect to the ArUco marker.

Fig.2. Left: Position and orientation. Right: Position and orientation error.

The charts are displayed in real-time which is something to consider before using it since it may consume a lot of the processor to draw the charts and therefore make the program to run slow or even the computer. The recommendation is to run the charts only when it is really necessary.

usage: PBVS.py [-h] [-sc | --show_charts | --no-show_charts] [-sg | --show_gui | --no-show_gui]

Implements the Position-Based Visual Servo.

options:
  -h, --help            show this help message and exit
  -sc, --show_charts, --no-show_charts
                        Shows the charts of position and orientationin real time
  -sg, --show_gui, --no-show_gui
                        Shows the gui to reach the desired pose of the ArUco marker (default: True)

To save the current pose of the ArUco marker relative to the camera the key q can be pressed. This will save the pose into a .npy file in the /bin directory.

To show the charts in real-time

python3 PBVS.py --show_charts

To not show the GUI

python3 PBVS.py --no-show_gui

MIT License

Copyright (c) [2022] [Angelo Espinoza]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.