Autonomous calibration of onboard robot cameras using fiducial markers
Project number: 18-58-45017
Full project name: Autonomous calibration of onboard cameras of a robotic system using fiducial markers located on the robot body.
Principal investigator: professor Evgeni Magid
Robustness of robotic systems (RS) is directly related to reliable operation of its onboard sensors, which require periodic calibration as a part of standard maintenance procedures. Calibration of a measuring device is a process of determining a relationship between device's readings and input measurement values. A standard set of RS onboard sensors can vary significantly depending on specific types of RS, but in most cases, it includes at least one optical sensor. Optical sensors by the type of device are divided into monoblock (for example, cameras) and two-unit (consisting of a radiating source and a receiver).
Calibration of a monoblock optical sensor of the "camera" type refers to the process of determining and adjusting the external and internal parameters of the camera. As a result, the reference value of the optical signals at the input will be matched, taking into account the specified accuracy, with the readings of the output value. Calibration of onboard robot cameras is a mandatory procedure for RS that allows using computer vision algorithms, successfully carry out sensor fusion and subsequent data processing, which are the key elements for solving fundamental problems in autonomous robot control.
Currently, calibration procedure of RS cameras is accomplished by service engineers in a laboratory using the classical “checkerboard” method. However, as a result of RS operation, especially when operating in adverse conditions, camera parameters tend to change over time, e.g., under mechanical forces. Such changes make it necessary to re-diagnose both RS and cameras calibration technical conditions, which is difficult to perform in field conditions. In addition to the classical “checkerboard” calibration method, alternative approaches that are based on fiducial markers (FM), e.g., such as AprilTag, have recently become popular. However, today, research work on autonomous camera calibration of all robot types using alternative methods is in its infancy stage both in Russia and abroad.
Our Project is aimed to develop alternative FM-based camera calibration methods for autonomous RS calibration in field conditions. Markers are placed on RS body surface, whereas the calibration procedure will be automated. Therefore, both human factor negative effects in case of the standard non-automated calibration procedure and a need to use external calibration equipment will be eliminated.
The key tasks that were solved within the framework of the project in order to improve the robustness of the RS are the developing of a system of mathematical calculations and experimental techniques for the analysis and multiparameter evaluation of the marker optimality for calibrating the cameras of the mobile RS, and the developing of a method and system for calculating the optimal location of the marker on the surface of the RS for the further possibility of performing autonomous calibration. Thus, RTS used in adverse conditions will gain the ability to independently adjust the parameters of on-board optical sensors, which will increase the accuracy of image analysis and recognition, which in turn is critical not only for the accuracy of manipulations and movements, but also for the reliability and fault tolerance of RS as a whole.
During the first stage (2018) of the Project, the following work was performed and the following results were accomplished:
During the second stage (2019) of the Project, the following work was performed and the following results were accomplished:
Mobile robot Sevosila Engineer
At the first and second stages of the Project our team participated in scientific events to popularize the research results:
In order to highlight and popularize the results of the project carried out under Contract No. 18-58-45017 / 18 on the topic: "Autonomous calibration of onboard cameras of a robotic system using fiducial markers located on the robot body", the research team took part in a number of events, including papers presentations at international conferences.
The first stage of the Project:
1. Professor Evgeni Magid presented the research paper on the topic: “Pilot virtual experiments in comparison of coordinate systems labels Aruco and AprilTag resistance to rotation” (Пилотные виртуальные эксперименты по сравнению систем координатных меток Aruco и AprilTag на устойчивость к вращению). Authors: A. Zakiev, K. Shabalina, T. Tsoy, E. Magid on the Vth Pan-Russian Research and Practice Workshop “Unmanned vehicles with elements of artificial intelligence” was held in Saint Petersburg. In 2019 the workshop was organized by Russian association of artificial intelligence (RAAI) in collaboration with St. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences (SPIIRAS).
Link to news: https://kpfu.ru/itis/science/kafedra-intellektualnoi-robototehniki/lirs/novosti-laboratorii/uchastie-lirs-v-v-vserossijskom-nauchno-367280.html
The website of the conference: http://2019.ai-uv.ru/
2. From 17 to 20 April 2019 the XIV International Conference on Electromechanics and Robotics "Zavalishin's Readings" was held in Kursk, Russia. The first conference was organized in 2006 by the Institute of Innovative Technologies in Electromechanics and Robotics of the St. Petersburg State University of Aerospace Instrumentation (SUAI) and was dedicated to the 65th anniversary of the Institute. This time the conference is held at the Southwest State University (SWSU) in collaboration with the XIII International Scientific and Technical Conference "VIBRATION-2019. The vibration technology, mechatronics and controlled machines".
Professor Evgeni Magid, a Head of the Laboratory of Intelligent Robotic Systems presented the paper: “Virtual Experiments on ArUco and ArTag Systems Comparison for Fiducial Marker Rotation Resistance”. Authors: Aufar Zakiev, Ksenia Shabalina, Tatyana Tsoy, Evgeni Magid.
In the news: https://kpfu.ru/itis/science/kafedra-intellektualnoi-robototehniki/lirs/novosti-laboratorii/predstavlenie-dokladov-lirs-na-xiv-mezhdunarodnoj_366043.html
The conference website: http://confs.guap.ru/zav-read
3. From 10 to 13 January 2019 in Beppu (Oita Prefecture) was held an annual International Conference on Artificial Life and Robotics, ICAROB 2019.
Topic of the paper presented by professor Magid: “Pilot Virtual Experiments on ArUco and AprilTag Systems Comparison for Fiducial Marker Rotation Resistance”. Authors: Aufar Zakiev, Ksenia Shabalina, Kuo-Lan Su, Kuo-Hsien Hsia, Evgeni Magid.
In the news: https://kpfu.ru/itis/science/kafedra-intellektualnoi-robototehniki/lirs/novosti-laboratorii/lirs-predstavila-rezultaty-nauchnoj-raboty-na.html
The conference website: http://alife-robotics.co.jp/
4. The 11th International Conference on Developments in e-Systems Engineering (DeSE2018) of the Institute of Electrical and Electronics Engineers (IEEE) is held from the 2nd to the 5th September in Cambridge (Great Britain). IEEE is the world leader in the development of standards for electrical engineering and hardware for computing systems and networks.
Along with other reports Professor Evgeni Magid, Head of the Intelligent Robotics Department of Higher Institute of Information Technology and Intelligent Systems (ITIS)presented in a special session “Advanced Robotics” presented a paper on relevant topic: “Real-time Video Server Implementation for a Mobile Robot”. Authors: Evgeni Magid, Roman Lavrenov, Tatyana Tsoy, Mikhail Svinin, Ramil Safin.
In the news: https://media.kpfu.ru/news/uchenye-kfu-vystupili-s-dokladami-na-mezhdunarodnoy-ezhegodnoy-konferencii-dese-v-kembridzhe
The conference website: https://dese.org.uk/city-of-cambridge/
5. Professor Evgeni Magid presented the joint work of LIRS at the 20th International Conference on Speech and Computer (SPECOM 2018) and the 3rd International Conference on Interactive Collaborative Robotics (ICR 2018) that are held September 18-22 in Leipzig, Germany. Professor presentd a paper on the topic: “Comparing Fiducial Markers Performance for a Task of a Humanoid Robot Self-Calibration of Manipulators: a Pilot Experimental Study”). Authors: Ksenia Shabalina, Artur Sagitov, Mikhail Svinin, Evgeni Magid.
In the news: https://kpfu.ru/itis/science/kafedra-intellektualnoi-robototehniki/lirs/novosti-laboratorii/komanda-lirs-predstavila-rezultaty-svoih-342379.html
The conference website: http://specom.nw.ru/icr2019/
6. Professor Evgeni Magid presented a scientific paper “Mobile robot's defective stereo camera calibration using standard calibration patterns” (“Калибровка дефектной стереопары мобильного робота с использованием типовых шаблонов”) at the International Scientific and Practical Conference Progress Vehicles and Systems 2018. Authors: Ramil Safin, Roman Lavrenov, Evgeni Magid.
The second stage of the Project:
1. From 7 to 10 October 2019, the XII International Conference IEEE Developments in e-Systems Engineering (DeSE) was held at the Higher Institute of Information technologies and intelligent systems (ITIS) of KFU, organized jointly by the Liverpool John Moores University, IEEE, the Higher Institute of ITIS and the Naberezhnye Chelny Institute of KFU.
Apart from the guests from foreign and domestic higher education institutions and companies, the conference was attended by employees and students of LIRS. LIRS employee Aufar Zakiev presented the paper on the following topic: “Validation of Fiducial Marker Systems Performance with Rescue Robot Servosila Engineer Onboard Camera in Laboratory Environment”. Authors: Tatyana Tsoy, Aufar Zakiev. Ksenia Shabalina, Ramil Safin, Evgeni Magid, Subir Kumar Saha.
LIRS employee Roman Lavrenov presented the paper on the topic: “Face drawing by KUKA 6 axis robot manipulator”. Authors: Maxim Pichkalev, Roman Lavrenov, Ramil Safin, Kuo-Hsien Hsia.
In the news: https://kpfu.ru/itis/science/kafedra-intellektualnoi-robototehniki/lirs/novosti-laboratorii/v-vysshej-shkole-informacionnyh-tehnologij-i_378780.html
In the news: https://media.kpfu.ru/news/v-kfu-prokhodit-12-ya-mezhdunarodnaya-konferenciya-ieee-developments-e-systems-engineering
The conference website: https://dese.org.uk/developments-in-esystems-engineering-2019/
2. From 10 to 13 January 2019 in Beppu (Oita Prefecture) was held an annual International Conference on Artificial Life and Robotics (ICAROB 2019). Professor Evgeni Magid presented the paper on the topic: “Transfer of learned exploration strategies for a mobile robot from a simulated world to real environments”. Authors: Artur Sagitov, Tetsuto Takano, Shohei Muto, Evgeni Magid.
The website of the conference: http://www.psct.ru/
3. In 2019 LIRS presented a report at the XIV All-Russian Scientific and Practical Conference “Perspective Systems and Control Problems” on the topic: “Development of a system for optimal transmission of video data from the cameras of the Servosila Engineer mobile robot”.
The conference website: http://www.psct.ru/
4. From April 15 to April 18, 2020, the XV International Conference on Electromechanics and Robotics “Zavalishin's Readings 2020” was held at the St. Petersburg State University of Aerospace Instrumentation together with XIII International Scientific and Technical Conference "Vibration-2019. The vibration technology, mechatronics and controlled machines" in Ufa State Oil Technical University. This year the participation of the conference was virtual on the Leader-ID platform according to Order No. 398 of the Ministry of Science and Higher Education of the Russian Federation of March 14, 2020 in the current epidemiological situation in the world.
Employee and graduate student Ramil Safin presented the work on the topic: “Evaluation of visual SLAM methods in USAR applications using ROS/Gazebo simulation”, Authors: Ramil Safin, Roman Lavrenov, Edgar. A. Martinez-Garcia.
In the news: https://kpfu.ru/itis/science/kafedra-intellektualnoi-robototehniki/lirs/novosti-laboratorii/na-xv-mezhdunarodnoj-konferencii-po-388390.html
The conference website: http://confs.guap.ru/zav-read
5. From July 19 to July 24 due to the epidemiological situation in the world were held online three flagship conferences of the IEEE Computational Intelligence Society under one roof: The 2020 International Joint Conference on Neural Networks (IJCNN 2020), the 2020 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE 2020) and the 2020 IEEE Congress on Evolutionary Computation (IEEE CEC 2020).
LIRS employee Aufar Zakiev presented a paper in one of the sections of the conference IJCNN 2020 on the topic: “Virtual Experiments on ArUco and AprilTag Systems Comparison for Fiducial Marker Rotation Resistance under Noisy Sensory Data”. Authors: Aufar Zakiev, Tatyana Tsoy, Ksenia Shabalina, Evgeni Magid, Subir Kumar Sakha.
In the news: https://kpfu.ru/itis/science/kafedra-intellektualnoi-robototehniki/lirs/novosti-laboratorii/sotrudnik-lirs-vystupil-s-dokladom-na-vsemirnom.html
The conference website: https://wcci2020.org/
6. On February 3-8 at the National Institute of Technology Calicut (NITC, Kozhikode, India) took place the FDP on Robotics, Artificial Intelligence and Control (RAIAC-2020). It was held with support from the Indian Robotics Society, the Indo-Korean Joint Network for Robotics, and the Indian Institute of Technology (IITD), Delhi. On the first day of the symposium, Professor, a head of the Laboratory of Intelligent Robotic Systems Evgeni Magid, gave a talk: “Interaction between a Robot and a Human”.
Professor Magid also gave a speech on the topic of which was the Search and Rescue Robotics in Earthquakes, Flood, and Land Slide Disasters. He also presented the results of the international project “Informational system for management of flood and land slide disaster areas using a distributed heterogeneous robotic team”.
Presentation type: keynote speech (lecture).
In the news: https://kpfu.ru/itis/science/kafedra-intellektualnoi-robototehniki/lirs/novosti-laboratori
Publications on Project:
- Zakiev A., Shabalina K., Tsoy T., Magid E. Pilot Virtual Experiments on ArUco and ArTag Systems Comparison for Fiducial Marker Rotation Resistance // 14th International Conference on Electromechanics and Robotics "Zavalishin's Readings" (Kursk, Russia; 17-20 April 2019) - р. 455-464.
- Закиев А.А., Шабалина К.С., Цой Т.Г., Магид Е.А. Пилотные виртуальные эксперименты по сравнению систем координатных меток ArUco и AprilTag на устойчивость к вращению // V Всероссийский научно-практический семинар «Беспилотные транспортные средства с элементами искусственного интеллекта» (БТС-ИИ-2019), (г. Санкт-Петербург, Россия; 22-24 мая 2019) - c. 211-220.
- Zakiev A., Shabalina K., Magid E. Pilot Virtual Experiments on ArUco and AprilTag Systems Comparison for Fiducial Marker Rotation Resistance // International Conference on Artificial Life and Robotics (ICAROB 2019) (Oita, Japan; 10-12 January 2019) - p. 132-135.
- Сафин Р.Н., Лавренов Р.О., Магид Е.А. Калибровка дефектной стереопары мобильного робота с использованием типовых шаблонов // Международная научно-практическая конференция "Прогресс транспортных средств и систем - 2018" (г. Волгоград, Россия; 8-12 октября 2018) - с. 166-167.
- Safin R. ROS-based Multiple Cameras Video Streaming for a Teleoperation Interface of a Crawler Robot / Lavrenov R., Martinez-Garcia E.A., Magid E. // Journal of Robotics, Networking and Artificial Life. - 2018. - Vol. 5 (3). - p. 184-189.
- Magid Е., Lavrenov R., Tsoy Т., Svinin M., Safin R. Real-time Video Server Implementation for a Mobile Robot // The 11th International Conference on the Developments in eSystems Engineering (Cambridge, England; 2-5 September 2018) - p. 180-185.
- Shabalina K. Comparing Fiducial Markers Performance for a Task of a Humanoid Robot Self-calibration of Manipulators: A Pilot Experimental Study / Sagitov A., Svinin M., Magid E. // Lecture Notes in Computer Science (incl. subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). - 2018. - Vol. 11097. - p. 249-258.
- Tsoy T., Zakiev A., Shabalina K., Safin R., Magid E., Saha S.K. Validation of Fiducial Marker Systems Performance with Rescue Robot Servosila Engineer Onboard Camera in Laboratory Environment // The 12th International Conference on Developments in eSystems Engineering (Kazan, Russia; 7-10 October 2019) - p. 495-499.
- Pichkalev M., Lavrenov R., Safin R., Hsia K.-H. Face drawing by KUKA 6 axis robot manipulator // The 12th International Conference on Developments in eSystems Engineering (Kazan, Russia; 7-10 October 2019) - p. 709-714.
- Sagitov A., Takano T., Muto S., Magid E. Transfer of learned exploration strategies of a mobile robot from a simulated to real environments // International Conference on Artificial Life and Robotics (ICAROB 2019) (Oita, Japan; 10-12 January 2019) - p. 120-123.
- Сафин Р.Н., Цой Т.Г., Лавренов Р.О., Магид Е.А. Разработка системы оптимальной передачи видеоданных с камер мобильного робота Сервосила Инженер // XIV Всероссийская научно-практическая конференция «Перспективные системы и задачи управления» (г. Нальчик, Россия; 01-05 апреля 2019), с. 304-313.
- Safin R., Lavrenov R., Martinez-Garcia E. A. Evaluation of Visual SLAM Methods in USAR Applications Using ROS/Gazebo Simulation // Proceedings of the 15th International Conference on Electromechanics and Robotics «Zavalishin's Readings» (ER(ZR)-2020) (Ufa, Russia; 15-18 April 2020) - p. 371-382.
- Zakiev A., Shabalina K., Magid E., Saha S.K. Virtual Experiments on ArUco and AprilTag Systems Comparison for Fiducial Marker Rotation Resistance under Noisy Sensory Data // Proceedings of the International Joint Conference on Neural Networks (IJCNN 2020) (Glasgow, Scotland; 19-24 July 2020) – p. 1-6.