Full project name: Information System of Emergency Situations Management in the areas of Floods and Landslides with Assistance of Distributed Heterogeneous Group of Robots.
The LIRS team, led by Professor Evgeni Magid, who has extensive international experience in universities and research institutes in the field of search and rescue robotics (Tsukuba University, Carnegie Mellon University, University of Bristol, etc.), for 3 years will be implementing a project on “Information System for Emergency Situations Management in Floods and Landslides with Assistance of Distributed Heterogeneous Group of Robots” in a collaboration with two teams - from Japan and under the management of Professor Fumitosi Matsuno, Kyoto University (Kyoto University), which is the 35th place in the QS World University Ranking and with a team from Thailand under the guidance of Professor Jackrit Sutkorn, Mahidol University, which is 380 place in the QS World University Ranking.
Figure 1 presents the general idea of the proposed project. Heterogeneous groups of robots will work as separate national teams (which in turn may be divided into distributed subgroups), supplying data to a centralized information system for emergency management. The robots will perform the tasks of building thematic maps of the disaster area from the surface of the water, under the water, from the air and from the ground, including observation of the destroyed buildings. Figure 2 shows the distribution of independent tasks between national teams and their interaction by common tasks within the project.
Fig. 1 Heterogeneous distributed groups of robots collect the information necessary to create separate thematic maps, which will then be combined into a multi-layered map of the disaster zone.
Fig. 2. The proposed information system and plan for joint research: the R symbol define tasks of the Russian team, J - the tasks of the Japanese team, T - tasks of the Thai team.
In Asia, including the Asian part of the Russian Federation, there is a high level of risk of natural disasters, among which floods are the most frequent and the most severe losses in terms of casualties and the economy. Therefore, the development of an information system (IS) for managing a disaster management process, which makes it possible to increase the effectiveness and speed of elimination of consequences, is today one of the most pressing and sought-after tasks facing the scientific community.
Three countries participating in the project - the Russian Federation, Thailand and Japan - at certain times of the year are at risk of floods and landslides caused by torrential rains and the rapid melting of large amounts of snow cover. Using the long-term experience of our three national teams in search and rescue robotics for disaster management, the proposed project will develop a robotic information collection IC for effective disaster management in a disaster zone. The main objective of the project is the development and testing of a prototype IP, which can then become an international standard for the elimination of both national and international natural disasters.
The IS will collect data using distributed heterogeneous groups of robotic complexes, including various types of unmanned aerial vehicles (UAV), unmanned ground robots (UGV), autonomous crewless ships (ACS) and autonomous underwater vehicle (AUV). Separate maps built by distributed groups of robots will be combined into a single multi-layer thematic map of the disaster area, which will help search and rescue teams to speed up the evacuation of survivors, assess the risk of further destruction of buildings and environmental pollution, which in turn will significantly increase the chances of rescuing survivors and increase the safety of rescuers during search operations.
The new control strategies, interfaces, protocols, robot models, algorithms and software developed by the project will be tested both with the help of simulation and in the framework of field experiments. To demonstrate the effectiveness of the IS developed in the framework of the project, its testing will be conducted in cooperation with the local authorities of each of the three countries. As a result of the project, an understanding of the mechanisms associated with technologically sound decision making for effective emergency management will be improved. The development and implementation of a new generation of robotic tools for disaster management, which are in great demand by international and national emergency agencies, will make a huge contribution to the scientific, technological and industrial development of society.
Scientific novelty of the research in the Project is as follows:
A unique prototype of an unparalleled robotic information system (IS) for managing the process of eliminating the consequences of natural disasters will be developed and tested within the project.
The international teams will develop a new platform and new strategies for managing the collaborative behavior of heterogeneous robots in the tasks of collecting information, monitoring and mapping large-scale disaster zones, including land, underwater and air areas prone to natural disasters, in particular floods and landslides caused by heavy rains. New control strategies, interfaces and protocols for interaction and data exchange between heterogeneous robots will be tested in simulation and verified in field experiments.
At a beginning level of the project we are planning to develop new algorithms for simultaneous localization and mapping (SLAM) for a heterogeneous group of robots in a natural disaster, new route planning algorithms for effective coverage of areas in an unstructured environment after a natural disaster, and new algorithms for autonomous robot return when communication is lost. Moreover, we expect that our joint research on robotic systems operating in landslide conditions will lead to the creation of a new class of mobile vehicles using new management principles. The development of macro simulators (which are in responsibility of the Japanese team) based on GIS and a graphical user interface can create conditions for a new direction in the field of virtual reality, dealing with emergency modeling. If the project is successful, new scientific and practical solutions will improve the understanding of the mechanisms involved in technologically sound solutions for effective emergency management on an international scale.
During the period of the first stage (2019-2020), the following results were obtained, aimed at the Project implementation:
1. A data exchange protocol was developed, which includes the motivation for the development of a data exchange protocol in the conditions of a search and rescue operation, the conditions of a search and rescue operation. This protocol was presented by groups of scientists from Japan and Thailand. The protocol is based on the analysis and systematization of the conditions in which it should be used, and the requirements that are made to it. The requirements for the protocol are motivated by the conditions in which it should be applicable: they include both technical requirements and considerations of applicability in practice by a large number of specialists from different countries. Much attention is given to avoiding mistakes made in similar projects, and ensuring ease of implementation on a large number of robots with different hardware and software.
This protocol is universal and is planned to be implemented in the Japanese and Thai research teams.
2. Mobile robotic systems (MRS) of the laboratory were implemented in the ROS/Gazebo environment:
3. Software for media and pollution generation was developed (Microsimulator №1). This software was tested according to the developed testing methods on MRS implemented in the simulation environment. The development was conducted in the C++ programming language in the Ubuntu 16 operating system, based on the Linux. Its functionality includes:
The microsimulator is a modular software that can be added to new functionality. Its code is documented and open for use.
This software is original and has no analogues.
4. The software was developed to simulate sensor errors and failures (Microsimulator №2). This software was integrated into the Microsimulator №1 and tested according to the developed testing methods on the MRS implemented in the simulation environment.
The development was conducted in the C++ programming language in the Ubuntu 16 operating system, based on the Linux . Its functionality includes:
This software is original and has no analogues.
5. Special equipment and components kit were selected and purchased. The Russian national team carried out pilot testing of the equipment in cases where it was necessary.
6.The results of the work for 2019 were summed up. Publications collected within the framework of the 1-year grant report, cooperation with foreign partners, as well as scientific activities aimed at highlighting and popularizing the interim results of the project were analyzed.
During the period of the first stage (2020-2021), the following results were obtained, aimed at the Project implementation:
1. The Russian project team developed and tested a multifunctional microsimulator that simulates an emergency disaster zone and allows testing various algorithms for autonomous and semi-autonomous operation of both individual mobile robots and heterogeneous groups of mobile robots. New functionality appeared during the gradual development of the microsimulator, it was assigned a new serial number. So, at the end of the first stage, we presented the basic Microsimulator №2, and by the end of the second stage, we completed the development of the final version of the microsimulator – Microsimulator №5.
2. A comparative analysis was carried out in the virtual environment of Gazebo within three groups of currently existing algorithms for SLAM:
All the considered methods were presented by researchers after 2015 and are currently being actively developed. Moreover, such a comparison of these methods using the qualitative and quantitative characteristics of the algorithms has not been conducted before. The Russian team developed and implemented a new algorithm for simultaneous localization and mapping of MultiCam-SLAM, based on the integration of data from three onboard cameras of the robot. The developed algorithm was tested in the Gazebo environment and during the comparative analysis showed a higher accuracy of operation relative to the original ORB-SLAM2 algorithm. All other tested algorithms and the new proprietary algorithm were implemented in the Microsimulator №3 in the form of additional tabs that allow connecting any of the listed algorithms to any of the robots added in the simulation (Fig. 1A). The developed architecture of the Microsimulator №3 with new functionality represents a new approach to simulation and has not been previously developed by other research teams. During the Microsimulator №3 testingaccording to pre-prepared methods and experimental programs, models of Hector-Quadrotor (UAV), Husky (UGV), Jackal (UGV) and Turtlebot3 (UGV) robots were used.
(Fig. 1A) The result of performing a collaborative SLAM algorithm by two mobile robots
3. The data exchange protocols developed at the first stage of the Project between heterogeneous robots within one team, and exchange protocols between different robot teams (in collaboration with foreign Project partners) were implemented in Microsimulator №3 as an additional tab. The protocol was tested in the Microsimulator №3 using ready-made robot models on real robots in the problem of interaction between the domestic crawler robot "Servosila Engineer" and the Tiago Base robot (PMB-2) of Spanish production (Fig. 1B) and on real robots in the task of exchanging data between different groups of robots formed from the PMB-2 robot and several homemade robotic devices on the Artik10 platform. Testing experimentally confirmed the capabilities of the protocol to ensure the cooperative operation of a heterogeneous group of robots in the task of path planning and in the task of SLAM.
(Fig. 1B) PMB-2 (left) and "Servosila Engineer" (right) robots in a test experiment to interaction protocol
4. An algorithm has been developed for detecting a communication break and autonomous return of the mobile robot in the event of a communication system failure, taking into account possible environmental changes. The algorithm was added to Microsimulator №4 and tested on robots of various designs in a variety of environments created in the Gazebo simulator. The algorithm was also tested on real mobile robots PMB-2 and "Servosila Engineer". The algorithm can be applied to most mobile robots with different types of control and will reduce the likelihood of losing expensive equipment. For testing on real robotic devices, additional sensors and equipment were integrated into them. At the same time, the software and hardware problems that arose were solved.
5. Close interaction was organized between the managers and staff of the national research teams, which, due to the epidemiological situation, took place in an online format in the form of permanent meetings and working meetings via Zoom software (Figure 1C), and joint participation in a number of conferences. On March 19, 2020 a quadripartite international agreement on scientific cooperation was signed between Ritsumeikan University (Japan), Kyoto University (Japan), Kazan Federal University and Mahidol University (Thailand), which will expand further cooperation, including outside the framework of the project. The organization of joint sections at the international conferences "Developments in eSystems Engineering 2020" and "Siberian Conference on Control and Communications 2021" helped to attract additional interest in the interim results of the project. The leaders of all three teams joined the Organization, Program and Technical Committees of international conferences (ICAROB-2020, ICAROB-2021, DeSE-2020, DARS-2021 and SWARM-2021, SIBCON-2021, etc.), which helped to strengthen their positions and international recognition in the scientific community. During the second stage, the Russian team published 17 scientific articles (15 in publications indexed in the Scopus database, 1 in the RSCI database), 10 of which were co-authored with foreign Project teams.
(Fig. 1C) National team leaders and staff during an online meeting via Zoom, March 12, 2021
An important scientific result of the Project second stage is the formulation of criteria for evaluating the quality of the route to optimize the algorithms of route planning by single robots and robots operating in a group. Based on the review of path planning algorithms for optimal map coverage by one robot and a group of robots, using two-dimensional (lidar) and three-dimensional (camera) data, a new method for goal planning for robots was developed that solves the problem of optimal map coverage. The final Microsimulator №5 with integrated algorithms (Fig. 1D) is a basis for interaction with scientific groups in Japan and Thailand, and also allowed the publication of a number of papers in journals and conference proceedings.
(Fig. 1D) The environment created in the Microsimulator №5 simulating the emergency zone
Publications prepared for the first stage of the Project (2019-2020):
- Magid E., Pashkin A., Simakov N., Abbyasov B., Suthakorn J., Svinin M. and Matsuno F. Artificial intelligence based framework for robotic search and rescue operations conducted jointly by international teams // 14th International Conference on Electromechanics and Robotics " zavalishin's="" readings"="" (kursk,="" russia;="" 17-20="" april="" 2019)="" -="" p.="" 15-26.
- Moskvin, I., Lavrenov, R. (2019). Modelling Tracks and Controller for Servosila Engineer Robot. Proceedings of 14th International Conference on Electromechanics and Robotics "Zavalishin's Readings", 154, p. 411-422.
- Magid E., Tsoy T., Matsuno F., Suthakorn J., Svinin M. (2019). Disaster Risk and Reduction Management with Search and Rescue Robotics Approach. The 2019 International Workshop of ICAROB in Taiwan (2019 ICAROB Taiwan).
- Simakov N., Lavrenov R., Zakiev A., Safin R., Martinez-Garcia E.A. Modeling USAR maps for the collection of information on the state of the environment // The 12th International Conference on the Developments in eSystems Engineering (Kazan, Russia; 7-10 October 2019).
- Pashkin A., Lavrenov R., Zakiev A., Svinin M. Pilot communication protocols for group of mobile robots in USAR scenarios // The 12th International Conference on the Developments in eSystems Engineering (Kazan, Russia; 7-10 October 2019).
- Zakiev A. Partially unknown environment exploration algorithm for a mobile robot / Lavrenov R., Magid E., Svinin M., Matsuno F. // Journal of Advanced Research in Dynamical and Control Systems. - 2019. - Vol. 11(08). - p. 1743-1753.
- Denisov E., Sagitov A., Yakovlev K., Su K.-L., Svinin M., Magid E. Towards Total Coverage in Autonomous Exploration for UGV in 2.5D Dense Clutter Environment // The 16th International Conference on Informatics in Control, Automation and Robotics (Prague, Czech Republic; 29-31 July 2019) - p. 409-416.
- Denisov, E., Sagitov, A., Lavrenov, R., Su, K.-L., Svinin, M., Magid, E. (2019). DCEGen: Dense Clutter Environment Generation Tool for Autonomous 3D Exploration and Coverage Algorithms Testing. Lecture Notes in Computer Science, 11659, p. 216-225.
- Шабалина К.С., Сагитов А.Г., Магид Е.А. Моделирование мобильного робота Аврора Юниор в среде ROS/GAZEBO // V Всероссийский научно-практический семинар «Беспилотные транспортные средства с элементами искусственного интеллекта» (БТС-ИИ-2019), (г. Санкт-Петербург, Россия; 22-24 мая 2019) - c. 221-229.
- Лавренов Р.О., Магид Е.А., Мацуно Ф., Свинин М.М., Сутакорн Дж. Разработка и имплементация сплайн-алгоритма планирования пути в среде ROS/Gazebo // Труды СПИИРАН, 18(1) (г. Санкт-Петербург) - с. 57-84.
- Имамеев Д.Т., Лавренов Р.О., Сагитов А.Г., Шабалина К.С., Магид Е.А., Мартинес-Гарсия Э.А., Су К.-Л. Разработка процедуры автономной параллельной парковки мобильного робота Unior // Международная научно-техническая конференция "ИНТЕРСТРОЙМЕХ - 2019" (г. Казань, Россия; 12-13 сентября 2019) - с. 58.
- Никифоров Н.А., Лавренов Р.О., Сагитов А.Г., Шабалина К.С., Магид Е.А., Свинин М.М., Хсиа К.-Х. Сравнительный анализ алгоритмов распознавания дорожных знаков для беспилотного автомобиля Unior компании Avrora Robotics // Международная научно-техническая конференция "ИНТЕРСТРОЙМЕХ - 2019" (г. Казань, Россия; 12-13 сентября 2019) - с. 62.
- Сафин Р.Н., Цой Т.Г., Лавренов Р.О., Магид Е.А. Разработка системы оптимальной передачи видеоданных с камер мобильного робота “Сервосила Инженер” // XIV Всероссийская научно-практическая конференция «Перспективные системы и задачи управления», с. 304-313.
Publications prepared for the second stage of the Project (2020-2021):
- Abbyasov B., Lavrenov R., Zakiev A., Tsoy T., Magid E., Svinin M., Martinez-Garcia E.A. Comparative analysis of ROS-based centralized methods for conducting collaborative monocular visual SLAM using a pair of UAVs // Proceedings of the 23rd International Conference on Climbing and Walking Robots and Support Technologies for Mobile Machines (CLAWAR 2020) (Moscow, Russia; 24-26 August 2020) - p. 113-120.
- Abbyasov B., Lavrenov R., Zakiev A., Yakovlev K., Svinin M., and Magid E. Automatic Tool for Gazebo World Construction: From a Grayscale Image to a 3D Solid Model // International Conference on Robotics and Automation (ICRA) (Paris, France; 31 May - 31 August 2020) - p. 7226-7232.
- Mingachev E. Comparison of ROS-based monocular visual SLAM methods: DSO, LDSO, ORB-SLAM2 & DynaSLAM / Lavrenov R., Tsoy T., Matsuno F., Svinin M., Suthakorn J., Magid E. // Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). - 2020. - №12336. - p. 222-233.
- Bereznikov D., Zakiev A. Network Failure Detection and Autonomous Return for PMB-2 Mobile Robot // International Conference on Artificial Life and Robotics (ICAROB 2020) (Oita, Japan; 13-16 January 2020) - p. 444-447.
- Guo D., Bai Y., Svinin M., Magid E. Robust Adaptive Multi-Agent Coverage Control for Flood Monitoring // The 15th Siberian Conference on Control and Communications (SIBCON 2021) (Kazan, Russia; 13-15 May 2021).
- Mingachev E. Comparative Analysis of Monocular SLAM Algorithms Using TUM & EuRoC Benchmarks / Lavrenov R., Magid E., Svinin M. // Smart Innovation, Systems and Technologies. - 2020. - №187. - p. 343-356.
- Imameev D., Zakiev A., Tsoy T., Bai Y., Svinin M., Magid E. LIDAR-based Parking Spot Search Algorithm // The 13th International Conference on Machine Vision (ICMV), 1160502 (Rome, Italy; 02-06 November 2020)
- Magid E., Lavrenov R., Zakiev A. On Simulation of Flood and Land Slide Disaster Areas in Urban Environments for Robotized Search and Rescue // Proceedings of The 5th Asia Future Conference (Alabang, Metro Manila & Los Banos, Laguna Philippines; 09-13 January 2020).
- Moskvin I., Lavrenov R., Magid E., Svinin M. Modelling a Crawler Robot Using Wheels as Pseudo-Tracks: Model Complexity vs Performance // 2020 IEEE 7th International Conference on Industrial Engineering and Applications (ICIEA 2020), p. 235-239.
- Bai Y., Svinin M., Magid E. Multi-Robot Control for Adaptive Caging and Tracking of a Flood Area // Proceedings of 59th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE) (Chiang Mai, Thailand; 23-26 September 2020) - p. 1452-1457.
- Chebotareva E., Magid E., Carballo A., Hsia K.-H. Basic User Interaction Features for Human-Following Cargo Robot TIAGo Base // Proceedings of 13th International Conference on Developments in eSystems Engineering (DeSE) (Sharjah, UAE; 14-17 December 2020)
- Gavrilova L., Kotik A., Tsoy T., Martinez-Garia E.A., Svinin M., Magid E. Facilitating a preparatory stage of real-world experiments in a humanoid robot assisted English language teaching using Gazebo simulator // Proceedings of 13th International Conference on Developments in eSystems Engineering (DeSE) (Sharjah, UAE; 14-17 December 2020)
- Dobrokvashina A., Lavrenov R., Martinez-Garcia E.A., Bai Y. Improving model of crawler robot Servosila ”Engineer” for simulation in ROS/Gazebo // Proceedings of 13th International Conference on Developments in eSystems Engineering (DeSE) (Sharjah, UAE; 14-17 December 2020)
- Khazetdinov A., Zakiev A., Tsoy T., Lavrenov R., Hsia K.-H. Standard-complaint Gazebo warehouse modelling and validation // Proceedings of 13th International Conference on Developments in eSystems Engineering (DeSE) (Sharjah, UAE; 14-17 December 2020)
- Magid E. Automating pandemic mitigation / Zakiev A., Tsoy T., Lavrenov R., & Rizvanov A. // Advanced Robotics. - 2021.
- Березников Д.В. Система обеспечения автономного возвращения колесного робота при потере связи с удаленным оператором / Закиев А.А., Магид Е.А. // Известия ЮФУ. Технические науки. - 2020. - №1 (211). - с. 90-98.
- Magid E., Tsoy T., Matsuno F., Suthakorn J., Svinin M. Collaborative robotic framework for emergency situations management in areas of flood and landslide disasters // Proceedings of the International conference "Modern robotics problems" (20 April 2021).