Disaster management information system for disaster management
Full project name: Information System of Emergency Situations Management in the areas of Floods and Landslides with Assistance of Distributed Heterogeneous Group of Robots.
The LIRS team, led by Professor Evgeni Magid, who has extensive international experience in universities and research institutes in the field of search and rescue robotics (Tsukuba University, Carnegie Mellon University, University of Bristol, etc.), for 3 years will be implementing a project on “Information System for Emergency Situations Management in Floods and Landslides with Assistance of Distributed Heterogeneous Group of Robots” in a collaboration with two teams - from Japan and under the management of Professor Fumitosi Matsuno, Kyoto University (Kyoto University), which is the 35th place in the QS World University Ranking and with a team from Thailand under the guidance of Professor Jackrit Sutkorn, Mahidol University, which is 380 place in the QS World University Ranking.
Fig. 1 presents the general idea of the proposed project. Heterogeneous groups of robots will work as separate national teams (which in turn may be divided into distributed subgroups), supplying data to a centralized information system for emergency management. The robots will perform the tasks of building thematic maps of the disaster area from the surface of the water, under the water, from the air and from the ground, including observation of the destroyed buildings. Figure 2 shows the distribution of independent tasks between national teams and their interaction by common tasks within the project.
Fig. 1. Heterogeneous distributed groups of robots collect the information necessary to create separate thematic maps, which will then be combined into a multi-layered map of the disaster zone
Fig. 2. The proposed information system and plan for joint research: the R symbol define tasks of the Russian team, J - the tasks of the Japanese team, T - tasks of the Thai team
In Asia, including the Asian part of the Russian Federation, there is a high level of risk of natural disasters, among which floods are the most frequent and the most severe losses in terms of casualties and the economy. Therefore, the development of an information system (IS) for managing a disaster management process, which makes it possible to increase the effectiveness and speed of elimination of consequences, is today one of the most pressing and sought-after tasks facing the scientific community.
Three countries participating in the project - the Russian Federation, Thailand and Japan - at certain times of the year are at risk of floods and landslides caused by torrential rains and the rapid melting of large amounts of snow cover. Using the long-term experience of our three national teams in search and rescue robotics for disaster management, the proposed project will develop a robotic information collection IC for effective disaster management in a disaster zone. The main objective of the project is the development and testing of a prototype IP, which can then become an international standard for the elimination of both national and international natural disasters.
The IS will collect data using distributed heterogeneous groups of robotic complexes, including various types of unmanned aerial vehicles (UAV), unmanned ground robots (UGV), autonomous crewless ships (ACS) and autonomous underwater vehicle (AUV). Separate maps built by distributed groups of robots will be combined into a single multi-layer thematic map of the disaster area, which will help search and rescue teams to speed up the evacuation of survivors, assess the risk of further destruction of buildings and environmental pollution, which in turn will significantly increase the chances of rescuing survivors and increase the safety of rescuers during search operations.
The new control strategies, interfaces, protocols, robot models, algorithms and software developed by the project will be tested both with the help of simulation and in the framework of field experiments. To demonstrate the effectiveness of the IS developed in the framework of the project, its testing will be conducted in cooperation with the local authorities of each of the three countries. As a result of the project, an understanding of the mechanisms associated with technologically sound decision making for effective emergency management will be improved. The development and implementation of a new generation of robotic tools for disaster management, which are in great demand by international and national emergency agencies, will make a huge contribution to the scientific, technological and industrial development of society.
Scientific novelty of the research in the Project is as follows:
A unique prototype of an unparalleled robotic information system (IS) for managing the process of eliminating the consequences of natural disasters will be developed and tested within the project.
The international teams will develop a new platform and new strategies for managing the collaborative behavior of heterogeneous robots in the tasks of collecting information, monitoring and mapping large-scale disaster zones, including land, underwater and air areas prone to natural disasters, in particular floods and landslides caused by heavy rains. New control strategies, interfaces and protocols for interaction and data exchange between heterogeneous robots will be tested in simulation and verified in field experiments.
At a beginning level of the project we are planning to develop new algorithms for simultaneous localization and mapping (SLAM) for a heterogeneous group of robots in a natural disaster, new route planning algorithms for effective coverage of areas in an unstructured environment after a natural disaster, and new algorithms for autonomous robot return when communication is lost. Moreover, we expect that our joint research on robotic systems operating in landslide conditions will lead to the creation of a new class of mobile vehicles using new management principles. The development of macro simulators (which are in responsibility of the Japanese team) based on GIS and a graphical user interface can create conditions for a new direction in the field of virtual reality, dealing with emergency modeling. If the project is successful, new scientific and practical solutions will improve the understanding of the mechanisms involved in technologically sound solutions for effective emergency management on an international scale.
During the period of the first stage (2019-2020), the following results were obtained, aimed at the Project implementation:
1. A data exchange protocol was developed, which includes the motivation for the development of a data exchange protocol in the conditions of a search and rescue operation, the conditions of a search and rescue operation. This protocol was presented by groups of scientists from Japan and Thailand. The protocol is based on the analysis and systematization of the conditions in which it should be used, and the requirements that are made to it. The requirements for the protocol are motivated by the conditions in which it should be applicable: they include both technical requirements and considerations of applicability in practice by a large number of specialists from different countries. Much attention is given to avoiding mistakes made in similar projects, and ensuring ease of implementation on a large number of robots with different hardware and software.
This protocol is universal and is planned to be implemented in the Japanese and Thai research teams.
2. Mobile robotic systems (MRS) of the laboratory were implemented in the ROS/Gazebo environment:
3. Software for media and pollution generation was developed (Microsimulator №1). This software was tested according to the developed testing methods on MRS implemented in the simulation environment. The development was conducted in the C++ programming language in the Ubuntu 16 operating system, based on the Linux. Its functionality includes:
The microsimulator is a modular software that can be added to new functionality. Its code is documented and open for use.
This software is original and has no analogues.
4. The software was developed to simulate sensor errors and failures (Microsimulator №2). This software was integrated into the Microsimulator №1 and tested according to the developed testing methods on the MRS implemented in the simulation environment.
The development was conducted in the C++ programming language in the Ubuntu 16 operating system, based on the Linux . Its functionality includes:
This software is original and has no analogues.
5. Special equipment and components kit were selected and purchased. The Russian national team carried out pilot testing of the equipment in cases where it was necessary.
6.The results of the work for 2019 were summed up. Publications collected within the framework of the 1-year grant report, cooperation with foreign partners, as well as scientific activities aimed at highlighting and popularizing the interim results of the project were analyzed.
During the period of the second stage (2020-2021), the following results were obtained, aimed at the Project implementation:
1. The Russian project team developed and tested a multifunctional microsimulator that simulates an emergency disaster zone and allows testing various algorithms for autonomous and semi-autonomous operation of both individual mobile robots and heterogeneous groups of mobile robots. New functionality appeared during the gradual development of the microsimulator, it was assigned a new serial number. So, at the end of the first stage, we presented the basic Microsimulator №2, and by the end of the second stage, we completed the development of the final version of the microsimulator – Microsimulator №5.
2. A comparative analysis was carried out in the virtual environment of Gazebo within three groups of currently existing algorithms for SLAM:
All the considered methods were presented by researchers after 2015 and are currently being actively developed. Moreover, such a comparison of these methods using the qualitative and quantitative characteristics of the algorithms has not been conducted before. The Russian team developed and implemented a new algorithm for simultaneous localization and mapping of MultiCam-SLAM, based on the integration of data from three onboard cameras of the robot. The developed algorithm was tested in the Gazebo environment and during the comparative analysis showed a higher accuracy of operation relative to the original ORB-SLAM2 algorithm. All other tested algorithms and the new proprietary algorithm were implemented in the Microsimulator №3 in the form of additional tabs that allow connecting any of the listed algorithms to any of the robots added in the simulation (Fig. 3). The developed architecture of the Microsimulator №3 with new functionality represents a new approach to simulation and has not been previously developed by other research teams. During the Microsimulator №3 testingaccording to pre-prepared methods and experimental programs, models of Hector-Quadrotor (UAV), Husky (UGV), Jackal (UGV) and Turtlebot3 (UGV) robots were used.
Fig. 3. The result of performing a collaborative SLAM algorithm by two mobile robots
3. The data exchange protocols developed at the first stage of the Project between heterogeneous robots within one team, and exchange protocols between different robot teams (in collaboration with foreign Project partners) were implemented in Microsimulator №3 as an additional tab. The protocol was tested in the Microsimulator №3 using ready-made robot models on real robots in the problem of interaction between the domestic crawler robot "Servosila Engineer" and the Tiago Base robot (PMB-2) of Spanish production (Fig. 4) and on real robots in the task of exchanging data between different groups of robots formed from the PMB-2 robot and several homemade robotic devices on the Artik10 platform. Testing experimentally confirmed the capabilities of the protocol to ensure the cooperative operation of a heterogeneous group of robots in the task of path planning and in the task of SLAM.
Fig. 4. PMB-2 (left) and "Servosila Engineer" (right) robots in a test experiment to interaction protocol
4. An algorithm has been developed for detecting a communication break and autonomous return of the mobile robot in the event of a communication system failure, taking into account possible environmental changes. The algorithm was added to Microsimulator №4 and tested on robots of various designs in a variety of environments created in the Gazebo simulator. The algorithm was also tested on real mobile robots PMB-2 and "Servosila Engineer". The algorithm can be applied to most mobile robots with different types of control and will reduce the likelihood of losing expensive equipment. For testing on real robotic devices, additional sensors and equipment were integrated into them. At the same time, the software and hardware problems that arose were solved.
5. Close interaction was organized between the managers and staff of the national research teams, which, due to the epidemiological situation, took place in an online format in the form of permanent meetings and working meetings via Zoom software (Fig. 5), and joint participation in a number of conferences. On March 19, 2020 a quadripartite international agreement on scientific cooperation was signed between Ritsumeikan University (Japan), Kyoto University (Japan), Kazan Federal University and Mahidol University (Thailand), which will expand further cooperation, including outside the framework of the project. The organization of joint sections at the international conferences "Developments in eSystems Engineering 2020" and "Siberian Conference on Control and Communications 2021" helped to attract additional interest in the interim results of the project. The leaders of all three teams joined the Organization, Program and Technical Committees of international conferences (ICAROB-2020, ICAROB-2021, DeSE-2020, DARS-2021 and SWARM-2021, SIBCON-2021, etc.), which helped to strengthen their positions and international recognition in the scientific community. During the second stage, the Russian team published 16 scientific papers (15 in publications indexed in the Scopus database, 1 in the RSCI database), 10 of which were co-authored with foreign Project teams.
Fig. 5. National team leaders and staff during an online meeting via Zoom, March 12, 2021
An important scientific result of the Project second stage is the formulation of criteria for evaluating the quality of the route to optimize the algorithms of route planning by single robots and robots operating in a group. Based on the review of path planning algorithms for optimal map coverage by one robot and a group of robots, using two-dimensional (lidar) and three-dimensional (camera) data, a new method for goal planning for robots was developed that solves the problem of optimal map coverage. The final Microsimulator №5 with integrated algorithms (Fig. 6) is a basis for interaction with scientific groups in Japan and Thailand, and also allowed the publication of a number of papers in journals and conference proceedings.
Fig. 6. The environment created in the Microsimulator №5 simulating the emergency zone
During the period of the third stage (2021-2022), the following results were obtained, aimed at the Project implementation:
1. The Russian national team has improved the remote control system for the semi-autonomous remote-controlled robot "Servosila Engineer" and for the semi-autonomous remote-controlled UAV PX. A new concept of graphical interface was developed for the robot "Servosila Engineer" due to the implementation of some algorithms (working with sensors and navigation) using Robot Operating System (ROS), while the robot is controlled without the use of ROS, which complicates the control system and creates difficulties in the algorithms. The concept has been expanded by upgrading tabs with which controlling the mobility of the robot, the movement of the manipulator available on the basis of the robot, autonomous movement based on data coming from a laser rangefinder and setting the speed of movement of joints are possible. The developed graphical interface for UAVs is written in C++\Qt Framework and is designed to control quadrocopters using ROS. It shows data about compass, altitude, elevation, roll, pitch angles. Both developed GUI systems have been successfully tested in accordance with the methodology of testing developed by the Russian team.
2. The participants of the Russian team developed and implemented algorithms for integrating thematic maps of national teams into a single disaster zone model in the Microsimulator №5. A complex 3D model of the city of Shanghai (China) was modeled. The resulting 3D map of the disaster area contains radiation and chemical contamination, which are hazardous areas.
Fig. 7. Shanghai environment with radiation pollution in the Gazebo simulator
3. The implementation of communication protocols between Microsimulator №5 and Macrosimulator, developed by the Japanese team and which are a simulation of water behavior in a flood, was carried out. The developments of Japanese team include a 3D model of the water area and formulas describing the process of wave simulation during flooding and the equations of water rise. The designed prototype of the developments of the Japanese team was integrated into the Microsimulator №5.
Fig. 8. Example of water simulation in a state of slight rise and fall of the waves in ROS/Gazebo environment
Moreover, the Russian group developed and implemented algorithms for integrating thematic maps of national teams into a single disaster zone model in the form of Manhattan urban district (New York City) for real robotic complexes in the ROS environment (Fig. 9). Warthog, Hector Quadrotor, Jackal, and Husky robots were used for running the experiments.
Fig. 9. Virtual model of Manhattan in the Gazebo simulator
4. An autonomous return software module was developed and integrated into the control system of Tiago Base and "Servosila Engineer" mobile robots, allowing them to return to the initial point where the movement was started. This may be necessary in cases of a break in communication for a certain time interval or complete loss of communication between the robot and the operator (remote robot control mode).
5. Due to the coronavirus pandemic and recommendations of the Federal Service for Surveillance on Consumer Rights Protection and Human Wellbeing to limit visits to foreign countries, efficient cooperation between the heads and participants of the Project was established remotely. A series of working meetings held every three months allowed participants to demonstrate current interim results carried out as part of the Project. Scientific developments were also presented at the international and Russian scientific conferences. Publications were prepared jointly with leaders and participants of foreign national teams. During the third stage 16 scientific articles were prepared and published by the Russian team.
