KR102318489B1 - Real-time disaster information detection system and method using artificial intelligence and drone images - Google Patents

Abstract

The present invention relates to a real-time information detection system and a method using artificial intelligence and drone images, and more specifically, to a real-time information detection system and a method using artificial intelligence and drone images, wherein drones, artificial intelligence and real-time mapping techniques are connected as a single system to make disaster investigation practices convenient and to construct an advanced system capable of grasping a disaster situation swiftly. According to an aspect of the present invention, comprised are: a drone for photographing a disaster area through a camera mounted on a main body to obtain still image information and video information; a controller for controlling the flight of the drone by communicating wirelessly with the drone and receiving information related to the flight of the drone and the still image information and video information of the disaster area; a router for communicating wirelessly with the controller; and an analysis server connected to the router to receive the information related to the flight of the drone and the still image information and video information of the disaster area, which are transmitted through the router from the controller, and to store the same in a database, analyzing disaster type detection information by inference from the still image information based on a preset artificial intelligence model, analyzing human detection information by inference from the video information, and visualizing and providing the analyzed disaster type detection information and the human detection information through an online web.

Description

Real-time disaster information detection system and method using artificial intelligence and drone images

The present invention relates to a system and method for detecting real-time disaster information using artificial intelligence and drone images, and more particularly, by linking drones, artificial intelligence, and real-time mapping technologies into one system, it is convenient and promptly It relates to a real-time disaster information detection system and method using artificial intelligence and drone images that can build an advanced system that can understand the situation.

Currently, the method of using drones to investigate the causes of disasters is to arrive at the site when a disaster occurs, fly the drone, and shoot video mainly. Precise shooting.

In order to detect disaster phenomena such as flooding, flooding, and landslides in the disaster area, the worker must continuously watch the screen to find it.

In addition, even if a disaster site is found in a video or still image, it is difficult to grasp the location in detail, so it is necessary to re-fly the drone to acquire precise data such as an orthographic image. , and the degree of overlap should be taken into account so that it can be photographed precisely.

However, in general, when flying a drone, a worker may shoot the wrong area because he or she is flying a drone depending on his or her own experience.

In order to solve this problem, the disaster situation response method using a smart drone of Korea Patent No. 10-2166432 enables safe and accurate flight operation of the drone while accurately and quickly capturing people or objects from the images transmitted by the drone at the accident site. Demonstrates recognizing skills.

However, in the case of this prior art, it is possible to accurately grasp object information such as people or objects from the image information transmitted from the drone to take quick emergency measures, and to quickly identify the types of disasters such as floods or earthquakes in the disaster area. However, there is a problem in that it is not possible to precisely analyze and respond to the specific damage area for the disaster area.

In addition, the drone takes the movement route information of the drone in reverse and returns the drone to the starting position, but does not separately store the image information captured by the drone and the related movement coordinate information. There is a disadvantage in that the work efficiency is lowered because the drone has to fly a number of times to find the damaged area.

On the other hand, in the case of the AI-based disaster damage information detection method and system using the drone of Korea Patent No. 10-2203135, which is another prior art, the construction of a learning data set based on the drone's camera video information and the drone's flight log information By building an artificial intelligence model through

However, in the case of this prior art, only the disaster type and damage area can be derived from the drone image through the artificial intelligence model based on the pre-built learning data set, and since there is no separate object recognition function, there is no separate object recognition function. There is a problem that the structure cannot be taken.

In addition, the prior art is also track information (flight trajectory, sensor position and attitude information, etc.) or captured image information according to the movement of the drone, as in the technology of the disaster situation response method using the smart drone of the Korea Patent No. 10-2166432 above. Since there is no separate storage for the disaster area, there is a disadvantage in that the drone flight must be performed multiple times depending on the experience of the operator when a precise analysis of the disaster area is required.

Accordingly, by using artificial intelligence, it is possible to identify the disaster type and disaster damage area while performing drone flight to the disaster area, and also to identify objects such as people and objects to respond to rescue, and at the same time, the drone By automatically storing captured image information and drone movement-related information in a separate database in real time to enable post-processing, a technology is required to establish an effective flight plan and increase work efficiency when re-flying the drone.

Korean Patent Registration No. 10-2166432 Korean Patent No. 10-2203135

The present invention has been devised to solve the above problems, and it is possible to automatically detect the type of disaster and information on the affected area of the disaster using artificial intelligence and drone aerial video, and search for the missing person through object recognition including people and objects. We would like to present a method and its system.

In particular, it acquires still image information and video information of the drone for the disaster area through a separate application, detects the damage area and disaster type from the still image information of the drone using artificial intelligence, and uses the human ( object) and provide the detected disaster damage type information and object recognition information to the user's web in real time.

In addition, by storing the various movement-related information and shooting information of the drone in a separate database by table, it is possible to efficiently establish a flight plan when the drone is re-flying for precise shooting of the disaster-prone area, and the user can We want to make it easy to check related video information by inquiring history information about existing projects through the web.

According to an aspect of the present invention, a drone for acquiring still image information and video information by photographing a disaster area through a camera mounted on a main body, wirelessly communicating with the drone to control the flight of the drone, and flying the drone A remote controller that receives related information and still image information and video information of the disaster area, a router that wirelessly communicates with the remote controller, and a drone flight-related information and a still image of the disaster area that are connected to the router and transmitted from the remote controller through the router Receive information and video information and store it in a database, analyze the disaster type detection information by inferring the still image information based on a preset artificial intelligence model, analyze the human detection information by inferring the video information, It is characterized in that it is configured to include an analysis server for visualizing and providing disaster type detection information and person detection information through an online web (Web).

Real-time disaster information detection system and method using artificial intelligence and drone image according to the present invention, based on a pre-learned artificial intelligence model, receives still image information from a drone flying in a disaster-prone area, detects a disaster type, and By receiving video information from the drone and identifying objects such as people and objects, it is possible to accurately identify the type of disaster and quickly detect the missing person.

In addition, the real-time visualization service allows the user to check the video information currently being filmed by the drone on the web in real time and at the same time to check the detection information derived by the artificial intelligence model from the video information in real time. This has the effect of making it easier to understand information.

In addition, it is possible to provide more precise and diverse image information to users by generating and providing Point Cloud, DSM, 3D Mesh, orthographic images, etc., through precise processing of the still image information of the disaster area received by the drone. .

In addition, in the process of transmitting and receiving flight-related information, still image information, and video information of the drone, related information is automatically stored in the database, and various analysis information made on each server is stored in real time as a database, so that when the drone is re-flyed, the database It has the effect of easily establishing a precise flight path plan of the drone through various information related to the previous flight, such as the flight trajectory of the drone and the position and attitude information of the sensor stored in the .

In addition, there is an advantage that the user can easily search the list of previous disasters on the web through the database to check the video information of the disaster event.

1 is a diagram for explaining a basic investigation system of a real-time disaster information detection system using artificial intelligence and drone images according to an embodiment of the present invention.
2 is a system diagram showing the configuration of a real-time disaster information detection system using artificial intelligence and drone images according to an embodiment of the present invention.
3 is a diagram for explaining a data flow according to the configuration of a real-time disaster information detection system using artificial intelligence and drone images according to an embodiment of the present invention.
4 is a view showing a still image real-time inference and mapping screen according to an embodiment of the present invention.
5 is a view showing a video real-time inference screen according to an embodiment of the present invention.
6 is a view showing a video transmission/reception screen according to an embodiment of the present invention.
7 is a view for explaining the provided image information according to the precise processing of the drone image according to an embodiment of the present invention.
8 is a system diagram showing a table configuration of a database according to an embodiment of the present invention.
9 is a flowchart illustrating a method for detecting real-time disaster information using artificial intelligence and drone images according to an embodiment of the present invention.

Since the present invention can apply various transformations and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

Terms such as first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining a basic investigation system of a real-time disaster information detection system 10 using artificial intelligence and drone images according to an embodiment of the present invention, and FIG. 2 is an artificial intelligence according to an embodiment of the present invention. and a system diagram showing the configuration of a real-time disaster information detection system 10 using a drone image, and FIG. 3 is a configuration of a real-time disaster information detection system 10 using artificial intelligence and drone images according to an embodiment of the present invention. It is a diagram for explaining a data flow according to the present invention, and FIG. 4 is a diagram showing a still image real-time inference and mapping screen according to an embodiment of the present invention, and FIG. 5 is a diagram showing a real-time video inference screen according to an embodiment of the present invention 6 is a view showing a video transmission/reception screen according to an embodiment of the present invention, and FIG. 7 is a diagram for explaining provided image information according to the precise processing of a drone image according to an embodiment of the present invention, 8 is a system diagram showing a table configuration of the database 480 according to an embodiment of the present invention.

1 to 3 , the real-time disaster information detection system 10 using artificial intelligence and drone images according to an embodiment of the present invention wirelessly controls the drone 100 through the manipulator 200 to detect the disaster area. A camera mounted on the drone from the sky captures the overall damage area of the disaster area.

Thereafter, the controller 200 wirelessly communicates with the router 300 provided in the disaster site investigation vehicle the flight related information of the drone 100 and the still image information and video information of the disaster area to analyze connected to the router 300 . It is transmitted to the server 400 .

The analysis server 400 forms a system for detecting and mapping disaster type information and person/thing information by inferring transmitted still image information and moving image information based on a preset artificial intelligence model.

Here, the analysis server 400 is a computer device equipped with a high-performance GPU, and may be provided in an investigation vehicle that explores a disaster site and may be formed as a field operation artificial intelligence system that can be operated at a disaster site, or a separate disaster site. It can be processed by the management agency's automatic detection server to form an artificial intelligence system for internal operation that automatically detects with high performance.

The real-time disaster information detection system 10 using artificial intelligence and drone images according to an embodiment of the present invention includes a drone 100 , a controller 200 , a router 300 , and an analysis server 400 .

The drone 100 acquires still image information and video information by photographing a disaster area through a camera mounted on the main body.

The manipulator 200 wirelessly communicates with the drone 100 to control the flight of the drone 100 , and receives flight-related information of the drone 100 and still image information and video information of the disaster area. Here, the flight-related information of the drone 100 includes roll, pitch, and yaw values of the gimbal mounted on the drone in addition to the roll, pitch, and yaw values of the drone. It includes attitude information and position (X, Y, Z) information of the drone 100 including

The controller 200 includes a still image transmission application (App) for transmitting still image information obtained through the drone 100 in real time and a video transmission application (App) for transmitting the video information obtained through the drone 100 in real time. ) is provided.

That is, in the present embodiment, since the tablet is attached to the remote controller 200 , flight and data acquisition conditions of the drone 100 can be set. In addition, using a communication method called OcuSync 2.0, drone control and data transmission are performed at the same time.

OcuSync technology uses a unique method called the Cross Layer Protocol. In this method, the connection between the remote controller 200 and the drone 100 is not one, but multiple FDM (Frequency Division Multiflexing) and TDM (Time Division Multiflexing) networks are connected, and the frequency band and bandwidth are adjusted according to the surrounding radio wave interference situation. You can adjust it to keep it in optimal condition.

The router 300 is wirelessly connected to the remote controller 200 and wired to the analysis server 400 of the investigation vehicle, and performs wireless communication with the remote controller 200 via Wi-Fi to communicate the flight of the drone 100 from the controller 200. It receives information and still image information and video information of the disaster area and transmits it to the analysis server 400 .

The router 300 may mount the antenna outside the survey vehicle as needed to operate in the field.

The analysis server 400 receives flight-related information of the drone 100 transmitted from the remote controller 200 through the router 300, and still image information and video information of the disaster area, and stores it in the database 480, Based on the set artificial intelligence model, the still image information is inferred to analyze the disaster type detection information, the human detection information is analyzed by inferring the video information, and the analyzed disaster type detection information and person detection information are transmitted through the web screen. It performs the role of providing visualization.

The analysis server 400 includes an image transmission/reception server 410, a video transmission/reception server 420, an artificial intelligence (AI) inference server 430, a mapping server 440, a map visualization server 450, and a web service server 460. ), a precision processing server 470 and a database 480 may be included.

The image transmission/reception server 410 receives the still image information of the disaster area transmitted from the remote controller 200 through the router 300, and transmits the image and metadata of the received still image information to the database 480, while It is transmitted to the intelligent inference server 430 and the mapping server 440 .

Here, the metadata includes latitude, longitude, left and right tilt (roll), front and rear tilt (pitch), horizontal rotation (yaw), and camera model information.

The video transmission/reception server 420 receives the video information of the disaster area transmitted from the remote controller 200 through the router 300, and transmits the received video information to the database 480, to the artificial intelligence inference server 430. send In this case, when receiving data from the remote controller 200, a video transmission protocol called RTMP may be used.

On the other hand, in the present invention, it is characterized in that the still image information and video information acquired through the drone 100 are received in real time and stored in the database 480, and at the same time are visualized in real time through the analysis process of the analysis server 400, The image transmission/reception server 410 and the video transmission/reception server 420 of the present invention can be said to be characteristic components unlike general data transmission/reception technology.

The artificial intelligence (AI) inference server 430 detects a disaster type through object detection (geometry, converse coordinates) using images and metadata of still image information based on a preset artificial intelligence model. do.

In addition, the artificial intelligence inference server 430 detects people by identifying people and things through object detection in video information.

On the other hand, in order to select the optimal learning technique for each type of disaster, it is necessary to decide whether to use the classification, object detection, and segmentation models according to the disaster damage type classification system set according to the type of disaster damage, not the disaster. do.

Among them, object detection finds an object of interest in an image, creates a bounding box, and predicts a class to which the bounding box belongs. Object detection is suitable for structural damage sites such as building destruction, and division is suitable for natural environment damage sites such as river flooding, steep slopes, and wildfire damage.

In addition, in the case of the artificial intelligence model according to the present invention, since it is necessary to find a desired object in real time from images and videos, an optimal model for each data was selected.

In other words, the still image transmitted in real time from the drone 100 has a longer detection time than a video that transmits 24 to 30 frames per second, so the model size is large, but the 2-stage Cascade Mask R with high accuracy. -CNN model was used.

The basic principle of the Cascade Mask R-CNN model is that n classifiers can increase the classification accuracy by repeatedly classifying the bounding boxes detected in the classifier of the previous step (n-1) as input.

The learning data of the Cascade Mask R-CNN model was detected by submersion, collapse, loss, and landslide in still images and moving images. As training data, VIA (VGG Image Annotator) was used.

The training of the model was performed using an API called MMDetection based on Pytorch. The backbone of the model used ResNeXt101.

As a result of the training, the object area was detected with an accuracy of about 70% for each class.

On the other hand, since the amount of data transmitted per hour is large for video, the accuracy is low, but a light and fast 1-stage SSD (Single Shot Multibox Detector) model or YOLOv3 can be utilized, and YOLOv3 was used in this embodiment.

The target to be detected in the video is a person appearing in the drone video, and processing speed is of utmost importance to extract a specific object in real time.

The YOLOv3 model is a model that improves the speed and performance of the existing YOLO v2, and it shows a much faster detection performance with similar detection accuracy to RetinaNet based on AP50.

As the training data of the YOLOv3 model, 108,226 objects were extracted from 6,623 images composed by extracting the image and the human class of VisDrone, an open data set.

In the Visdrone data, which accounts for the largest portion of the images used as the evaluation dataset, there were a lot of images with a very large number of people or images that were not easy to distinguish with the naked eye. In high-altitude images that are difficult to recognize or when people are photographed vertically, the accuracy is low because it is not detected as a person or falsely detected. Therefore, the reasoning was repeated with the identifiable image, and the accuracy of 68.28% was shown in the 204 evaluation images of the relatively low flight altitude and the degree of recognition of the whole human body by giving an inclination to the shooting angle.

The mapping server 440 serves to generate ortho image information in real time by using the image and metadata of the still image information. In this embodiment, LiveDroneMap technology for mapping still images transmitted in real time is applied.

In other words, LiveDroneMap is a technology that determines the geometric position of an image by using the position (X, Y, Z) and posture (Rol, Pitch, Yaw) information of the sensor acquired at the time of capturing the still image transmitted in real time. .

In order to actually check the image in which the geometrical location is determined with the human eye and recognize the location, it must be superimposed with the map. The method of superimposing a still image, which is not periodically transmitted in real time, with a map can utilize the previously developed algorithm without particularly difficult additional development. However, data is transmitted periodically in real time and various aspects must be considered in order to visualize it.

In particular, in the present invention, not only still images but also information on objects detected in images are stored in the database 480 and must be visualized at the same time, so synchronization of images and detection information, communication for periodic information transmission, etc. must be considered. .

In the case of disaster type detection information using still image information, the map visualization server 450 visualizes a map screen on the user's Web, and object information detected from the still image information through the artificial intelligence inference server 430 . (disaster area recognition information) and ortho image information generated by the mapping server 440 are visualized on a map screen to provide real-time disaster type detection information.

In addition, in the case of human detection information using video information, a video screen is visualized on the user's web, and object information (human recognition information) detected from video information through the artificial intelligence inference server 430 is displayed on the video screen. It provides real-time person detection information by visualizing it.

On the other hand, the visualization technology according to the present invention is characterized in that the image transmitted in real time is geometrically corrected and then generated as an orthographic image, which can be seamlessly mashed up on the map.

The web service server 460 provides a web service screen on the Internet, and provides the user with disaster type detection information and person detection information analyzed by the artificial intelligence inference server 430 through the provided web service screen in real time.

Here, the web service screen includes a still image providing screen unit for providing inference and mapping information of still image information, and a video providing screen unit for providing inference and mapping information of moving image information.

The still image providing screen consists of a plurality of selection buttons, an information input window, an information output window, and an image information selection item consisting of an image display window, and the image information selection items include 1) project search/period, 2) project inquiry/management. , 3) inference/orthoimage/map visualization (result visualization), 4) individual photo inquiry, 5) individual still image visualization, and 6) individual image information.

That is, referring to FIG. 4 , the still image providing screen unit is a still image real-time inference and mapping service screen, and is the first screen the user encounters when accessing the service.

The screen can be accessed through a web browser URL. When you first connect, you will come across a page divided into six screens, and each screen is designed to be folded and unfolded as needed, allowing you to use the screen flexibly and widely.

The screen on the left is connected to the screen related to the project once, and a new project is automatically created when real-time inference and mapping are carried out. In the project list, you can search for projects that have been carried out in the past, and you can also search by date. When a specific project is selected, the map screen moves to the site that has been performed in the past, and a list is created where each individual image can be inquired at the bottom. Whenever each image is inquired, the corresponding image is visualized on the map and image viewer. Projects in which images are being transmitted in real time are displayed as live icons, and when a project marked live is selected, inferences and mappings are sequentially made from the images transmitted in real time, and they are visualized on the map and in the image viewer.

The video providing screen unit consists of a plurality of selection buttons, an information input window, an information output window, and a video information selection item consisting of an image display window, and the video information selection item includes project search (not shown), project inquiry/management (not shown). ), 1) real-time image, and 2) detected object information.

That is, referring to FIG. 5 , the video providing screen unit is a video real-time inference service screen, and is a screen for detecting a person in a video transmitted in real time. The user can see information about the person and the probability that the bounding box was processed in the video.

Also, referring to FIG. 6 , the video providing screen unit may provide a video transmission/reception screen for confirming whether a video is normally received and transmitted from the video transmission/reception server.

On this video sending/receiving screen, 1) video streaming to view the video streaming list, 2) video play, to show the list of streaming or streamed videos when selecting the list, 3) selection item consisting of video list can provide

The precision processing server 470 performs precise processing with WebODM using a plurality of still image information, generates Point Cloud, DSM, 3D Mesh, and orthographic images, and provides them to the user's web.

That is, referring to FIG. 7 , by processing multiple drone images taken with a drone in the field to produce one orthographic image with location coordinates, 3D model, Point Cloud, DSM, etc. can be checked.

The database 480 stores the flight-related information of the drone 100, images of still image information, metadata, and video information, and disaster type detection information and person detection information inferred through the artificial intelligence inference server 430 . It serves to store

Referring to FIG. 8 , the database 480 includes a detection object table 481 , a flight plan table 482 , a track information table 483 , a flight track table 484 , and a flight image information table 485 . It is configured so that most information generated during data processing is stored in the database 480 so that necessary information can be inquired in the future.

The detection object table 481 stores information of the object detected by the artificial intelligence inference server 430, and the flight plan table 482 stores flight start, end, and token information of the drone 100, and track information The table 483 stores the flight trajectory of the drone 100 and the position and attitude information of the sensor, and the flight track table 484 separately extracts and stores only information necessary for calculation from the track information, and the flight image information table 485 ) stores basic information of various types of transmitted image information.

In addition, of course, the database 480 may include a plurality of tables for storing additional additional information.

That is, the database 480 includes not only flight-related information, still image information and video information of the drone 100 , but also detection object information inferred through the artificial intelligence inference server 430 , and the mapping server 440 . It is characterized in that various information processed in each server, such as ortho image information and image and video information visualized through the visualization server 450, are automatically stored in real time.

Hereinafter, a method of detecting disaster information using the real-time disaster information detection system 10 using the artificial intelligence and drone images as described above will be described in detail with reference to FIG. 9 .

9 is a flowchart illustrating a method for detecting real-time disaster information using artificial intelligence and drone images according to an embodiment of the present invention.

Referring to FIG. 9 , by controlling the drone through a controller, flight related information of the drone and still image information and video information of a disaster area are acquired ( S110 ).

At this time, the still image information of the disaster area is obtained by driving the still image transmission application provided in the remote controller, and the video information of the disaster area is obtained by driving the video transmission application provided in the remote controller.

Thereafter, the flight-related information of the drone obtained from the remote controller and the still image information and video information of the disaster area are wirelessly communicated with the router and transmitted to the analysis server connected to the router (S120).

The analysis server receives the still image information transmitted from the router through the image transmission/reception server, transmits the image and metadata of the received still image information to the database and stores it, and transmits it to the artificial intelligence inference server and the mapping server.

At the same time, the analysis server receives the video information transmitted from the router through the video transmission/reception server, transmits the received video information to the database, stores it, and transmits it to the artificial intelligence inference server.

The artificial intelligence inference server infers disaster type detection information through object detection using images and metadata of still image information based on the preset Cascade Mask R-CNN model, and infers the disaster type detection information from the video information based on the preset YOLOv3 model. People and things are identified through detection to infer human detection information ( S130 ).

Thereafter, orthographic information is generated using the image and metadata of the still image information through the mapping server (S140).

Then, the map screen is visualized on the online web through the visualization server, and the object information detected from the still image information and the ortho image information generated by the mapping server are visualized on the map screen through the artificial intelligence inference server (S150) in real time. Provides disaster type detection information.

In addition, a video screen is visualized on the online web, and object information detected from video information is visualized on the video screen through an artificial intelligence inference server (S150) to provide real-time person detection information.

In this case, the web service screen may be provided online through the web service server, and disaster type detection information and person detection information may be provided to the user in real time through the provided web service screen.

Here, a detailed description of the web service screen is the same as the web service screen described above with reference to FIGS. 4 to 6 , and thus a detailed description thereof will be omitted.

In addition, when request information for precision-processed image information is input from the user through the online web, it uses a number of still image information through the precision processing server and processes it precisely with WebODM, such as Point Cloud, DSM, 3D Mesh, and orthographic image. can be created and provided on the online web.

On the other hand, detection object information inferred through the artificial intelligence inference server, ortho image information generated through the mapping server, and image and video information visualized through the visualization server are stored in the database in real time, of course.

As described above, the real-time disaster information detection system 10 and method using artificial intelligence and drone images according to the present invention is based on a pre-learned artificial intelligence model, and a still image from the drone 100 flying in the disaster area. By receiving information to detect the type of disaster, and by receiving video information from the drone 100 to identify objects such as people and objects, it is possible to perform on-site disaster cause investigation more quickly and accurately, and It has the effect of effectively conducting the search for missing persons.

In addition, it is possible to provide more precise and diverse image information to users by generating and providing Point Cloud, DSM, 3D Mesh, orthographic images, etc. through precise processing of the still image information of the disaster area received by the drone 100. It works.

In addition, in the process of transmitting and receiving flight-related information, still image information, and video information of the drone 100 , related information is automatically stored in the database 480 , and various analysis information performed in each server is stored in the database 480 in real time. By doing so, when the drone 100 is re-flyed, the flight trajectory of the drone 100 stored in the database 480 and the precise flight path plan of the drone 100 through various information related to the previous flight, such as the position and posture information of the sensor There is an effect that can be easily established. In addition, there is an advantage in that the user can easily inquire a list of previous disasters on the web through the database 480 to check image information of the disaster event.

The above-described preferred embodiments of the present invention have been disclosed for purposes of illustration, and various modifications, changes, and additions may be made by those skilled in the art with respect to the present invention within the spirit and scope of the present invention, and such modifications, changes and Additions should be considered to fall within the scope of the following claims.

10: Real-time disaster information detection system using artificial intelligence and drone images
100 : drone
200: remote controller
300: router
400: analysis server
410: image sending/receiving server
420: video transmission and reception server
430: artificial intelligence inference server
440 : mapping server
450: map visualization server
460: web service server
470: precision processing server
480 : database

Claims (26)

a drone for acquiring still image information and video information by photographing a disaster area through a camera mounted on the main body;
a controller for controlling the flight of the drone through wireless communication with the drone and receiving flight related information of the drone and still image information and video information of the disaster area;
a router for wireless communication with the remote controller; and
It is connected to the router, receives the flight-related information of the drone transmitted from the remote controller through the router, and still image information and video information of the disaster area, and stores it in a database, and stores the still image information based on a preset artificial intelligence model. Analysis server for analyzing the disaster type detection information by inferring, analyzing the human detection information by inferring the video information, and providing the analyzed disaster type detection information and person detection information visualized through an online web; consists of,
The controller is
A still image transmission application (App) for transmitting the still image information obtained through the drone in real time and a video transmission application (App) for transmitting the video information obtained through the drone in real time are provided,
The analysis server,
AI that detects disaster types through object detection using images and metadata of the still image information based on a preset artificial intelligence model, and detects people by identifying people and objects through object detection in the video information inference server;
a mapping server for generating ortho image information in real time by using the image and metadata of the still image information;
Visualize a map screen on the online web, and visualize the object information detected from the still image information through the artificial intelligence inference server and the ortho image information generated by the mapping server on the map screen to provide real-time disaster type detection information a map visualization server that visualizes a video screen on the online web and visualizes object information detected from the video information on the video screen through the artificial intelligence inference server to provide real-time person detection information; and
a database for storing flight-related information of the drone, image and metadata of the still image information, and the video information, and storing disaster type detection information and person detection information inferred through the artificial intelligence inference server;
an image transmitting/receiving server that receives the still image information transmitted through a router, transmits the image and metadata of the received still image information to the database, and transmits the image and metadata to the artificial intelligence inference server and the mapping server; and
a video transmission/reception server that receives the video information transmitted through the router, transmits the received video information to the database, and transmits it to the artificial intelligence inference server;
The artificial intelligence inference server,
Real-time disaster using artificial intelligence and drone image, characterized in that the disaster type is detected from the still image information using a preset Cascade Mask R-CNN model, and a person is detected from the video information using a preset YOLOv3 model information detection system. delete The method of claim 1,
The router and the analysis server are provided inside the investigation vehicle, and a real-time disaster information detection system using artificial intelligence and drone images, characterized in that it is connected by wire. delete delete The method of claim 1,
Real-time disaster information detection system using artificial intelligence and drone images, characterized in that the disaster types include flooding, collapse, loss, and landslides. delete The method of claim 1,
The database is
a detection object table for storing information on objects detected by the artificial intelligence inference server;
Flight plan table for storing flight start, end, and token information of the drone;
a track information table for storing the flight trajectory of the drone and the position and orientation information of the sensor;
Flight track table for extracting and storing only information necessary for calculation from track information; and
Flight image information table for storing basic information of various types of image information to be transmitted;
Real-time disaster information detection system using artificial intelligence and drone images, characterized in that it comprises a. The method of claim 1,
The analysis server,
A precision processing server that uses a large number of still image information and processes it precisely with WebODM to generate Point Cloud, DSM, 3D Mesh, and orthographic images and provide them to the online web;
Real-time disaster information detection system using artificial intelligence and drone images, characterized in that it further comprises. The method of claim 1,
The analysis server,
a web service server for providing a web service screen on-line and providing, in real time, disaster type detection information and person detection information analyzed by the analysis server through the provided web service screen to a user;
Real-time disaster information detection system using artificial intelligence and drone images, characterized in that it further comprises. 11. The method of claim 10,
The web service screen is
Real-time disaster using artificial intelligence and drone images, characterized in that it comprises a still image providing screen unit for providing inference and mapping information of still image information, and a video providing screen unit for providing inference and mapping information of moving image information information detection system. 12. The method of claim 11,
The still image providing screen unit,
It consists of a plurality of selection buttons, an information input window, an information output window, and an image information selection item consisting of an image display window, and the image information selection items include project search, project inquiry/management, result visualization, individual photo inquiry, and individual still images. Real-time disaster information detection system using artificial intelligence and drone images, characterized in that it includes visualization and individual still image information. 12. The method of claim 11,
The video providing screen unit,
It consists of a video information selection item consisting of a plurality of selection buttons, an information input window, an information output window, and an image display window, and the video information selection item includes project search, project inquiry/management, real-time image, and detection object information. Real-time disaster information detection system using artificial intelligence and drone images. In a disaster information detection method of a real-time disaster information detection system using artificial intelligence and drone images,
controlling the drone through a controller to obtain flight-related information of the drone and still image information and video information of the disaster area;
transmitting the flight-related information, the still image information, and the moving image information of the drone obtained from the controller wirelessly with a router to an analysis server connected to the router; and
Disaster type detection information by storing the flight-related information of the drone transmitted from the router through the analysis server, still image information and video information of the disaster area in a database, and inferring the still image information based on a preset artificial intelligence model and analyzing the human detection information by inferring the video information, and visualizing the analyzed disaster type detection information and person detection information through an online web;
The remote controller has a still image transmission application (App) to acquire still image information of the disaster area and transmits it to the router, and has a video transmission application (App) to acquire and transmit video information of the disaster area to the router,
The analysis server receives the still image information transmitted from the router through the image transmission/reception server provided in the analysis server, and transmits and stores the image and metadata of the received still image information to the database while storing the artificial information provided in the analysis server. It is sent to the intelligent inference server and mapping server,
Receives the video information transmitted from the router through the video transmission/reception server provided in the analysis server, transmits the received video information to the database, and transmits it to the artificial intelligence inference server of the analysis server while storing,
Based on a preset artificial intelligence model through the artificial intelligence inference server of the analysis server, disaster types are detected through object detection in images and metadata of still image information, and people and things are identified through object detection in video information. detect people,
Orthoimage information is generated using the image and metadata of the still image information through the mapping server provided in the analysis server,
The map screen is visualized on the online web through the visualization server provided in the analysis server, and the object information detected from the still image information and the ortho image information generated by the mapping server are visualized on the map screen through the artificial intelligence inference server. Provides real-time disaster type detection information, visualizes a video screen on the online web, and visualizes object information detected from video information on the video screen through an artificial intelligence inference server to provide real-time person detection information,
The artificial intelligence inference server detects a disaster type from still image information using a preset Cascade Mask R-CNN model, and detects a person from video information using a preset YOLOv3 model AI and drone images, characterized in that Real-time disaster information detection method using delete delete delete delete delete 15. The method of claim 14,
AI and drone image, characterized in that the detection object information inferred through the artificial intelligence inference server, the ortho image information generated through the mapping server, and image and video information visualized through the visualization server are stored in the database in real time. Real-time disaster information detection method using 15. The method of claim 14,
The database is
a detection object table for storing information on objects detected by the artificial intelligence inference server;
Flight plan table for storing flight start, end, and token information of the drone;
a track information table for storing the flight trajectory of the drone and the position and orientation information of the sensor;
Flight track table for extracting and storing only information necessary for calculation from track information; and
Flight image information table for storing basic information of various types of image information to be transmitted;
Real-time disaster information detection method using artificial intelligence and drone images, characterized in that it comprises a. 15. The method of claim 14,
In the step of visualizing the disaster type detection information and the person detection information analyzed through the analysis server through an online web,
Artificial intelligence and drone images characterized by generating Point Cloud, DSM, 3D Mesh, and orthographic images and providing them to the online web by using WebODM precision processing using a large number of still image information through the precision processing server of the analysis server. Real-time disaster information detection method using 15. The method of claim 14,
In the step of visualizing the disaster type detection information and the person detection information analyzed through the analysis server through an online web,
Artificial intelligence and drone image, characterized in that by providing a web service screen online through the web service server of the analysis server, and providing the disaster type detection information and the person detection information to the user in real time through the provided web service screen Real-time disaster information detection method using 24. The method of claim 23,
The web service screen is
Real-time disaster using artificial intelligence and drone images, comprising: a still image providing screen unit for providing inference and mapping information of still image information; and a video providing screen unit for providing inference and mapping information of moving image information How to detect information. 25. The method of claim 24,
The still image providing screen unit,
It consists of a plurality of selection buttons, an information input window, an information output window, and an image information selection item consisting of an image display window, and the image information selection items include project search, project inquiry/management, result visualization, individual photo inquiry, and individual still images. Real-time disaster information detection method using artificial intelligence and drone image, characterized in that it includes visualization and individual still image information. 25. The method of claim 24,
The video providing screen unit,
It consists of a video information selection item consisting of a plurality of selection buttons, an information input window, an information output window, and an image display window, and the video information selection item includes project search, project inquiry/management, real-time image, and detection object information. Real-time disaster information detection method using artificial intelligence and drone images.

Images