KR102369229B1 - Risk prediction system and risk prediction method based on a rail robot specialized in an underground tunnel - Google Patents

Abstract

The present invention relates to a risk prediction system and a risk prediction method based on a rail robot specialized in an underground tunnel. According to the present invention, artificial intelligence-based analysis is performed with image data taken by a CCTV in an underground tunnel and taken by a rail robot and sensor data collected by various environmental sensors installed on the rail robot. As a result, it is possible to provide real-time risk prediction and early response safety management service by checking whether an entering person is authorized, detecting a falling worker, detecting facility water leakage, performing flame and smoke detection, and detecting another abnormal situation. According to the present invention, based on the image and sensor information of the CCTV in the underground tunnel and the rail robot, an optimal artificial intelligence deep learning algorithm is applied and AI data learning is performed. As a result, high-accuracy risk prediction and early response safety management services can be provided. In addition, it is possible to enhance security and minimize human damage by automatically identifying authorized and unauthorized persons related to the underground tunnel and building a monitoring system capable of immediate detection and response in the event of an abnormal situation such as falling.

Description

Risk prediction system and risk prediction method based on rail robot specialized in underground tunnel

The present invention relates to a risk prediction system and risk prediction method based on a rail robot specialized in an underground tunnel that can predict risks in an underground tunnel and support safety management work by analyzing images and various sensor data with artificial intelligence.

The underground tunnel is a national important facility and national security facility that jointly installs and operates infrastructure such as electric power, communication, water supply, and heating underground in the city. Of course, huge property damage occurs.

Gonggu, an underground facility with a lot of underground facilities necessary for daily life, such as electricity, communication, gas, water supply, and sewage pipes, is operated on the basis of daily patrols by two people in one group. However, there is no system in place to monitor equipment failures or fires occurring in the common area, and in order to accurately grasp information and circumstances related to safety accidents in the common area, patrol personnel or business personnel directly go to the site and check with the naked eye. Since tunnels of several kilometers or more are connected like a maze, it is difficult to quickly determine the location even if a disaster occurs, and it is difficult for firefighters to enter, resulting in a lot of damage when an accident occurs.

According to the nature of work in the field, it is necessary to switch from the labor-intensive visual inspection management method to a system-oriented scientific management system using image data collected through artificial intelligence, and to respond preemptively to the increasing safety demand. For this purpose, it is necessary to convert the safety management system to a preventive system by removing disaster risk factors in advance through the artificial intelligence risk prediction service.

Registered Patent No. 10-2219809

The present invention analyzes the image data captured by CCTV in the underground tunnel, the sensor data collected by various environmental sensors installed in the rail robot, and the image data photographed by the rail robot with artificial intelligence to confirm whether an occupant is authorized, detect a worker's collapse, and facilities The purpose of this is to provide a rail robot-based risk prediction system and risk prediction method specialized in underground tunnels that can provide real-time risk prediction and early response safety management services through leak detection, flame and smoke detection, and other abnormal situation detection.

In order to solve the above problem, the image transmitted by the fixed surveillance camera installed in the underground tunnel and the image and environmental sensor data transmitted by the rail robot moving along the rail of the underground tunnel are analyzed with a deep learning-based learning model, A rail robot-based risk prediction system specializing in underground tunnels that provides high-accuracy risk prediction information for each facility and environment information in real time is provided.

The rail robot-based risk prediction system specialized in the underground tunnel includes: a rail robot that moves along the rail installed in the underground tunnel and is equipped with a camera and a plurality of environmental sensors; Fixed surveillance cameras installed in the underground tunnels; an image collecting unit for receiving and transmitting the video image captured by the rail robot and the fixed surveillance camera in real time; an image processing unit for decoding and pre-processing the image image transmitted by the image collection unit; a robot control unit that receives and transmits a plurality of sensor data and measurement location information measured and transmitted by the rail robot in real time, and controls the rail robot; a joint data learning unit that stores the image image pre-processed by the image processing unit and a plurality of sensor data transmitted by the robot control unit in a database, and learns with a deep learning-based learning engine; Common area for diagnosing in real time whether or not entrants and workers and facilities are dangerous by applying the image image transmitted by the image collection unit and the sensor data transmitted by the robot control unit to the deep learning-based learning engine learned by the common area data learning unit situation diagnosis department; And based on the diagnosis of the common area situation diagnosis unit, it includes a common area situation providing unit that identifies whether access is authorized and the dangerous situation of the underground common area and displays it in real time.

Specifically, the cavity data learning unit includes: an object detection module for detecting an object by measuring a distance from an image image to an object and estimating the size, width, and width of the detected object; an object tracking module that tracks the object by predicting the trajectory of the detected object; an abnormal event detection module for detecting the tracked object and detecting the collapse; and a deep learning-based object detection and tracking learning engine comprising a marker detection module that recognizes an Aruko marker attached to an object tracked by the object tracking module and determines whether an authorized person or a non-authorized person. .

The cavity data learning unit may be configured to use an artificial intelligence-based image segmentation convolutional neural network (DeepLab V3+) or Mask R-CNN to detect a leak in an image image.

The image processing unit decodes the image frame received from the rail robot, extracts feature points from each frame image using a deep learning-based feature map extraction engine, matches the same feature points among feature points of each frame image, and It may be configured to include an image shake correction module that calculates mobility and moves the feature point in a direction opposite to the movement direction and by a size.

When the rail robot detects an occupant during patrol, the rail robot moves within a certain distance capable of recognizing the occupant's face through the control of the robot controller, and then transmits an image of the occupant's face in a stationary state in real time, diagnosing the joint sphere situation The unit can be configured so that the rail robot patrols again through the robot controller after confirming whether the person is authorized by applying the face image of the person who is photographed in the stationary state of the rail robot to a deep learning-based learning engine.

Here, the communal data learning unit uses a deep learning-based object detection and tracking learning engine to detect an occupant from a real-time video image taken by the rail robot, and measure the distance to the detected entrant so that the face recognition of the entrant is possible. The robot control unit controls the rail robot to move within a distance, and the real-time image of the person transmitted by the rail robot is applied to the deep learning-based face detection and recognition learning engine within a certain distance where face recognition is possible to detect the person's face. It can be configured to detect and recognize whether or not an occupant is authorized.

The common tunnel data learning unit applies a plurality of sensor data collected for each underground tunnel area to the One-Class SVM-based sensor risk prediction learning engine to learn the environmental risk situation of the underground tunnel, or LSTM (Long Short-term Memory)-based It can be configured to learn the environmental risk situation of the underground tunnel by applying it to the sensor risk prediction learning engine.

On the other hand, in order to solve the problem, the rail robot-based risk prediction method specialized in the underground tunnel moves along the rail installed in the underground tunnel, and the rail robot installed with a camera and a plurality of environmental sensors shoots an image, and measures a plurality of sensor data. transmitting in real time; A fixed-type surveillance camera installed in the underground communal area capturing an image and transmitting the image in real time; Compensating the video image received from the rail robot in real time using a deep learning-based feature map extraction engine; Storing the video image transmitted by the rail robot and the fixed surveillance camera and the plurality of sensor data transmitted by the robot controller in a database, and learning with a deep learning-based learning engine; A step of diagnosing in real time whether a person is authorized or not and a dangerous situation of workers and facilities by applying the video image transmitted by the rail robot and the fixed surveillance camera and the sensor data transmitted by the rail robot to a deep learning-based learning engine; And based on the diagnosis, it includes the step of identifying whether or not access is authorized and the dangerous situation of the underground common area, and displaying it in real time.

In the step of the rail robot taking an image and transmitting it in real time, when the rail robot detects an entrant during patrol, it moves within a certain distance where the entrant's face recognition is possible through the control of the server, and then shoots the entrant's face in a stationary state In the step of transmitting an image in real time and diagnosing, the rail robot patrols again after checking whether the person is authorized by applying the face image of the person who is photographed in the stationary state to a deep learning-based learning engine. can be configured to do so.

Specifically, in the learning step, a person's face recognition is performed by detecting an occupant from a real-time video image taken by the rail robot using a deep learning-based object detection and tracking learning engine, and measuring the distance to the detected entrant. Controlling the rail robot to move within a predetermined distance as possible, and diagnosing includes applying a real-time image of an occupant transmitted by the rail robot to a deep learning-based face detection and recognition learning engine within a certain distance where face recognition is possible. It can be configured to detect and recognize the face of the person and confirm whether the person is an entrant or not.

According to the risk prediction system and risk prediction method based on the rail robot specialized in the underground tunnel according to the present invention, it is possible to predict the risks inside the underground tunnel (fire, water leakage, safety accident, pedestrian collapse), and to patrol the underground tunnel, such as checking whether unauthorized persons enter or not. By resolving facility monitoring using rail robots and artificial intelligence technology, it is possible to solve the problem of shortage of professional manpower for facility management in the underground tunnel, and it is possible to replace or supplement tasks that are difficult for workers to perform, thereby ensuring worker safety and risk Real-time monitoring of facilities and rapid response are possible.

According to the present invention, it is possible to strengthen security and minimize human casualties by automatically identifying authorized/non-authorized underground tunnels, and building a monitoring system that can detect and respond immediately when an abnormal situation such as a collapse occurs.

1 is a configuration diagram of a risk prediction system based on a rail robot specialized in an underground tunnel according to the present invention.
2 is two functional diagrams of an image processing unit included in an artificial intelligence analysis server.
3 is a flowchart of an image shake correction module included in the image processing unit.
4 is a functional diagram of various learning engines constituting a deep learning-based learning engine included in an artificial intelligence analysis server.
5 is a detailed functional diagram of an object detection and tracking learning engine.
6 is an exemplary diagram illustrating a process in which a deep learning-based learning engine classifies an image, converts an image into an image, and then processes the image into learning data through an annotation process.
7 is an exemplary diagram illustrating a system screen for determining whether a person is authorized by applying a facial image of an occupant taken with a fixed surveillance camera and a rail robot to a deep learning-based face detection and recognition learning engine.
8 is a method for detecting and tracking pedestrians by applying the image of pedestrians in the underground tunnel taken with a rail robot to an object detection and tracking learning engine, and recognizing Aruco markers attached to pedestrians to determine whether an authorized person is It is an example diagram showing the system screen.
9 is an exemplary diagram illustrating a scenario in which the rail robot authenticates the face of an entrant when the rail robot detects an entrant while patrolling the underground common area.
10 is an exemplary diagram illustrating a system screen for detecting various types of abnormal events by applying an underground tunnel image taken with a rail robot to an object detection and tracking learning engine.
11 is a flowchart illustrating a process of detecting a leak area by applying a leak image of an underground tunnel taken with a rail robot to a leak detection learning engine.
12 is an exemplary diagram illustrating a process of diagnosing an environmental risk situation of a rail robot by applying environmental sensor data measured for each underground tunnel area to a sensor risk prediction learning engine.
13 is a flowchart showing a process in which the underground tunnel specialized rail robot-based risk prediction method according to the present invention is performed.

Advantages and features of the present invention, and a method for achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings.

However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms.

In the present specification, the present embodiment is provided so that the disclosure of the present invention is complete, and to completely inform those of ordinary skill in the art to which the present invention belongs, the scope of the invention.

And the invention is only defined by the scope of the claims.

Accordingly, in some embodiments, well-known components, well-known operations, and well-known techniques have not been specifically described to avoid obscuring the present invention.

In addition, the same reference numerals refer to the same elements throughout the specification, and the terms used (referred to) in this specification are for the purpose of describing the embodiments, and are not intended to limit the present invention.

In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase, and elements and operations referred to as 'comprising (or having)' do not exclude the presence or addition of one or more other elements and operations. .

Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used with the meaning commonly understood by those of ordinary skill in the art to which the present invention belongs.

In addition, terms defined in a commonly used dictionary are not ideally or excessively interpreted unless they are defined.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

Referring to Figure 1, the rail robot-based risk prediction system specialized in the underground tunnel according to the present invention is a rail robot 100, a surveillance camera 200, an image collection unit 300, a robot control unit 400, an artificial intelligence analysis server ( 500) is included. The artificial intelligence analysis server 500 includes an image processing unit 510, a common ball data learning unit 520 equipped with a deep learning-based learning engine 530, a common ball situation diagnosis unit 590, and a common ball situation providing unit 600. include

A plurality of rail robots 100 automatically detect the on-site situation while moving along the rail installed in the underground tunnel. Power of the rail robot 100 is supplied from the rail, and data transmission and reception is possible using power line communication (PLC) or wirelessly. The rail robot 100 is provided with a camera and various sensors for measuring environmental data. Here, various sensors for measuring environmental data include temperature, humidity, oxygen, carbon monoxide, carbon dioxide, nitrogen dioxide, hydrogen sulfide, VOC (volatile organic compounds), and fine dust. Since several rail robots 100 patrol the underground tunnel, each rail robot 100 can communicate with each other by forming a network.

A fixed surveillance camera (CCTV, 200) is installed at the entrance of the underground common area and at the site where important facilities are located. The fixed surveillance camera 200 and the camera mounted on the rail robot 100 shoot and transmit various images such as occupant detection, fall detection, fire/smoke detection, and water leakage.

The image collection unit 300 receives the video image captured by the rail robot 100 and the fixed surveillance camera 200 and transmitted to the artificial intelligence analysis server 500 in real time. The image collection unit 300 uses a network video recorder (NVR) and has real-time control, storage and management functions with images transmitted from a plurality of cameras.

The robot control unit 400 receives a plurality of sensor data and measurement location information measured and transmitted by the rail robot 100 and transmits it to the artificial intelligence analysis server 500 in real time, and controls the rail robot 100 . The environmental data measured using the sensor and the location information where the corresponding environmental data was measured are transmitted together to enable confirmation on the server. The robot control unit 400 may receive data from the rail robot 100 and transmit it to the server 500 as well as receive control information from the server 500 to control the movement of each rail robot 100 .

The image processing unit 510 receives, decodes, and pre-processes the image transmitted by the image collection unit 300 . Referring to FIG. 2 , the image processing unit 510 includes an image processing unit 510a that performs general processing and pre-processing of a video image, and an image processing unit 510b that performs an image stabilization function through real-time image shake correction, which is a specialized function of the present invention. has two of It is preferable that the image image received from the rail robot 100 be pre-processed by the image processing unit 510b including the image stabilization function.

The image processing unit 510a includes an image receiving module 511 for receiving a video image transmitted by the rail robot 100 and the surveillance camera 200, an image decoding module 512 for decoding, and an image preprocessing module for performing pre-processing (513). The image processing unit 510b, which performs an image stabilization function through real-time image shake correction, includes an image receiving module 511 and an image decoding module 512 identically, and includes an image shake correction module 513 for a pre-processing function. do.

Referring to FIG. 3 , the image shake compensation module 513 determines how much each frame of a moving image is moved compared to the previous frame, and corrects the shake by moving the image in the opposite direction to the movement. The process of correcting the shake by the image shake correction module 513 using the feature points is as follows.

After decoding the video frame received from the rail robot 100, ① extracting feature points from each frame image using a deep learning-based feature map extraction engine, ② matching the same feature points among the feature points of each frame image, ③ matching Calculate the mobility of the selected feature point and move the feature point in the opposite direction and size.

The size and direction of movement of each feature can be known by comparing it with the previous frame. It is possible to perform more precise and faster shake correction by extracting the movement direction and size by using deep learning from the movement of the key points generated during the comparison and matching process between the current frame and the previous frame of the shake image. The present invention can extract feature points through a Convolutional Neural Network (CNN).

The common data learning unit 520 stores the image image pre-processed by the image processing unit 510 and the plurality of sensor data transmitted by the robot control unit 400 in a database, and learns with the deep learning-based learning engine 530 . The deep learning-based learning engine 530 is composed of various learning engines to suit each event situation (pedestrian, collapse, fire/smoke, face recognition, water leak, etc.). The common sphere data learning unit 520 classifies the images received from the image collection unit 300, converts them into images, and processes them into learning data through an annotation process. Referring to FIG. 6, as can be seen in (A) entrant face recognition learning data processing process, (B) pedestrian detection, falling learning data processing process, (C) fire and smoke detection learning data processing process, the common data learning unit 520 ) has a model that classifies each received video image and builds separate learning data according to the abnormal event situation.

Referring to FIG. 4 , the deep learning-based learning engine 530 includes an object detection and tracking learning engine 540 , a face detection and recognition learning engine 550 , a leak detection learning engine 560 , and a sensor risk prediction learning engine ( 570), and a fire and smoke detection learning engine 580. In the present invention, face image data, pedestrian detection data, collapse data, fire/smoke data, leak data, etc. were continuously collected and processed for several months to improve detection accuracy. During the collection period (based on 7 months), the image data collected by the rail robot 100 and the surveillance camera 200 is expected to be 4,572,288,000 pieces per year.

Collection amount formula: Collected data (20fps per second) x 14 units (cameras) x 60 seconds x 60 minutes x 24 hours x 210 days x target operation rate (90%) = 4,572,288,000 sheets

Referring to FIG. 5 , the deep learning-based object detection and tracking learning engine 540 measures the distance from the video image to the object, and the object detection module 541 detects the object by estimating the size, width, and width of the detected object. ), the object tracking module 542 that tracks the object by predicting the trajectory of the detected object, the abnormal event detection module 543 that detects the detection of the tracked object, and the collapse of the object tracking module 542 on the tracked object It consists of a marker detection module 544 that recognizes the attached Aruco marker and determines whether an authorized person or a non-authorized person is present. The object detection and tracking learning engine 540 detects and tracks the object using the Yolo v4 algorithm.

The marker detection module 544 may extract a marker image from the video image of the tracked object. Here, it is preferable to use an Aruko marker as the marker, but a Quick Response (QR) code may also be used. In addition, anything that can be recognized through image processing can be used as a marker.

In the case of Aruco markers, the coordinate information of the identification points can be extracted using software such as the Aruco module of OpenCV, and it is possible to accurately identify each marker image using the extracted identification points. Accordingly, it is possible to accurately determine whether an authorized person/non-authorized person is present through the marker detection module 544 .

Referring to FIG. 8 , the object detection and tracking learning engine 540 detects and tracks the pedestrian in the underground tunnel using the Yolo v4 algorithm, recognizes the Aruko marker attached to the pedestrian, and determines whether the person is authorized or not. and generate an alarm.

The deep learning-based face detection and recognition learning engine 550 first detects a face, detects a feature from the detected face, and authenticates the entrant through the process of recognizing the face. Yolo v4 algorithm is used for face detection and face landmark detection, and Deep-Face DNN is used for face recognition. Referring to FIG. 7 , the face detection and recognition learning engine 550 uses the Yolo v4 algorithm to detect faces and feature points of people entering the underground tunnel to recognize faces, and to determine whether an authorized person or a non-authorized person is used to generate an alarm. . The fire and smoke detection learning engine 580 also uses the Yolo v4 algorithm to detect fire and smoke in the same way.

Referring to FIG. 10 , an example in which operator detection, authorization/non-authorization detection, collapse detection, leak detection, fire/smoke detection, etc. are performed using the Yolo v4 algorithm is shown. Differentiating features of the leak detection learning engine 560 and the sensor risk prediction learning engine 570 will be described again below. Pedestrian detection, fall detection, face recognition, and fire/smoke detection are learned by processing avi data photographed and transmitted by the rail robot 100 and the surveillance camera 200 into a txt file (json format), and the rail robot 100 The thermal image image avi data taken and transmitted can be processed and learned into a txt file (json format) and used for fire smoke detection.

As discussed above, the deep learning-based learning engine 530 of the present invention detects an object, tracks the detected object, and determines whether the object behaves abnormally based on the tracking result. As shown in the example of FIG. 10 , the deep learning-based object detection and tracking learning engine 540 may analyze various abnormal behaviors by detecting and tracking the object.

In this embodiment, a CNN (Convolution Neural Network)-based object detection method is used for object detection under various lighting and environmental conditions of the underground pit. YOLOv4 maximized the performance of YOLO by applying various methods to improve the accuracy of deep learning that emerged after YOLOv3. The object detection network used by the object detection module 541 and the object tracking module 542 of this embodiment is based on the YOLOv4 network.

The leak detection learning engine 560 uses an artificial intelligence-based image segmentation convolutional neural network (DeepLab V3+) or Mask R-CNN to detect a leak in an image image.

The artificial intelligence network used by the leak detection learning engine 560 is DeepLab V3+, a deep convolution architecture for image segmentation, and is based on a convolutional neural network (CNN), a network widely used in the image recognition field. In the present invention, about 5,000 pieces of leak detection similar data were collected, and about 45,000 pieces of leak data from underground pits were collected and tested. By uploading the collected similar data and leak data, defining a class for each object, processing the learning data through image segmentation, and creating a learning data file, the DeepLab V3+ based leak detection learning engine 560 was learned.

In addition, the leak detection learning engine 560 may detect a leak in the video image using Mask R-CNN (Regions with Convolution Neural Networks). There is a limit to solving the unpredictable image distortion caused by the external environment and various types of nozzle characteristics through image processing and signal processing methods. In order to overcome this, a deep learning learning method that is robust to image distortion and can recognize various nozzle shapes and extract appropriate features is applied. Among the deep learning learning methods, convolutional neural networks (CNNs) show excellent performance in classifying various images and are widely used in computer vision fields.

Referring to FIG. 11 , the leak detection learning engine 560 inputs the underground tunnel video image captured by the rail robot 100 and transmitted to the Backbone architecture. The candidate region is extracted by applying the feature map extracted through the backbone architecture to the RPN (Region Proposal Network). Among the extracted candidate regions, only the region with the highest score is selected to detect the leak region and perform segmentation.

The common tunnel data learning unit 520 applies a plurality of sensor data collected for each underground common area to the One-Class SVM-based sensor risk prediction learning engine 570 to learn the environmental risk situation of the underground common area, or LSTM (Long Short -term Memory)-based sensor risk prediction learning engine 570 to learn the environmental risk situation of the underground tunnel.

The sensor risk prediction learning engine 570 based on One-Class SVM performs a function of detecting anomalies when normal data and abnormal data are not classified. The model is also trained using only normal samples, and the core idea of this methodology is to establish a discriminative boundary surrounding normal samples, and narrow this boundary as much as possible to consider all samples outside the boundary as abnormal. Through this, it is possible to diagnose the condition of facilities or equipment and prevent the occurrence of downtime or accidents through maintenance before failure occurs.

12, the present invention secures the corresponding sensor data (temperature, humidity, oxygen, carbon monoxide, carbon dioxide, nitrogen dioxide, hydrogen sulfide, VOC (volatile organic compound), fine dust) of 840 underground pits in units of 1 hour, and , was learned by applying the One-Class SVM-based sensor risk prediction learning engine (570).

One-Class SVM-based sensor risk prediction learning engine 570 generates an outlier in the input data, performs learning, detects the outlier, and classifies normal and abnormal. One-Class SVM-based sensor risk prediction learning engine 570 uses a certain range from input data as a data set, and adds randomly generated outliers to it to create a training data set. One-Class SVM-based sensor risk prediction learning engine 570 can predict and warn of environmental risks 1 hour after the learning result.

LSTM (Long Short-term Memory)-based sensor risk prediction learning engine 570 time-series measurement of each sensor value (temperature, humidity, oxygen, carbon dioxide, carbon monoxide, nitrogen dioxide, hydrogen sulfide, VOC (volatile organic compounds), fine dust) Risk prediction is made using the data of Recurrent neural networks (RNNs), which are widely used in time series prediction systems, have a directed cycle in which past events can influence future outcomes. The long short-term memory (LSTM)-based sensor risk prediction learning engine 570 may predict the environmental risk of the underground pit through time series analysis based on the communal environmental sensor data.

Referring to FIG. 9 , ① when the rail robot 100 detects an occupant during patrol, ② moves within a certain distance where the occupant’s face can be recognized through the control of the robot controller 400, and ③ the face of the entrant is detected in a stationary state. The captured video is transmitted in real time. ④ The common sphere situation diagnosis unit 590 applies the face image of the entrant photographed in the stationary state of the rail robot 100 to the deep learning-based learning engine 530 to check whether the entrant is authorized or not, and then operates the robot control unit 400. Through this, the rail robot 100 proceeds to patrol again.

Specifically, the common data learning unit 520 detects the occupants from the real-time video image taken by the rail robot 100 using the deep learning-based object detection and tracking learning engine 540, and the distance to the detected entrants The rail robot 100 is controlled through the robot control unit 400 to move within a certain distance where the face recognition of the person is possible by measuring can be applied to the deep learning-based face detection and recognition learning engine 550 to detect and recognize the face of an occupant and confirm whether or not the entrant is authorized.

In the example of FIG. 9 , an occupant was detected at a distance of about 4 m in the automatic patrol mode. A deep learning-based object detection and tracking learning engine 540 is used for occupant detection. The detailed configuration of the deep learning-based object detection and tracking learning engine 540 has been described in detail above. Then, it moves to a distance of about 3 m or less where face recognition is possible, and transmits an image of the face of the occupant in a stationary state, and a deep learning-based face detection and recognition learning engine 550 is used for face recognition of the entrant. The detailed configuration of the deep learning-based face detection and recognition learning engine 550 has been described in detail above.

The joint sphere situation diagnosis unit 590 is a deep learning-based learning engine 530 in which the joint sphere data learning unit 520 learns the image image transmitted by the image collection unit 300 and the sensor data transmitted by the robot control unit 400 . diagnosing in real time whether access is authorized or not, and the dangerous situation of workers and facilities. The types and functions of the deep learning-based learning engine 530 have been described in detail above, and the deep learning-based learning engine 530 suitable for each situation is applied.

Based on the diagnosis of the common area situation diagnosis unit 590, the common area situation providing unit 600 identifies whether or not an occupant is authorized and the dangerous situation of the underground common area, and displays it in real time. In other words, whether access is authorized/unauthorized, collapse of underground tunnels, fire/smoke, water leakage, and environmental risk situations measured by environmental sensor data are classified into risk symptom stages (normal, warning, dangerous, etc.) and provided to the manager in real time.

Referring to FIG. 13 , the risk prediction method based on the rail robot specialized in the underground tunnel is performed in the following steps. A detailed description of the parts overlapping with those described above has been omitted.

First, while moving along the rail installed in the underground tunnel, the rail robot 100 with a camera and a plurality of environmental sensors installed takes an image, measures a plurality of sensor data, and transmits it in real time (S610).

The fixed surveillance camera 200 installed in the underground pit takes an image and transmits it in real time (S620).

The video image received from the rail robot 100 is corrected for real-time image shake using a deep learning-based feature map extraction engine (S630).

The joint data learning unit 520 stores the video image transmitted by the rail robot 100 and the fixed monitoring camera 200 and the plurality of sensor data transmitted by the robot control unit 400 in a database, and a deep learning-based learning engine to learn (S640).

The joint sphere situation diagnosis unit 590 applies the image image transmitted by the rail robot 100 and the fixed surveillance camera 200 and the sensor data transmitted by the rail robot 100 to a deep learning-based learning engine to determine whether or not the person is authorized. and dangerous situations of workers and facilities are diagnosed in real time (S650).

The common area situation providing unit 600 detects whether or not an occupant is authorized based on the diagnosis and the dangerous situation of the underground common area, and displays it in real time (S660).

In the step (S610) of the rail robot taking an image and transmitting it in real time, when the rail robot detects an occupant during patrol, it moves within a certain distance where the occupant's face can be recognized through the control of the server, and then moves to the occupant's face in a stationary state. In the step of diagnosing and transmitting the captured image in real time (S650), the rail robot applies the face image of the person who is photographed in the stationary state to the deep learning-based learning engine to check whether the person is authorized or not, and then the rail robot Let the patrol proceed again.

In addition, the learning step (S640) uses the deep learning-based object detection and tracking learning engine 540 to detect the occupants from the real-time video image taken by the rail robot 100, and determine the distance to the detected entrants. In the step of measuring and controlling the rail robot to move within a certain distance where face recognition is possible, and diagnosing (S650), the real-time image of the person transmitted by the rail robot 100 is deep learning within a certain distance where face recognition is possible. It is applied to the face detection and recognition learning engine 550 based on detecting and recognizing the face of the entrant to confirm whether or not the entrant is authorized.

The present invention has been described above. The present invention applies an optimal artificial intelligence deep learning algorithm based on CCTV, rail robot image information and sensor information in the underground common area, and develops a high-accuracy risk prediction service and early response safety management service through AI data learning. It is of great significance in that it can automatically identify authorized persons/unauthorized persons and establish a monitoring system that can detect and respond immediately in case of abnormal situations such as collapse, to strengthen security and minimize human casualties.

Although the present invention has been described in detail with reference to preferred embodiments so far, those skilled in the art to which the present invention pertains can practice the present invention in other specific forms without changing the technical spirit or essential features thereof, The embodiments are to be understood in all respects as illustrative and not restrictive.

And, the scope of the present invention is to be specified by the claims to be described later rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts are included in the scope of the present invention. should be interpreted as

100...rail robot
200...Surveillance camera
300...image collection unit
400...Robot control unit
500...Artificial Intelligence Analysis Server
510...image processing unit
511...Video receiving module
512...Video decoding module
513...image preprocessing module, image shake correction module
520...
530... a learning engine based on deep learning
540...Object detection and tracking learning engine
541...object detection module
542...object tracking module
543...abnormal event detection module
544...Marker detection module
550... face detection and recognition learning engine
560...Leak Detection Engine
570...Sensor Hazard Prediction Engine
580...fire and smoke detection learning engine
590...Community Situation Diagnosis Department
600...Community Situation Provision Department

Claims (10)

A rail robot that moves along the rail installed in the underground tunnel and is equipped with a camera and a plurality of environmental sensors;
Fixed surveillance cameras installed in the underground tunnels;
an image collection unit for receiving and transmitting the video image captured by the rail robot and the fixed surveillance camera in real time;
an image processing unit for decoding and pre-processing the image image transmitted by the image collection unit;
a robot control unit that receives and transmits a plurality of sensor data and measurement location information measured and transmitted by the rail robot in real time, and controls the rail robot;
a joint data learning unit for storing the image image preprocessed by the image processing unit and a plurality of sensor data transmitted by the robot control unit in a database, and learning with a deep learning-based learning engine;
Common area for diagnosing in real time whether or not entrants and workers and facilities are dangerous by applying the image image transmitted by the image collection unit and the sensor data transmitted by the robot control unit to the deep learning-based learning engine learned by the common area data learning unit situation diagnosis department; and
Including a common area situation providing unit that identifies whether access is authorized and the dangerous situation of an underground common area based on the diagnosis of the common area situation diagnosis unit and displays it in real time,
When the rail robot detects an occupant during patrol, the rail robot moves within a certain distance capable of recognizing the occupant's face through the control of the robot controller and then transmits an image of the entrant's face in a stationary state in real time,
The common sphere situation diagnosis unit applies the face image of the entrant photographed in the stationary state of the rail robot to a deep learning-based learning engine to check whether the entrant is authorized, and then allows the rail robot to patrol again through the robot control unit. A rail robot-based risk prediction system specialized for underground tunnels.
According to claim 1,
The joint data learning unit
an object detection module for detecting an object by measuring the distance from the video image to the object and estimating the size, width, and area of the detected object;
an object tracking module that tracks the object by predicting the trajectory of the detected object;
an abnormal event detection module for detecting the tracked object and detecting the collapse; and
Including a deep learning-based object detection and tracking learning engine consisting of a marker detection module that recognizes an Aruco marker attached to an object tracked by the object tracking module and determines whether an authorized person or a non-authorized person is A rail robot-based risk prediction system specializing in underground tunnels.
According to claim 1,
The common tunnel data learning unit uses an artificial intelligence-based image segmentation convolutional neural network (DeepLab V3+) or Mask R-CNN to detect a leak in an image image.
According to claim 1,
The image processing unit
After decoding the video frame received from the rail robot, using a deep learning-based feature map extraction engine, feature points are extracted from each frame image, the same feature points are matched among the feature points of each frame image, and the mobility of the matched feature points is calculated. A rail robot-based risk prediction system specializing in underground tunnels, characterized in that it includes an image shake correction module that moves the feature point in the opposite direction and size of the movement direction.
delete According to claim 1,
The communal data learning unit uses a deep learning-based object detection and tracking learning engine to detect occupants from real-time video images taken by the rail robot, measure the distance to the detected occupants, and within a certain distance where the face recognition of the entrants is possible. Control the rail robot through the robot control unit to move,
Underground characterized in that it detects and recognizes the face of an occupant by applying the real-time image of the occupant transmitted by the rail robot to a deep learning-based face detection and recognition learning engine within a certain distance where face recognition is possible to check whether entrants are authorized. A rail robot-based risk prediction system specializing in the common area.
According to claim 1,
The common tunnel data learning unit applies a plurality of sensor data collected for each underground tunnel area to the One-Class SVM-based sensor risk prediction learning engine to learn the environmental risk situation of the underground tunnel, or LSTM (Long Short-term Memory) based A rail robot-based risk prediction system specializing in underground tunnels, characterized in that it learns the environmental risk situation of the underground tunnel by applying it to the sensor risk prediction learning engine.
A step of moving along the rail installed in the underground tunnel, the rail robot installed with a camera and a plurality of environmental sensors to take an image, measure a plurality of sensor data and transmit the data in real time;
A fixed-type surveillance camera installed in the underground communal area capturing an image and transmitting it in real time;
Compensating the video image received from the rail robot in real time using a deep learning-based feature map extraction engine;
Storing the video image transmitted by the rail robot and the fixed surveillance camera and the plurality of sensor data transmitted by the robot controller in a database, and learning with a deep learning-based learning engine;
A step of diagnosing in real time whether a person is authorized or not and a dangerous situation of workers and facilities by applying the video image transmitted by the rail robot and the fixed surveillance camera and the sensor data transmitted by the rail robot to a deep learning-based learning engine; and
Based on the diagnosis, it includes the step of identifying whether access is authorized and the dangerous situation of the underground common area and expressing it in real time,
The step of the rail robot taking an image and transmitting it in real time,
When the rail robot detects an occupant during patrol, the rail robot moves within a certain distance where the occupant's face can be recognized through the control of the server, and then transmits an image of the occupant's face in a stationary state in real time.
The diagnosing step is
Rail robot-based risk prediction based on rail robot specializing in underground tunnels, characterized in that the rail robot performs patrol again after confirming whether or not the person is authorized by applying the face image of the person who is photographed in the stationary state to the deep learning-based learning engine method.
delete 9. The method of claim 8,
The learning step is
Using a deep learning-based object detection and tracking learning engine, the rail robot detects entrants from the real-time video image taken by the rail robot, and measures the distance to the detected entrant so that the rail robot moves within a certain distance where the entrant's face recognition is possible. control,
The diagnosing step is
Underground characterized in that it detects and recognizes the face of an occupant by applying the real-time image of the occupant transmitted by the rail robot to a deep learning-based face detection and recognition learning engine within a certain distance where face recognition is possible to check whether entrants are authorized. A risk prediction method based on a rail robot specialized in a common area.

Images