KR20200123537A - Method for detecting sinkhole using deep learning and data association and sinkhole detecting system using it - Google Patents

Abstract

The present invention relates to a method for detecting a sinkhole. To solve the problems of an existing sinkhole detection method which can apply only a sinkhole detection algorithm, such as a light convolutional neural network (CNN) algorithm, of lower performance having limits incapable of fully securing correctness and reliability in a detection result and tracking a correct location of a sinkhole to issue a warning, the method of the present invention performs a series of processing steps of: performing binary segmentation in a thermal image collected through a thermal image camera based on a temperature difference to extract candidate sinkhole areas; applying CNN transfer learning to classify a real sinkhole from the candidate sinkhole areas; and performing data association to track the classified sinkhole. Accordingly, the sinkhole can quickly and correctly be detected from the image photographed by the thermal image camera and an occurrence probability of a sinkhole can be expected, such that a warning is generated, thereby preventing various kinds of accidents caused by the sinkhole.

Description

A sinkhole detection method using deep learning and data combination, and a sinkhole detection system using the same {Method for detecting sinkhole using deep learning and data association and sinkhole detecting system using it}

The present invention relates to a method for detecting a sinkhole, and more particularly, for example, using a thermal imaging camera, such as an infrared (infrared ray; IR) camera or a far-infrared ray (FIR) camera. A sinkhole detection method using deep learning and data combination, which is configured to determine whether a sinkhole exists in the basement in a non-destructive manner through the photographed thermal image more quickly and accurately compared to conventional methods, and It relates to a sinkhole detection system used.

In addition, the present invention, for example, because only the sinkhole detection algorithm with relatively low performance, such as the light CNN algorithm, is applicable, so that the accuracy and reliability of the detection result is not sufficiently guaranteed, and the accurate location of the sinkhole is tracked to warn In order to solve the problem of the conventional sinkhole detection methods, which had a limitation that cannot be performed, a candidate sinkhole region is extracted by performing binary segmentation by temperature deviation on a thermal image collected through a thermal imaging camera. And, a series of processing steps of classifying the actual sinkhole from candidate sinkhole regions by applying CNN transfer learning, and performing data association to track the classified sinkhole (sinkhole tracking) are performed. Sink using deep learning and data combination, which is configured to detect sinkhole occurrence early and prevent various accidents due to sinkhole by generating an alarm by predicting the possibility of sinkhole occurrence in advance. It relates to a hole detection method.

In addition, the present invention, by using the sinkhole detection method using deep learning and data combination configured as described above, the sinkhole is configured to be quickly and accurately detected in real time from the image captured by the thermal imaging camera. It relates to a sinkhole detection system.

In general, sinkholes are caused by depressions or holes in the ground in the form of a collapsed part of the surface, and most of the sinkholes that occur naturally are caused by karst processes.However, with the recent development of underground exploration technology, various causes It has been found that sinkholes are generated.

That is, it is known that the main causes of sinkholes are dissolution, cover-subsidence, convert-collapse, and humans, and recently, among these causes, construction activities in urban areas have increased. As a result, human-induced sinkholes are rapidly increasing.

In addition, since sinkholes caused by humans cause serious economic losses as well as personal injury, sinkholes are one of many disasters that must be detected in advance to prevent such losses.In particular, sinkholes occurring in residential areas It can be said that it is very important to detect early.

To this end, conventionally, as the most common method of identifying a sink hole, there is a method of using a drill in the area where the sink hole appears, or using a cone penetrometer test (CPT).

However, since these methods are contact inspection methods, there is a problem that may deteriorate the sink hole, and in recent years, a GPR (Ground Penetrating Radar) method is most commonly used to detect the sink hole in a non-destructive manner.

Here, the GPR method has a disadvantage in that the signal may be dispersed due to various very different conditions existing in the basement layer, and as another method, similar to the GPR method, using seismic waves to analyze the probability of a sinkhole appearing in a specific area In order to confirm that there is no sinkhole in a specific area, methods such as Electrical Resistivity Tomography (ERT) and Dynamic Penetration Super Highway (DPSH) are used.

In addition, a ground-based interferometric synthetic aperture radar is used to monitor and estimate large sinkholes. In general, this kind of system is installed at a high position to observe a large area. There is a limit to the fact that the location can affect the monitoring results.

Moreover, airborne laser scanning technology, which detects sinkhole risk using aerial laser scanning data, is generally used as a method of collecting detailed 3D information on surfaces and objects, but 3D data is generally It contains information on the surface shape and sink hole included in the terrain, but since the sink hole does not form a shape that is significantly different from the surrounding environment, it is not easy to detect the sink hole by this method, and the airborne laser method is common. While vegetation, soil color, and surface temperature, which cannot be distinguished by photographs, can be used to represent subsidence areas, there is a disadvantage in that it requires a lot of time and cost to collect data, and the results are greatly affected by the surrounding environment.

Here, as an example of the prior art related to the sinkhole detection apparatus and method for detecting the sinkhole in a non-destructive manner as described above, for example, first, according to Korean Patent Publication No. 10-1857961, a drone and A thermal image acquisition unit for acquiring a thermal image using a thermal imaging camera; A candidate detection unit for extracting a sinkhole candidate region by analyzing the thermal image obtained by the thermal image acquisition unit; And a verification unit for extracting a final sinkhole area excluding an erroneous detection area from the sinkhole candidate area extracted from the candidate detection unit using pattern classifiers, wherein the pattern analyzer includes a light version of the circuit neural network algorithm Sinkhole by combining the classification result obtained using (the light-CNN algorithm) and the classification result obtained by applying the characteristics of the oriented center symmetric-local binary pattern (OCSLBP) to a boosted random forest (BRF) classifier. An automatic sinkhole detection system using a drone and thermal imaging camera configured to extract an area has been proposed.

In addition, as another example of the prior art regarding the sinkhole detection apparatus and method for non-destructively detecting the sinkhole as described above, for example, "Early sinkhole detection using a drone-based thermal camera and image processing" According to the above, as in Patent Publication No. 10-1857961, a thermal image is obtained using a drone and a thermal imaging camera, and a sinkhole is obtained using a light version of the circuit neural network algorithm (the light-CNN algorithm), OCSLBP and BRF. Technical details of a sinkhole detection method using a drone configured to extract an area and a thermal imaging camera have been presented.

As described above, in the related art, various technical contents have been proposed for a sinkhole detection system and method for detecting a sinkhole, but the contents of the prior art as described above have the following problems.

In other words, the contents of the prior art as described above have limitations in detection time and detection range due to the limit of battery capacity due to the configuration of acquiring IR images by attaching an IR camera to a drone, and analysis of images collected in real time In order to do this, since there are various problems in transmitting an image to an external device such as a computer, there is a limitation in that it is impossible to perform image processing and analysis in real time.

In addition, the contents of the prior art as described above have a problem in that the accuracy and reliability of sinkhole detection are not sufficiently guaranteed due to the application of a light CNN algorithm with relatively low performance for shortening the learning time and real-time processing.

More specifically, as high-performance GPUs have recently been developed and a large amount of training data is available, various deep learning-based methods have been proposed that can improve the technology for detecting objects of similar groups such as sinkholes. .

For example, RCNN (Regional based Convolutional Neural Network) has been proposed for fast processing, but it is not fast enough to be implemented in a real-time system, and the You Only Look Once (YOLO) framework is a single convolution of individual components of object detection. It is configured to predict multiple regions and class probabilities for the box by integrating it into a solution network, but the accuracy of the YOLO framework is still affected by small objects appearing in the processed image, so it is difficult to detect sinkholes. There is a problem that it is difficult to apply high-speed RCNN and YOLO.

Here, recently, based on the fact that the surface temperature of the sinkhole region is significantly different from the surface temperature of the surrounding region, a method of using thermal imagery has been proposed to detect the sinkhole.

In more detail, in order to reduce the high cost of the conventional airborne laser method, an unmanned aerial vehicle equipped with a far-infrared (FIR) camera for monitoring a wide area is used to observe the temperature difference over time to record the sinkhole. The optimal time was found, and the candidates for the sinkhole region in the recorded image were segmented using an adaptive binarization method, and then a lightweight convolutional neural network (light CNN) and a boosted random forest (BRF) were used. Thus, potential sinkholes were classified as actual sinkholes.

However, the conventional methods of detecting sinkholes with a thermal imaging camera can give an early warning message about the sinkhole, but the result is not stable because the output image of the thermal imaging camera contains noise, and the exact location of the sinkhole. There is a limitation that it cannot be tracked and warned.

Therefore, as described above, the accuracy and reliability of the detection result is not sufficiently guaranteed due to the application of a sinkhole detection algorithm with relatively low performance, such as the light CNN algorithm, and a limit in which a warning cannot be issued by tracking the exact location of the sinkhole. In order to solve all the problems of the conventional sinkhole detection methods that existed, it is configured to enable fast and accurate image analysis and sinkhole detection in real time through a more advanced high-performance algorithm compared to existing detection devices and methods. It is desirable to provide a new sinkhole detection method and a sinkhole detection system using the same, which is configured to detect sinkhole occurrence early or predict the possibility of sinkhole occurrence to prevent various accidents due to sinkhole in advance. There is still no device or method that satisfies all such demands.

Korean Patent Publication No. 10-1857961 (2018.05.09.)

"Early sinkhole detection using a drone-based thermal camera and image processing", Chunho Chang, Infrared Physics & Technology 78 (2016) 223-232, 2016.08.24.

The present invention is to solve the problems of the prior art as described above, therefore, an object of the present invention is to determine whether or not a sinkhole exists in the basement in a non-destructive manner through a thermal image captured using a thermal imaging camera. It is an object to provide a sinkhole detection method using deep learning and data combination that is configured to make a more rapid and accurate judgment than existing methods, and a sinkhole detection system using the same.

In addition, another object of the present invention is, for example, since only a sinkhole detection algorithm with relatively low performance is applicable, such as the light CNN algorithm, the accuracy and reliability of the detection result is not sufficiently guaranteed, and the exact location of the sinkhole is determined. In order to solve the problem of the conventional sinkhole detection methods, which have limitations in which tracking and warning cannot be performed, a candidate sinkhole is performed by performing binary segmentation by temperature deviation on the thermal image collected through a thermal imaging camera. A series of sinkhole tracking that extracts regions, classifies actual sinkholes from candidate sinkhole regions by applying CNN transfer learning, and performs data association to track the classified sinkholes. Deep learning and data combination, which is configured to prevent various accidents due to sinkholes in advance by detecting sinkhole occurrence early, predicting the possibility of sinkhole occurrence, and generating an alarm by being configured to perform processing steps It is intended to provide a sinkhole detection method using.

In addition, another object of the present invention is to quickly and accurately detect the sinkhole in real time from the image captured by the thermal imaging camera by using the sinkhole detection method using deep learning and data combination configured as described above. It is to provide a sinkhole detection system that is configured to be capable of.

In order to achieve the above object, according to the present invention, processing for detecting a sinkhole from a thermal image captured by a thermal imaging camera is performed by a computer or dedicated hardware, using deep learning and data combination. In the sinkhole detection method, the processing is performed by performing binary segmentation by temperature deviation based on the fact that the temperature of the sinkhole location is lower than the temperature of the surrounding area, and the candidate in the video frame of the input thermal image A candidate sinkhole segmentation step in which a process of outputting a sinkhole position is performed; A sinkhole classification step in which a process of classifying an actual sinkhole among candidate sinkholes selected in the candidate sinkhole dividing step is performed; And a sinkhole tracking step in which a process for tracking the actual sinkholes classified in the sinkhole classification step is performed, and a sinkhole detection method using deep learning and data combination. Is provided.

Here, the candidate sinkhole division step is configured to perform a process of determining a candidate sinkhole position by applying a dual-thresholding method using the following equation.

(Here, I is the input image, (x, y) is the position of a specific pixel, and T _l and T _h are each a predetermined threshold.)

In addition, the step of dividing the candidate sinkhole includes a morphological operation including opening, closing, dilate, and erode to remove edge noise of the divided region. After removing the morphological noise by applying, by applying a connected component analysis using a heuristic filter algorithm, the step of removing shapes similar to sinkholes including trees, buildings, and cars. It characterized in that it is configured.

In addition, in the sinkhole classification step, a process of classifying an actual sinkhole among the candidate sinkholes selected in the candidate sinkhole partitioning step is performed using a convolutional neural network (CNN) transfer learning method. It characterized in that it is configured.

Here, the sinkhole classification step is configured to perform the CNN transfer learning using a ResNet (Residual Network) including ResNet-50.

Moreover, in the sinkhole tracking step, by using data association using a Hungarian algorithm (HA), the actual sinkholes classified in the sinkhole classification step are tracked in the video frame to which the sinkhole belongs. It is characterized in that it is configured to perform a process of allocating to each of the tracks.

Here, in the sinkhole tracking step, when N is the number of tracklets and the distance matrix D is represented by the following equation,

(a) performing a process of subtracting the minimum distance of each row from all items in the same row of the distance matrix D using the following equation;

(b) performing a process of subtracting the minimum distance of each column from all items in the same column of the distance matrix D using the following equation; And

(c) When n is a row or column containing at least one zero, the processing of steps (a) and (b) is repeated until the condition of n = N is satisfied, and the distance matrix D is It is characterized in that it comprises a step of performing a process of assigning each sinkhole to a corresponding tracklet based on a value of 0.

In addition, the detection method is based on an average detection precision (AP), an average recall (AR), and an overlapping threshold using the following equation. It characterized in that it is configured to further include an accuracy evaluation step of evaluating the accuracy of the.

(Here, TP means True Positive, FN means False Negative, FP means False Positive, and FN means False Negative, respectively)

In addition, the detection method further comprises an alarm generating step of transmitting the fact to a preset destination and generating an alarm when it is determined that the sinkhole is found or the possibility of occurrence of the sinkhole is high based on a predetermined criterion, It is characterized in that it is configured to prevent various accidents caused by the sinkhole in advance.

Furthermore, according to the present invention, there is provided a sinkhole detection system, comprising: an image acquisition unit for capturing a thermal image; And an image analysis unit for detecting a sinkhole by performing image analysis on the thermal image obtained by the image acquisition unit or the thermal image input from the outside, wherein the image analysis unit includes the deep learning and data described above. A sinkhole detection system is provided, characterized in that the sinkhole is detected by using a sinkhole detection method using a combination.

As described above, according to the present invention, a candidate sinkhole region is extracted by performing binary segmentation by temperature deviation on a thermal image collected through a thermal imaging camera, and CNN transfer learning is applied. By using deep learning and data combination, which is configured to perform a series of processing steps to classify the actual sinkhole from the candidate sinkhole regions, and perform data association to track the classified sinkhole. By providing a sinkhole detection method, for example, since only a sinkhole detection algorithm with relatively low performance, such as the light CNN algorithm, can be applied, the accuracy and reliability of the detection result is not sufficiently guaranteed, and the exact location of the sinkhole is tracked. Thus, it is possible to solve the problem of the sinkhole detection methods of the prior art, which had limitations in which warnings cannot be issued.

In addition, according to the present invention, it is possible to more quickly and accurately determine whether or not a sinkhole exists in the basement in a non-destructive manner through a thermal image captured using a thermal imaging camera as described above. A sinkhole detection method using deep learning and data combination is provided to prevent various accidents caused by sinkholes by early detection of sinkhole occurrence, predicting the possibility of sinkhole occurrence, and generating an alarm. can do.

In addition, according to the present invention, by using the sinkhole detection method using deep learning and data combination configured as described above, the sinkhole can be detected quickly and accurately in real time from the image captured by the thermal imaging camera. It can provide a sinkhole detection system.

1 is a flowchart schematically showing the overall configuration of a sinkhole detection method using deep learning and data combination according to an embodiment of the present invention.
FIG. 2 is a conceptual diagram schematically illustrating a processing flow of a sinkhole detection method using deep learning and data combination according to an embodiment of the present invention shown in FIG. 1.
Figure 3 is a binary processed by performing binary classification by temperature deviation in a candidate sinkhole segmentation step of a sinkhole detection method using deep learning and data combining according to an embodiment of the present invention with a thermal image taken with a thermal imaging camera. It is a diagram showing each classification image.
4 is a diagram showing a detailed configuration of a heuristic filter algorithm applied to remove shapes similar to sinkholes in a candidate sinkhole segmentation step of a sinkhole detection method using deep learning and data combination according to an embodiment of the present invention.
5 is a diagram showing results of accuracy evaluation for a sinkhole detection method using deep learning and data combination according to an embodiment of the present invention, organized in a table.

Hereinafter, a specific embodiment of a sinkhole detection method using deep learning and data combination and a sinkhole detection system using the same according to the present invention will be described with reference to the accompanying drawings.

Here, it should be noted that the contents described below are only one embodiment for carrying out the present invention, and the present invention is not limited only to the contents of the embodiments described below.

In addition, in the following description of the embodiments of the present invention, for portions that are the same as or similar to the content of the prior art, or that are judged to be easily understood and implemented at the level of those skilled in the art, the detailed description thereof is provided to simplify the description. It should be noted that is omitted.

That is, the present invention can determine whether or not a sinkhole exists in the basement in a non-destructive manner through a thermal image captured using a thermal imaging camera, as described later, compared to conventional methods. The present invention relates to a sinkhole detection method using deep learning and data combination, and a sinkhole detection system using the same.

In addition, the present invention, as described later, is applicable only to the sinkhole detection algorithm with relatively low performance, such as the light CNN algorithm, so that the accuracy and reliability of the detection result is not sufficiently guaranteed, and the accurate location of the sinkhole is tracked and alerted. In order to solve the problem of the conventional sinkhole detection methods, which had a limitation that cannot be performed, a candidate sinkhole region is extracted by performing binary segmentation by temperature deviation on a thermal image collected through a thermal imaging camera. And, a series of processing steps of classifying the actual sinkhole from candidate sinkhole regions by applying CNN transfer learning, and performing data association to track the classified sinkhole (sinkhole tracking) are performed. Sink using deep learning and data combination, which is configured to detect sinkhole occurrence early and prevent various accidents due to sinkhole by generating an alarm by predicting the possibility of sinkhole occurrence in advance. It relates to a hole detection method.

In addition, the present invention, as described later, using the sinkhole detection method using deep learning and data combination as described above, so that the sinkhole can be quickly and accurately detected in real time from an image photographed by a thermal imaging camera. It relates to a configured sinkhole detection system.

Subsequently, with reference to the drawings, a sinkhole detection method using deep learning and data combination according to the present invention and specific details of a sinkhole detection system using the same will be described.

First, referring to FIGS. 1 and 2, FIG. 1 is a flowchart schematically showing the overall configuration of a sinkhole detection method using deep learning and data combination according to an embodiment of the present invention, and FIG. A conceptual diagram schematically showing a processing flow of a sinkhole detection method using deep learning and data combination according to an embodiment of the present invention.

As shown in FIGS. 1 and 2, the sinkhole detection method using deep learning and data combination according to an embodiment of the present invention is broadly divided into a candidate sinkhole for determining a candidate sinkhole position in a video frame of a thermal image. A sinkhole classification step (S20) and a sinkhole classification step of classifying an actual sinkhole among candidate sinkholes selected in the candidate sinkhole segmentation step (S10) and the candidate sinkhole segmentation step (S10) It may be configured to include a sinkhole tracking step (S30) in which a process of tracking the actual sinkholes classified in (S20) is performed.

More specifically, first, in the candidate sinkhole segmentation step (S10), after receiving a thermal image taken with a thermal imaging camera such as an IR camera or an FIR camera, the temperature at the sinkhole location is lower than the temperature of the surrounding area. Based on this, binary segmentation by temperature deviation is performed to output a candidate sinkhole position in a given video frame.

Subsequently, the sinkhole classification step S20 is a sink for classifying an actual sinkhole among the candidate sinkholes selected in the candidate sinkhole partitioning step S10 using a convolutional neural network (CNN) transfer learning method. The function of a hall classifier is performed.

Finally, the sinkhole tracking step (S30) is a sinkhole classification step (S20) using data association using Hungarian Algorithm (HA) in order to deal with the data association problem in real-time sinkhole tracking. The data collected in the sinkhole classification step (S20) is allocated based on the data distribution of the current frame and the previous frame by allocating the actual sinkholes classified in to the tracklets of the video frame to which the corresponding sinkhole belongs. The process of assigning to a tracklet online is performed.

Here, a problem in which an unstable result may be output in the sinkhole classification step S20 can be solved by using the direction voting technique proposed in the present invention as described later.

More specific details of each of the above steps will be described. First, referring to FIG. 3, FIG. 3 shows a thermal image captured by a thermal imaging camera (FIG. 3A) and a temperature deviation in the candidate sinkhole dividing step (S10). Each of the binary segmented images (FIG. 3B) processed by performing binary segmentation by

3A and 3B, the grayscale image of the thermal imaging camera image corresponds to the input of the candidate sinkhole segmentation step (S10), and the segmentation algorithm searches for a cold region in the thermal image to determine the candidate sinkhole. The sensing process is performed.

At this time, since the surrounding environment may have a darker gray color than the sinkhole, a dual-thresholding method is applied, and this double-thresholding process can be expressed as Equation 1 below.

[Equation 1]

Here, in [Equation 1], I is an input image, (x, y) is a location of a specific pixel, and T _l and T _h each represent a threshold value.

In addition, T _l was determined by applying the Otsu algorithm (see Otsu, Nobuyuki. "A threshold selection method from gray-level histograms." IEEE transactions on systems, man, and cybernetics 9.1 (1979): 62-66.), As a result of applying to the data set, it was found that T _h = T _l + 20 provided the best performance.

Here, noise still remains in the binary image even after binary division as described above, and thus, in order to remove edge noise of the divided region, for example, opening, closing, expansion ( Morphological operations such as dilate, erode, etc. can be applied.

In addition, after morphological noise is removed, connected component analysis is applied using an algorithm of a heuristic filter to remove shapes similar to sinkholes, such as trees, buildings, cars, etc. In the segmented image, each block of white pixels is considered a connected component that passes through a heuristic filter.

In more detail, referring to FIG. 4, FIG. 4 is to remove edge noise of a segmented area in the sinkhole segmentation step (S10) of the sinkhole detection method using deep learning and data combination according to an embodiment of the present invention. A diagram showing a detailed configuration of a heuristic filter algorithm applied to remove shapes similar to sinkholes after removing harmful morphological noise.

Here, in [Algorithm 1] shown in FIG. 4, CC _i is the i-th connection element in the image, w _i is the area of CC _i , h _i is the height of CCi, and s _i is the i-th connection in the area of CC _i The number of white pixels of the component, t ₁ to t ₇ denotes a threshold, respectively.

That is, the algorithm of the heuristic filter applied to the embodiment of the present invention may be configured as shown in [Algorithm 1] shown in FIG. 4, and flicker energy classification may be considered to remove noise from the divided image. .

In addition, threshold values used in the heuristic filter may be empirically selected, and the threshold values selected in this embodiment are (t ₁ , t ₂ , t ₃ , t ₄ , t ₅ , t ₆ , t ₇ ,) = ( 5, 100, 0, 2, 5, 75, 1000, 15).

Next, the processing of the sinkhole classification step (S20) by CNN transfer learning will be described. Recently, due to the development of computer technology, deep learning shows the most improved performance in classifying images. Accordingly, in the present invention, In order to distinguish between sinkholes and other objects, a CNN classifier was implemented as follows.

More specifically, image classification using CNN transfer learning consists of two steps: training and prediction, and in the training step, the CNN model is trained using a set of known data images labeled with that type. When the model is trained, it is used to predict the object type of the new image.

However, in order to train a CNN model, a large data set, a large amount of computation, and a computer with excellent performance are required because of a huge number of parameters. Therefore, if the data set is insufficient, it is difficult to train a CNN.

Here, the pre-trained model may be transmitted to work in a category that does not belong to the original data set. Accordingly, in the present invention, a transfer learning method is used to learn the CNN sinkhole classification model.

That is, recently, the most efficiently trained model used in computer vision and deep learning is ResNet (Residual Network) learned from the Image Net Dataset containing millions of images and 1000 object types, and in the present invention, various versions of ResNet Among them, ResNet-50 was selected and applied.

More specifically, ResNet-50 has 49 convolution layers, 1 fully connected layer, and 1 classification layer that calculates the classification score of each image for 1000 object types. ), and in the present invention, in order to classify the sinkhole region, the final layer is replaced with a new layer that has only two object types other than the sinkhole and the sinkhole, and this layer is a backpropagation fine-tuning approach ( It is trained from scratch using a back-propatation fine-tune approach.

In addition, in order to train a CNN for the purpose of classifying sinkhole images in a video frame, one data set for training and another data set for evaluation must be prepared. In the present invention, 7000 data sets are used for training. Sinkhole images and 7000 non-sinkhole images were applied, and 1000 images were applied for each class in the evaluation data set.

Here, the image other than the sinkhole may be an object having a certain shape in a video frame having a surface temperature similar to that of an actual sinkhole, such as a car, a human, or a tree, or may be simply a background image.

When training a CNN using the transfer learning method, a small learning rate was initially used because the weight of the previously trained CNN was relatively good, and there was no need to modify it early, and the optimization process was performed up to 50,000 iterations. The run, and the resulting accuracy of the trained CNN classifier was over 99%.

Next, specific contents of the sinkhole tracking step S30 by the HA algorithm will be described.

After the sinkhole is detected and recognized by the CNN classifier, the problem of tracking is to allocate the sinkhole to each tracklets, which are individual trajectories of consecutive frames. To solve this problem, in the present invention, data association ) For Hungarian Algorithm (HA).

Assuming that N sinkholes are found in a video frame, the problem is how to identify the tracklet to which the sinkhole belongs, and suppose that s _ij is the score between the data distribution of the ith sinkhole and the jth tracklet, This score is calculated using a correlation between the sinkhole image in the tracklet and the detected sinkhole image, that is, if the correlation score is high, it means that the probability that the sinkhole belongs to each tracklet is high.

을 얻고, x_ij는 i 번째 싱크홀과 j 번째 트랙렛의 관계를 정의하며, i 번째 싱크홀이 j 번째 트랙렛의 일부인 경우에만 x_ij = 1, 그렇지 않으면 x_ij = 0 이므로, 따라서

이거나 하나의 검출된 싱크홀은 단지 하나의 트랙 세트에 속하며, 여기서,

, 그리고 N은 트랙렛의 수이다. Distance to Solve Problems Using HA

And x _ij defines the relationship between the i-th sinkhole and the j-th tracklet, and x _ij = 1 only if the i-th sinkhole is part of the j-th _tracklet , otherwise x _ij = 0, thus

Or one detected sinkhole belongs to only one set of tracks, where:

, And N is the number of tracklets.

That is, HA is a method of solving this kind of optimization problem, and can be expressed as the following [Equation 2], whereby the total cost function can be minimized.

[Equation 2]

In addition, the distance matrix of the assignment problem can be expressed as [Equation 3] below.

[Equation 3]

In the first step of the HA, a minimum distance between each sinkhole and a tracklet is found in the following [Equation 4], and then all weights d _ij are subtracted by each minimum weight.

[Equation 4]

Similarly, the second step of HA is a process of subtracting the minimum distance of each column from all items in the same column of the distance matrix by following [Equation 5], and after this step, the distance matrix D contains a value of 0. do.

[Equation 5]

In the third step, if n is a horizontal/vertical line containing at least one zero, if n=N, we can make the assignment x _ij based on the zero value in the distance matrix D, otherwise, the third Repeat the second and third steps until the conditions of the step are met.

Here, since the collected data set is a video captured by a drone, the video is not stable because the sinkhole and other objects are fixed and the camera is moving, and in some cases, the actual sinkhole of the image cannot be detected in the segmentation and classification steps. Cases can arise.

In this case, the previous sinkhole position of the previous frame is used as a small translation distance and is placed in the current frame. In this case, the transformation distance is calculated based on the previous frame and other sinkhole transformations of the current frame.

This conversion correction is performed under the assumption that all sinkholes of the video frame move in the same direction, and this approach helps to solve the problem of missing sinkholes in the detection and recognition process of the system.

Accordingly, as described above, a sinkhole detection method using deep learning and data combination according to an embodiment of the present invention can be implemented, and then, deep learning and data according to an embodiment of the present invention configured as described above can be implemented. The results of verifying the actual performance of the sinkhole detection method using coupling will be described.

More specifically, in the present invention, hardware of CPU: Intel Core i7, GPU: GTX 1070, Ram: 16Gb was used to efficiently perform real-time sinkhole detection, and to evaluate the proposed sinkhole tracking algorithm, the drone The collected video captured in was used.

First, after subdividing the input video into a training data set and a test data set, using the following [Equation 6] and [Equation 7] to evaluate the performance, the average detection precision (AP) and average Recall (Average Recall; AR) was estimated respectively.

[Equation 6]

[Equation 7]

Here, in [Equation 6] and [Equation 7], TP denotes True Positive, FN denotes False Negative, FP denotes False Positive, and FN denotes False Negative, respectively.

As described above, it is possible to know whether the sinkhole detected by the overlapping threshold is FP or TP, and FP is determined by testing all test data overlapping below the overlapping threshold of the actual sinkhole, The TP is determined by testing all test data that overlap above the overlapping threshold of the actual sinkhole.

Further, referring to FIG. 5, FIG. 5 is a diagram illustrating results of accuracy evaluation for a sinkhole tracking algorithm according to an embodiment of the present invention, organized in a table.

In the results shown in Fig. 5, in general, an overlap threshold value of 0.5 is used to evaluate the performance of the tracking method, and as a result of calculating the AR and AP as described above, as shown in Fig. 5, the overlap threshold value It can be seen that the AP and AR scores decrease as this increases.

As described above, the present invention proposes an approach that combines CNN object classifier and HA data association in order to efficiently perform real-time synchole tracking. As a result of the experiment, the proposed method is for AP and AR respectively at an overlap threshold of 0.5. 89% and 88.7% were achieved.

Here, the sinkhole detection method using deep learning and data combination proposed in the present invention does not detect a potential sinkhole candidate in the segmentation step when the input video is not stable, or the input potential candidate is omitted from the training set. In some cases, the performance is poor, but nevertheless, it can be seen that the sinkhole detection method using deep learning and data combination proposed in the present invention can be usefully used in practical fields such as sinkhole search using an infrared camera. have.

In addition, the sinkhole detection method using deep learning and data combination according to the present invention, although not shown, corresponds to a preset destination when a sinkhole is found or it is determined that the possibility of occurrence of a sinkhole is high based on a predetermined criterion. By further including an alarm generation step of transmitting and generating an alarm, it may be configured to prevent various accidents due to the sinkhole in advance.

Above, through the configuration as described above, according to the present invention, a thermal imaging camera and an image input from the outside are received, and an image analysis is performed through the algorithm as described above to detect a sink hole, and an alarm when a sink hole occurs. A sinkhole detection method and sinkhole detection system configured to automatically perform a process for detecting sinkholes in real time using a thermal imaging camera by being configured to execute a series of processing processes occurring by a computer or dedicated hardware Can be easily implemented with a simple configuration and low cost.

Accordingly, as described above, a sinkhole detection method using deep learning and data combination according to the present invention and a sinkhole detection system using the same can be implemented.

In addition, by implementing the sinkhole detection method using deep learning and data combination according to the present invention as described above, according to the present invention, the binary division by temperature deviation on the thermal image collected through the thermal imaging camera ( Binary segmentation) is performed to extract candidate sinkhole regions, CNN transfer learning is applied to classify the actual sinkholes from candidate sinkhole regions, and the classified sinkholes are classified by performing data association. By providing a sinkhole detection method using deep learning and data combination, which is configured to perform a series of processing steps of sinkhole tracking, for example, a sinkhole detection algorithm with relatively low performance, such as the light CNN algorithm. Since only applicable, the accuracy and reliability of the detection result is not sufficiently guaranteed, and the problem of the sinkhole detection methods of the prior art can be solved, which has a limitation in that a warning cannot be issued by tracking the exact location of the sinkhole.

In addition, according to the present invention, it is possible to more quickly and accurately determine whether or not a sinkhole exists in the basement in a non-destructive manner through the thermal image captured using a thermal imaging camera as described above. A sinkhole detection method using deep learning and data combination is provided to prevent various accidents caused by sinkholes by early detection of sinkhole occurrence, predicting the possibility of sinkhole occurrence, and generating an alarm. can do.

Moreover, according to the present invention, by using the sinkhole detection method using deep learning and data combination configured as described above, the sinkhole can be detected quickly and accurately in real time from the image captured by the thermal imaging camera. It can provide a sinkhole detection system.

In the above, the details of the sinkhole detection method using deep learning and data combination according to the present invention and the sinkhole detection system using the same have been described through the embodiment of the present invention as described above. Therefore, the present invention is not limited to the contents described in the present invention, and various modifications, changes, combinations, and substitutions, etc., according to design needs and other various factors by those of ordinary skill in the art to which the present invention pertains. It is natural that it is possible.

Claims (10)

In the sinkhole detection method using deep learning and data combination configured to perform processing for detecting a sinkhole from a thermal image captured by a thermal imaging camera by a computer or dedicated hardware,
The above treatment,
Based on the fact that the temperature of the sinkhole location is lower than the temperature of the surrounding area, binary segmentation based on temperature deviation is performed to output the candidate sinkhole location in the video frame of the input thermal image. A sinkhole segmentation step;
A sinkhole classification step in which a process of classifying an actual sinkhole among candidate sinkholes selected in the candidate sinkhole dividing step is performed; And
A sinkhole detection method using deep learning and data combination, characterized in that it comprises a sinkhole tracking step in which a process of tracking actual sinkholes classified in the sinkhole classification step is performed.
The method of claim 1,
The step of dividing the candidate sinkhole,
A sinkhole detection method using deep learning and data combination, characterized in that the processing of determining a candidate sinkhole location by applying a dual-thresholding method is performed using the following equation.

(Here, I is the input image, (x, y) is the position of a specific pixel, and T _l and T _h are each a predetermined threshold.)
The method of claim 1,
The step of dividing the candidate sinkhole,
In order to remove the edge noise of the segmented area, morphological operation including opening, closing, dilate, and erode is applied to remove morphological noise, Deep learning and data, characterized in that it further comprises the step of removing shapes similar to sinkholes including trees, buildings, and vehicles by applying connected component analysis using a heuristic filter algorithm. A sinkhole detection method using coupling.
The method of claim 1,
The sinkhole classification step,
Deep learning and data, characterized in that a process of classifying an actual sinkhole among candidate sinkholes selected in the candidate sinkhole segmentation step is performed using a convolutional neural network (CNN) transfer learning method. A sinkhole detection method using coupling.
The method of claim 4,
The sinkhole classification step,
A sinkhole detection method using deep learning and data combination, characterized in that configured to perform the CNN transfer learning using ResNet (Residual Network) including ResNet-50.
The method of claim 1,
The sinkhole tracking step,
Processing of allocating the actual sinkholes classified in the sinkhole classification step to tracklets of the video frame to which the sinkhole belongs, using data association using Hungarian Algorithm (HA) A sinkhole detection method using deep learning and data combination, characterized in that configured to perform.
The method of claim 6,
The sinkhole tracking step,
When N is the number of tracklets and the distance matrix D is expressed by the following equation,

(a) performing a process of subtracting the minimum distance of each row from all items in the same row of the distance matrix D using the following equation;

(b) performing a process of subtracting the minimum distance of each column from all items in the same column of the distance matrix D using the following equation; And

(c) When n is a row or column containing at least one zero, the processing of steps (a) and (b) is repeated until the condition of n = N is satisfied, and the distance matrix D is A sinkhole detection method using deep learning and data combination, comprising the step of performing a process of assigning each sinkhole to a corresponding tracklet based on a value of 0.
The method of claim 1,
The detection method,
Accuracy evaluation to evaluate the accuracy of the sinkhole detection result based on the average detection precision (AP), average recall (AR), and overlapping threshold using the following equation A sinkhole detection method using deep learning and data combination, characterized in that it further comprises a step.

(Here, TP means True Positive, FN means False Negative, FP means False Positive, and FN means False Negative, respectively)
The method of claim 1,
The detection method,
When a sinkhole is found or it is determined that the possibility of occurrence of a sinkhole is high based on a predetermined criterion, the fact is transmitted to a preset destination and an alarm generating step is further included to prevent various accidents due to the sinkhole. Sinkhole detection method using deep learning and data combination, characterized in that configured to prevent the.
In the sinkhole detection system,
An image acquisition unit for photographing a thermal image; And
And an image analysis unit for detecting a sinkhole by performing image analysis on the thermal image obtained by the image acquisition unit or the thermal image input from the outside,
The image analysis unit,
A sinkhole detection system configured to detect a sinkhole by using the sinkhole detection method using deep learning and data combination according to any one of claims 1 to 9.

Images