KR20230085299A - System and method for detecting damage of structure by generating multi-scale resolution image - Google Patents

Abstract

According to an embodiment of the present invention, a structure damage detection method performed by a structure damage detection system using multi-resolution image generation includes the steps of: (a) receiving an image of a structure captured with a vision camera; (b) generating a high-resolution image which satisfies the minimum standard from the received image; (c) dividing the high-resolution image into multi-resolution images of a first scale to an N^th scale and generating damage detection result images for each scale through a damage detection network; (d) stacking each damage detection result image on the first scale damage detection result image, and calculating a class-specific statistical value for pixel-level damage classification based on the stacking result; and (e) determining the class with the largest statistical value as a final label of a high-resolution image detection result pixel to derive a final classification result image. The present invention automates the derivation of final damage detection result.

Description

Structure damage detection system and method using multi-resolution image generation

The present invention relates to a structure damage detection system and method using multi-resolution image generation.

In general, when evaluating the exterior damage of a large facility, it is difficult to evaluate the entire structure due to limitations in human access, so digital image data is acquired and utilized through an unmanned body such as a drone or robot equipped with a vision camera. The amount of damage evaluation for the acquired data is very large, so the existing method based on expert manpower not only reduces objectivity, but also causes a large loss of time and economic cost, so an artificial intelligence-based automated detection network is used. Next, there is a problem in that the resolution of digital image data acquired in the process of quantitatively evaluating the detected damaged area is insufficient for precise evaluation criteria.

Automated damage evaluation through such unmanned body-based image data can be classified into two types according to the process.

First, the drone-based vision data acquisition method uses a drone equipped with a vision camera to acquire digital images in places that are difficult to access by manpower. Although it has the advantage of being able to acquire digital images of parts that are difficult to access by personnel without limiting the shooting angle, there is a high possibility that the quality of data will be deteriorated due to problems such as vibration of the unmanned aerial vehicle itself or motion blur during movement. In particular, for flight safety reasons, it is necessary to maintain a safe separation distance of several meters (m) between the target structure and the unmanned aerial vehicle, so the resolution of the acquired image may be less than the resolution required to detect a meaningful minimum standard value in the quantitative evaluation of damage, This can make precise evaluation difficult.

Next, the artificial intelligence-based damage detection network utilization method is a method of automatically detecting damage through an artificial intelligence-based network in order to process a large amount of data acquired from large structures. Unlike conventional manpower-based damage assessment, it has the advantage of being able to evaluate damage by efficiently processing data in a short period of time, but has a fatal problem in that the detection result varies depending on the resolution size of the input image. In particular, it is difficult to detect damage when the resolution is too large or insufficient, and it is difficult to optimize the resolution because the size of the resolution that guarantees an optimal detection result for each image is different depending on the shooting environment.

In this way, digital images of the area of interest are acquired through a drone equipped with a vision camera for damage assessment of large facilities that are difficult to access by human resources. At this time, in the case of drones, it is difficult to take close-up shots due to securing a safe distance, and due to the limited resolution of vision cameras, the acquired digital images often lack resolution for precise damage detection and quantitative evaluation. In addition, the detection rate of various damage detection networks based on artificial intelligence used for the purpose of automated damage detection varies depending on the resolution of the input image.

Republic of Korea Patent Registration No. 10-1780057 (Title of Invention: Method and Apparatus for Restoring High Resolution Images)

A structure damage detection method using a structure damage detection system according to an embodiment of the present invention generates a high-resolution image that satisfies the minimum standard value for quantitative evaluation of exterior damage of a facility, and then corrects and divides the image into multi-scale resolutions, The damage detection result is stacked to correspond to the detection result of the high-resolution image. It is intended to automate the derivation of the final damage detection result by calculating a statistical value for each pixel class based on the stacking result and reflecting the class with the largest value as a new label in the detection result pixel of the high-resolution image.

A structure damage detection method performed by a structure damage detection system through multi-resolution image generation according to an embodiment of the present invention includes the steps of (a) receiving an image of a structure captured by a vision camera; (b) generating a high-resolution image that satisfies the minimum standard value from the received image; (c) dividing the high-resolution image into multi-resolution images of first to Nth scales, and generating damage detection result images for each scale through a damage detection network; (d) stacking each damage detection result image on a first scale damage detection result image, and calculating a statistical value for each class for damage classification in pixel units based on the stacking result; and (e) deriving a final classification result image by determining a class having the largest statistical value as a final label of a detection result pixel of the high resolution image.

A structure damage detection system using multi-resolution image generation according to another embodiment of the present invention includes a memory storing a program for providing a structure damage detection method; and a processor that executes a program stored in a memory, wherein the processor receives an image of a structure photographed by a vision camera according to the execution of the program, generates a high-resolution image that satisfies a minimum standard value from the received image, and generates a high-resolution image Divide into multi-resolution images of the first scale to the Nth scale, generate damage detection result images for each scale through the damage detection network, and stack each damage detection result image on the damage detection result image of the first scale, Based on the stacking result, statistical values for each class are calculated for damage classification in pixel units, and the class with the largest statistical value is determined as the final label of the pixel as a result of detection of the high-resolution image to determine the damage of the structure that derives the final classification result image. detect

According to the structure damage detection method by the structure damage detection system according to an embodiment of the present invention, after acquiring a digital image through a vision camera mounted on a drone, it is determined whether or not the detection standard is suitable, and a suitable high-resolution image is based on a deep learning network. After restoring, the effect of automatically detecting and stacking various damages of the generated multi-resolution image to supplement the damage detection ability of the AI-based network, which is variable according to the size of the input image through statistical techniques, to automate the derivation of the final detection result. there is

1 is a diagram showing the configuration of a structure damage detection system according to an embodiment of the present invention.
Figure 2 is a schematic diagram for explaining a structure damage detection method using a structure damage detection system according to an embodiment of the present invention.
Figure 3 is a schematic diagram showing a high-resolution image generation process using a structure damage detection system according to an embodiment of the present invention.
4 is a schematic diagram showing a multi-resolution image generation and damage detection process using a structure damage detection system according to an embodiment of the present invention.
5 and 6 are schematic diagrams showing a damage classification process based on statistical decision making using a structure damage detection system according to an embodiment of the present invention.
7 is a schematic diagram showing a process of generating a final damage detection result using a structure damage detection system according to an embodiment of the present invention.
8 and 9 compare image damage detection results for each resolution for experimental verification through the structure damage detection system according to an embodiment of the present invention.
10 is a flowchart illustrating a structure damage detection method using multi-resolution image generation according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice it. However, the present invention may be embodied in many different forms and is not limited to the embodiments set forth herein. And, in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be “connected” to another part, this includes a case where it is “directly connected” and a case where it is “electrically connected” with another element interposed therebetween. In addition, when a part "includes" a certain component, it means that it may further include other components without excluding other components unless otherwise stated.

In this specification, 'unit' includes a unit realized by hardware or software, or a unit realized using both, and one unit may be realized using two or more hardware, and two or more units may be implemented. The above units may be realized by one piece of hardware. Meanwhile, '~unit' is not limited to software or hardware, and '~unit' may be configured to be in an addressable storage medium or configured to reproduce one or more processors. Therefore, as an example, '~unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Functions provided within components and '~units' may be combined into smaller numbers of components and '~units' or further separated into additional components and '~units'. In addition, the components and '~units' may be implemented to regenerate one or more CPUs in the device.

Hereinafter, a structure damage detection system (hereinafter, referred to as a 'structure damage detection system') using multi-resolution image generation according to an embodiment of the present invention will be described in detail with reference to the drawings.

1 is a diagram showing the configuration of a structure damage detection system according to an embodiment of the present invention.

Referring to FIG. 1 , a structure damage detection system 100 according to an embodiment of the present invention includes a communication module 110, a memory 120, a processor 130, and a database.

The communication module 110 interworks with the communication network to provide a communication interface capable of transmitting and receiving images captured by the vision camera to the structure damage detection system 100, and in particular, plays a role in transmitting and receiving data between the vision camera device and the management server. can be done Here, the communication module 110 may be a device including hardware and software necessary for transmitting and receiving a signal such as a control signal or a data signal with another network device through a wired or wireless connection.

For example, a vision camera may be mounted and used on an attachable unmanned vehicle such as an unmanned vehicle, a climbing robot, or a drone to photograph a target structure.

The memory 120 may have a program recorded thereon to provide a structure damage detection method. Also, the memory 120 may temporarily or permanently store data processed by the processor 130 . Here, the memory 120 may include volatile storage media or non-volatile storage media, but the scope of the present invention is not limited thereto.

A structure damage detection program is stored in the memory 120 . The memory 120 stores various types of data generated during the execution of an operating system for driving the structure damage detection system 100 or a structure damage detection program.

The processor 130 executes the program stored in the memory 120, and performs the following processing according to the execution of the structure damage detection program.

The processor 130 receives an image of a structure captured by a vision camera through the communication module 110, generates a high-resolution image satisfying a minimum standard value from the received image, and converts the high-resolution image to a first scale to a second scale. Dividing into N-scale multi-resolution images, generating damage detection result images for each scale through a damage detection network, stacking each damage detection result image on the damage detection result image of the first scale, and pixel based on the stacking result Statistical values for each class for unit damage classification are calculated, and the class with the largest statistical value is determined as the final label of the pixel as a result of detection of the high-resolution image to detect damage to the structure that derives the final classification result image. Each specific step of the structure damage detection process according to the execution of the program will be described later with reference to FIGS. 2 to 7 .

The processor 130 may include any type of device capable of processing data. For example, it may refer to a data processing device embedded in hardware having a physically structured circuit to perform a function expressed as a code or command included in a program. As an example of such a data processing device built into hardware, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific ASIC integrated circuits), field programmable gate arrays (FPGAs), etc., but the scope of the present invention is not limited thereto.

The database may be a medium that organically combines and stores commonly required data to perform the structure damage detection method. The database may store images and learning data related to structure damage. Such a database may be included as a component separate from the memory 120 or may be built in a partial area of the memory 120 .

Figure 2 is a schematic diagram for explaining a structure damage detection method using a structure damage detection system according to an embodiment of the present invention.

Referring to FIG. 2 , the processor 130 may include detailed modules that perform various functions according to the execution of programs stored in the memory 120 . The detailed module includes a high-resolution image generation unit 140, a multi-resolution image generation and damage detection unit 150, a damage classification unit 160, and a damage detection result generation unit 170.

First, the processor 130 receives an image of a structure photographed by a vision camera, and generates a high-resolution image that satisfies a minimum standard value from the received image.

For example, a structure image may be acquired by photographing an exterior area of a target bridge that is difficult to access by manpower through a vision camera mounted on a drone (UAV). The original image obtained at this time is a precise quantitative evaluation of the detection result of the external damage of the structure due to camera performance problems such as camera performance problems such as field of view (FOV), resolution, and lens of the vision camera and shooting environment problems such as illumination of the shooting location, shooting angle, shooting separation distance, etc. may not be suitable for To solve this, a minimum criterion for the physical size of one pixel defining the damaged area is required. For example, the minimum meaningful crack thickness proposed by the Ministry of Land, Infrastructure and Transport to evaluate the safety of structures is 200 μm.

Therefore, the minimum standard value of the present invention defines the minimum crack thickness of 200 μm as the minimum physical size of one pixel. That is, the resolution of an image that satisfies the criterion may be defined as a high-resolution image. For example, in the case of a 1K resolution image, the pixel resolution for quantitative evaluation may be insufficient depending on the camera working distance, and this cannot be defined as a high-resolution image.

Accordingly, in order to calculate the minimum resolution size of the high-resolution image suitable for damage evaluation, Equation 1 is derived based on the pinhole camera model, and the minimum resolution (minimum reference value) of the high-resolution image is calculated.

<Equation 1>

Here, R _P is the minimum pixel resolution of the high-resolution image, W _D is the working distance, C _S is the size of the camera sensor, and C _FL is the focal length of the camera.

If the pixel resolution of the input image is smaller, it must be recreated to a suitable size before damage detection for precise quantitative evaluation of damage. A scale factor (λ) for regenerating an input image to satisfy the value can be calculated by Equation 2 below.

<Equation 2>

Here, λ is a scale factor for increasing the resolution of the input image, and R _I is the resolution of the input image.

Figure 3 is a schematic diagram showing a high-resolution image generation process using a structure damage detection system according to an embodiment of the present invention.

Referring to FIG. 3 , the high-resolution image generating unit 140 may omit this step by calculating 1 as the λ value when the input image ^VR of the algorithm corresponds to the resolution criterion suitable for quantifying the minimum damage. On the other hand, when it is determined that the resolution is insufficient when the value of λ is greater than 1, the high-resolution image generating unit 140 generates a high-resolution digital image that meets the criteria.

Specifically, the high-resolution image generating unit 140 extracts ?汰? from the received image ^VR ? ?汰? to extract shallow features. Feature module, deep feature module that extracts deep features from the received image, extracted ?汰? It includes an upscale module that upscales feature data of an image constructed using features and deep features to a high-resolution size, and a restoration module that generates a high-resolution image from feature data of the upscaled image.

As the first part, ?汰? The feature means a feature of a low-resolution (LR) image that is the input image ^VR . ?汰? A feature module consists of a single convolutional layer. In this case, the convolution layer is composed of 64 kernels of size 3 x 3, and a stride of 1 may perform feature extraction of the LR image.

As the second part, deep features refer to high-frequency information composed of lines or edges, which are the biggest difference between LR images and high-resolution (HR) images paired with them. To this end, the deep feature module uses the RIR structure, and is composed of 10 residual groups each consisting of 20 residual blocks and one convolution layer. Exemplarily, the residual groups of the RIR structure have a long skip connection, and a plurality of residual blocks constituting the inside of the residual group are connected through a short skip connection to have a very deep structure. Skip connection passes through each residual group, residual block, and the last layer, the convolution layer, and outputs deep features and input ? By adding the features, only differences from the LR image and the HR image are additionally learned. Residual learning is defined as learning the high-frequency information more effectively by the main network by bypassing the low-frequency information constituting most of the LR image through the skip connection. At this time, each convolution layer has the same shape as the single convolution layer of the first part. In order to more efficiently extract the high-frequency information of the image, a Channel Attention (CA) mechanism is performed inside each residual block. The CA mechanism extracts the average value of each channel constituting the image, and based on this, improves discrimination learning ability by focusing on more useful channels. Inside each residual block, CA mechanisms can be performed through Global Average Pooling, convolution, ReLU, and sigmoid layers.

As the third part, the upscaling module is a step of expanding feature data of an image that has passed through each residual block and residual group to a high resolution size to be finally performed. It is composed of a deconvolution layer and consists of 256 kernels of size 3 x 3, with a stride of 1, and the size of each pixel can be expanded by λ times calculated in Equation 2 above.

As the fourth part, the reconstruction module, which is generated as a high-resolution image (V ^SR ) through a single convolution layer, consists of three kernels of size 3 x 3, and a high-resolution image (V ^SR ) for damage detection with a stride of 1 is generated. can

4 is a schematic diagram showing a multi-resolution image generation and damage detection process using a structure damage detection system according to an embodiment of the present invention.

The multi-resolution image generation and damage detection unit 150 divides the high-resolution image (V ^FI ) into multi-resolution images of a first scale (1 ^st Scale) to an N-th scale (N ^th Scale), and the damage detection network 151 Through this, a damage detection result image (V ^D ) for each scale is generated.

Illustratively, a high-resolution image (V ^SR ) generated by the high-resolution image generating unit 140 or a high-resolution image that satisfies the minimum standard among the original images of the vision camera is used to generate a multi-resolution image and damage the damage detection unit 150 for detecting damage. It may be input as the final image (V ^FI ).

)로 생성할 수 있다. 이때 제1 스케일은 입력 이미지인 고해상도 이미지 크기, 제N 스케일의 이미지는 손상 검출 네트워크(151)의 학습 이미지 크기로 정의하며, 다음 단계에서 진행될 확률적 판단의 신뢰도를 확보하기위해 제1 스케일 내지 제N 스케일은 최소 200 단계 이상의 스케일로 구성된 다중 해상도 이미지로 생성되는 것이 바람직하다.The multi-resolution image generation and damage detection unit 150 converts the input high-resolution image (V ^FI ) to a multi-resolution image (of a first scale to an N-th scale) based on a bicubic interpolation algorithm.

) can be created. At this time, the first scale is defined as the size of the high-resolution image, which is the input image, and the image of the Nth scale is defined as the size of the training image of the damage detection network 151. The N scale is preferably created as a multi-resolution image consisting of at least 200 scales.

)를 DCNN(Deep convolution neural network)에 입력하여, 낮은 레벨(Low-level) 특징을 추출하고, 다중 해상도별 생성된 특징 맵을 누적하여 제1 특징 맵을 생성하고, 제1 특징 맵과 낮은 레벨 특징을 누적하여 제2 특징 맵을 생성하고, 제2 특징 맵의 특징을 각 스케일별 픽셀 단위의 손상 라벨로 분류하고, 손상 검출 네트워크(151)의 학습 클래스별 손상 검출 결과로서 업샘플링 층을 통해 손상 검출 결과 이미지(

)를 생성할 수 있다.Then, the multi-resolution image generation and damage detection unit 150 is a multi-resolution image (

) into a deep convolution neural network (DCNN) to extract low-level features, generate a first feature map by accumulating feature maps generated by multiple resolutions, and generate a first feature map and a low-level feature map. The features are accumulated to generate a second feature map, the features of the second feature map are classified into damage labels in units of pixels for each scale, and as a damage detection result for each learning class of the damage detection network 151, through an upsampling layer. Damage detection result image (

) can be created.

Illustratively, the damage detection network 151 may be constructed by performing transfer learning to make DeepLab v3+[1], which is a kind of artificial intelligence network, suitable for damage detection.

)를 다운샘플링하여 손상 특성을 추출하는 엔코더(Encoder)와 손상 특성을 업샘플링하여 픽셀의 다중 라벨(Label)을 분류하는 디코더(Decoder)로 구성된다. 여기서 CNN 기반 네트워크인 Resnet-50 네트워크를 엔코더와 디코더의 콘볼루션 레이어로 사용한다. The damage detection network 151 is an input image (

) is composed of an encoder that extracts damage characteristics by downsampling and a decoder that classifies multiple labels of pixels by upsampling the damage characteristics. Here, the Resnet-50 network, a CNN-based network, is used as the convolutional layer of the encoder and decoder.

)를 아트러스 컨볼루션(Atrous convolution)을 통과시켜 특징 맵(Feature map)을 형성하고, 아트러스 컨볼루션은 디코더에서 사용되는 낮은 레벨(Low-level)의 특징들을 추출한다. 여기서 아트러스 컨볼루션은 기존의 컨볼루션과 달리 필터(Filter) 사이에 몇 개의 간격을 두어, 파라미터(Parameter)의 수를 늘리지 않고 수용 필드(Receptive field)를 증가시켜 고해상도의 특징 맵을 생성하여 학습에 매우 효율적이다. The encoder is an input image (

) is passed through Atrous convolution to form a feature map, and Atrous convolution extracts low-level features used in the decoder. Unlike conventional convolution, here, atrus convolution puts several gaps between filters, increases the receptive field without increasing the number of parameters, and creates high-resolution feature maps for learning. very efficient for

In addition, the encoder generates a feature map of an input image through a deep convolution neural network (DCNN), and accumulates multi-resolution feature maps through ASPP (Atrous spatial pyramid pooling) composed of a convolution layer and a pooling layer to generate a first feature. create a map Also, the first feature map may be upsampled 4 times. The first feature map generated here is an accumulated form of results passing through filters of various sizes, and improves detectability for various scales of the object (structure) to be detected.

Next, the decoder accumulates the second feature map by combining the 4-times upsampled first feature map and low-level features extracted from the encoder through DCNN. As a result, in the combined second feature map, high-frequency features such as edges and lines may be emphasized through a 3×3 convolutional layer.

)로부터 추출된 특징은 각 스케일별 픽셀 단위의 손상 라벨로 분류될 수 있다. 일 예로, 손상 라벨은 크랙, 크랙 같은(Non-crack), 엣지(Non-crack) 및 배경을 포함한다.That is, multi-resolution images (

) can be classified as damage labels in units of pixels for each scale. As an example, damage labels include crack, crack-like (Non-crack), edge (Non-crack), and background.

)로서 업샘플링 층을 통해 생성된다. Finally, the feature of the damage label is classified in pixel units for each corresponding region, and the damage detection result for each learning class of the damage detection network 151 is the damage detection result image (

) is generated through the upsampling layer.

5 and 6 are schematic diagrams showing a damage classification process based on statistical decision making using a structure damage detection system according to an embodiment of the present invention.

)를 도출할 수 있다.Referring to FIG. 5, the damage classification unit 160 stacks each damage detection result image (V ^D ) on a damage detection result image (V ^D ) of a first scale (1 ^st Scale), and based on the stacking result, in pixel units. The final classification result image (

) can be derived.

Specifically, as shown in FIG. 5, the damage classification unit 160 upsamples the damage detection result image (V ^D ) generated for each scale (1 ^st Scale - N ^th Scale) to the same size as the maximum resolution. there is. At this time, the upsampling method uses a bicubic interpolation algorithm, and N damage detection result images (V ^D ) upsampled to the same resolution are damage detection result images (V ^D ) of the 1st scale. can be stacked on

)를 생성할 수 있다. That is, the damage classification unit 160 stacks the N damage detection result images V ^D on the damage detection result image V ^D of the first scale ( ^1st Scale), and accumulates results for each class in pixel units. image(

) can be created.

)는 손상 검출 결과 이미지(V^D)의 픽셀 단위로 분류된 손상라벨을 포함할 수 있다. 이때, 손상 라벨 중에서 배경의 경우, 손상에 해당하지 않으므로, 손상 분류를 위한 클래스에서 제외한다. 따라서, 손상 클래스는 크랙, 크랙 같은(Non-crack) 및 엣지(Non-crack)로 구성될 수 있다.For example, the cumulative result image (

) may include damage labels classified in pixel units of the damage detection result image V ^D . At this time, the background among the damage labels is excluded from the damage classification class because it does not correspond to damage. Accordingly, damage classes may consist of cracks, crack-like (non-crack) and edge (non-crack).

)는 수식3에 의해 산출된다. Cumulative result image generated for each class label (ℓ) (

) is calculated by Equation 3.

<Equation 3>

는 누적 결과 이미지의 픽셀이고, ℓ 은 네트워크 학습 클래스의 손상 라벨로 정의하며 해당 네트워크에서 1은 크랙(Crack), 2는 크랙 같은(Crack-like), 3은 엣지(Edge)로 정의한다.here

is the pixel of the accumulated result image, and ℓ is defined as a damage label of the network learning class. In the network, 1 is defined as crack, 2 as crack-like, and 3 as edge.

)를 기초로 누적 횟수에 대한 확률값을 도출하여 픽셀 단위 이미지(

)를 생성할 수 있다. 이때 픽셀별 확률값은 수식4를 통해 산출될 수 있다.Referring to FIG. 6 , the damage classification unit 160 generates cumulative result images for each class (

) by deriving a probability value for the number of accumulations based on the pixel unit image (

) can be created. At this time, the probability value for each pixel may be calculated through Equation 4.

<Equation 4>

)에서 동일한 위치(p, q)의 픽셀을 기준으로 각각의 라벨에 해당하는 확률값을 비교하여 해당 픽셀의 최종 라벨값 L(p, q)을 결정할 수 있다. 크랙 같은과 엣지는 논-크랙(Non-crack) 유형으로 손상으로 확률값을 더해 L_NC(p, q)로, 크랙(Crack)에 대한 확률값은 L_Cr(p, q)로 정의한다. 이어서, L_NC(p, q)와 L_Cr(p, q)값을 비교하여 L_Cr(p, q)의 값이 더 크거나 같은 경우 L(p, q)은 크랙(C)으로 결정하고, L_Cr(p, q)의 값이 작은 경우, L(p, q)은 논-크랙(NC)으로 결정되며, 최종 분류 결과 이미지(V^C.R)가 생성될 수 있다. As shown in FIG. 6, the damage classification unit 160 generates a pixel unit image (

), the final label value L(p, q) of the corresponding pixel may be determined by comparing probability values corresponding to respective labels based on pixels at the same location (p, q). Cracks and edges are non-crack types, and the probability value is added as damage to L _NC (p, q), and the probability value for crack is defined as L _Cr (p, q). Subsequently, L _NC (p, q) and L _Cr (p, q) values are compared, and if the value of L _Cr (p, q) is greater than or equal to, L (p, q) is determined as crack (C) , L _Cr (p, q) is small, L (p, q) is determined to be non-crack (NC), and a final classification result image (V ^CR ) may be generated.

7 is a schematic diagram showing a process of generating a final damage detection result using a structure damage detection system according to an embodiment of the present invention.

Referring to FIG. 7 , the damage detection result generating unit 170 generates a binary image V ^B through an image processing process in order to assign new label values to pixels of the damage detection result image V ^D , and the final classification result. A final damage detection result image V ^F may be generated by multiplying the image V ^CR by the binary image V ^B in units of pixels.

In this case, the final damage detection result image V ^F is a final damage detection result generated through a multi-resolution image stacking algorithm using an SR network. That is, the crack damage detection ability is improved compared to the damage detection result of the input image, and quantitative evaluation of the minimum thickness of 200 μm is possible.

Hereinafter, a method for learning and constructing a network for experimental verification of a structure damage detection system according to an embodiment of the present invention will be described with reference to FIGS. 8 and 9 .

8 and 9 show comparison of image damage detection results for each resolution for experimental verification through the structure damage detection system according to an embodiment of the present invention.

Exemplarily, in the present invention, in order to build a high-resolution image network more suitable for detecting damage on the exterior of a structure, a learning dataset for image resolution and a learning dataset for damage detection are respectively configured and learned. First, the training dataset for high resolution uses the same 800 training images of the DIV2K dataset, which is a widely known training dataset in single image super-resolution, and uses real cities as a verification dataset, We used the Urban100 dataset consisting of 100 high-resolution photos of architectural structures. A total of 800 training datasets in various sizes of HD size 1,280 x 720 or more are augmented to 8 through image flip and 90, 180, and 270 degree image rotation, resulting in 6,400 training datasets, an increase of 8 times the total 800 training data. was used in The learning data was used for network learning by repeating a total of 1,000 times, and 16 batch sizes were selected as the number of data used for each learning step.

Next, as a learning dataset for damage detection, a new dataset related to exterior damage is newly configured and network learning is performed. For learning, 1,756 crack damage images were learned by fixing the size to 360 x 480. At this time, a region is defined using a polygon tool for damage learning. Damage existing in an input image may be automatically detected using a network for which damage learning has been completed, and a resultant image, which is an output image, may be acquired.

8 shows image damage detection results for each resolution, FIG. 8(a) is a low-resolution image, FIG. 8(b) is a low-resolution image damage detection result, FIG. 8(c) is an original image, and FIG. Original image damage detection result, Fig. 8(e) is a high-resolution image, and Fig. 8(f) is a high-resolution image damage detection result.

Referring to FIG. 8, for experimental verification of the proposed network, an experiment was performed using Jangdeokgyo Bridge in Gangneung as a test bed. The image was acquired for about 2 minutes at 30 fps using a DJI PHANTOM 4 Pro drone equipped with a vision camera, and the size of the acquired image is 1,920 × 1,440. In order to verify whether the damage detection result of the constructed network differs depending on the image resolution, the original image is regenerated using a bicubic interpolation algorithm with a low-resolution image of 480 × 360 and a high-resolution image of 3,840 × 2,880, respectively.

8(b), 8(d), and 8(f) show crack detection results for each resolution, and crack detection results are all different depending on low resolution, original, and high resolution images even though they are the same image, and the crack detection ability of the network affects the resolution You can confirm that you receive . In addition, when the pixel corresponding to the crack area is enlarged, it can be seen that it is composed of pixels of various numbers and sizes depending on the resolution, even though it is the same area, and this is defined as different physical areas during quantitative evaluation, and as a result cause a big difference in value. The physical area can be calculated by multiplying the scale index (s), which means the actual physical size of one pixel in the image, and the number of pixels in the crack area. s can be calculated based on the pinhole camera and the specifications of the camera used at the time of image acquisition and the shooting distance value, etc. Table 1 compares the maximum width and length values of the enlarged crack area for each resolution. . As the resolution increases, the value not only decreases, but also the quantification result shows that both the maximum width and length of the crack differ by more than 1 mm even though the crack area is the same.

<Table 1> Comparative result of physical area of crack area detected in image by resolution

Figure 9 is a comparison of the drawings to verify the damage detection result of the multi-resolution stacked image, Figure 9 (a) is the actual data image, Figure 9 (b) is the damage detection result of the original image, Figure 9 ( c) is the damage detection result of the multi-resolution stacked image, FIG. 9(d) is a false alarm image of the original image, and FIG. 9(e) is a false alarm image of the damage detection result from the multi-resolution stacked image. .

As a result of calculating λ to determine whether the resolution of 1,920 × 1,440 of the acquired original image meets the resolution standard suitable for minimum damage quantification, the value is 2, and the size of 3,840 × 2,880 through the previously built deep learning-based high-resolution image generation network to generate a high-resolution image that satisfies After performing damage detection based on an artificial intelligence network built by generating multi-resolution images of 250-step scale for the generated high-resolution image, the probability value for each label of each pixel is calculated through statistical techniques to obtain the final classification result label in high resolution. It is mapped to the damage detection result of the image. Figure 9 (b) is the damage detection result for the original image with a size of 1,920 × 1,440, and Figure 9 (c) uses the deep learning-based multi-resolution layered image damage detection network to double the resolution of the original image. , This is the result of final damage detection after stacking the detection results of the multi-resolution image of the 250-step scale on the generated high-resolution image of size 3,840 × 2,160. At this time, in order to quantitatively compare the crack detection results, ground truth images were manually generated for each resolution as shown in FIG. 9 (a).

As shown in FIG. 9, in order to quantitatively compare the detection results, the damage detection results of the actual data image, the original image, and the multi-resolution stacked image were compared. Among the false alarms, the results of false negative alarms that were not detected despite actual cracks correspond to the pink areas in FIGS. 9(d) and 9(e). Conversely, a false positive alarm that detects an area that is not an actual crack as a crack can be supplemented through image processing algorithms such as filtering in the quantification process, but in the case of false negative alarms, these supplementary measures are insufficient. In addition, undetected crack damage during the damage evaluation of facilities has a high importance in terms of safety and maintenance, and the recall, which is an evaluation index, is calculated and summarized as shown in Table 2.

<Table 2> Recall rate (%) of image crack detection results

Table 2 adds verification results of cases 2 and 3 as well as case 1 shown in FIGS. 8 and 9, and compares damage detection results of low-resolution and high-resolution images together. Compared to the result of the original image, it can be seen that the detection ability of the algorithm result image is improved by about 9.73% on average. In particular, the crack detection ability of the network is different according to each resolution of the three cases, and the resolution with the best detectability for each case is different, so it can be seen that it is difficult to optimize the resolution of the input image when detecting cracks.

Hereinafter, descriptions of components performing the same function among the components shown in FIGS. 1 to 9 will be omitted.

10 is a flowchart illustrating a structure damage detection method using multi-resolution image generation according to an embodiment of the present invention.

Referring to FIG. 10, the structure damage detection method performed by the structure damage detection system through multi-resolution image generation of the present invention includes receiving an image of a structure captured by a vision camera (S110), and from the received image Generating a high-resolution image that satisfies the reference value (S120), dividing the high-resolution image into multi-resolution images of the first to Nth scales, and generating damage detection result images for each scale through a damage detection network (S130). ), stacking each damage detection result image on the damage detection result image of the first scale, calculating a statistical value for each class for damage classification in pixel units based on the stacking result (S140), and the class of the largest statistical value Deriving a final classification result image by determining a final label of a pixel as a detection result of the high-resolution image (S150).

In step S120, ?汰? The step of extracting a shallow feature, the step of extracting a deep feature from a received image, the extracted ?汰? Upscaling feature data of an image constructed using features and deep features to a high-resolution size, and generating a high-resolution image from the feature data of the upscaled image.

Step S130 is a step of generating a high-resolution image as a multi-resolution image of a first scale to an N-th scale based on a bicubic interpolation algorithm, inputting the multi-resolution image to a deep convolution neural network (DCNN), and low level extracting (low-level) features and generating a first feature map by accumulating feature maps generated for each resolution; generating a second feature map by accumulating the first feature map and low-level features; Classifying the features of the feature map into damage labels in units of pixels for each scale, and generating a damage detection result image through an upsampling layer as a damage detection result for each learning class of the damage detection network.

Step S140 includes upsampling the damage detection result image generated for each scale to the same size as the maximum resolution, stacking N damage detection result images upsampled at the same resolution on the damage detection result image of the first scale, and Generating an accumulation result image for each class in pixel units, and deriving a probability value for the accumulation number based on the accumulation result image.

Generating a binary image through an image processing process to assign new label values to pixels of the damage detection result image after step S150, and multiplying the final classification result image by pixel units to generate a final damage detection result image Include steps.

The method described above may be implemented in the form of a recording medium including instructions executable by a computer, such as program modules executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

Those skilled in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention based on the above description. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. The scope of the present invention is indicated by the following claims, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be interpreted as being included in the scope of the present invention.

100: structure damage detection system
110: communication module
120: memory
130: processor
140: high-resolution image generating unit
150: multi-resolution image generation and damage detection unit
160: damage classification unit
170: damage detection result generating unit

Claims (10)

A method for detecting damage to a structure performed by a structure damage detection system through multi-resolution image generation,
(a) receiving an image of a structure photographed by a vision camera;
(b) generating a high-resolution image that satisfies a minimum reference value from the received image;
(c) dividing the high-resolution image into multi-resolution images of a first scale to an N-th scale, and generating a damage detection result image for each scale through a damage detection network;
(d) stacking each damage detection result image on the first scale damage detection result image, and calculating a statistical value for each class for damage classification in a pixel unit based on the stacking result; and
(e) deriving a final classification result image by determining a class having the largest statistical value as a final label of a pixel as a result of detecting the high-resolution image. Methods for detecting damage to structures.
According to claim 1,
The step (b) is
From the received image, ?汰? Extracting features (shallow features);
extracting a deep feature from the received image;
The extracted ?汰? Upscaling feature data of an image constructed using features and deep features to a high-resolution size; and
A method of detecting damage to a structure comprising generating a high-resolution image from feature data of the upscaled image.
According to claim 1,
The step (c) is
generating a multi-resolution image of a first to Nth scale from the high-resolution image based on a bicubic interpolation algorithm;
generating a first feature map by inputting the multi-resolution image to a deep convolution neural network (DCNN), extracting low-level features, and accumulating feature maps generated for each multi-resolution;
generating a second feature map by accumulating the first feature map and the low level feature, and classifying features of the second feature map into damage labels in units of pixels for each scale; and
And generating the damage detection result image through an upsampling layer as a damage detection result for each learning class of the damage detection network.
According to claim 1,
The step (d) is
up-sampling the damage detection result image generated for each scale to a size equal to a maximum resolution;
stacking N damage detection result images upsampled to the same resolution on the damage detection result image of the first scale; and
A method of detecting damage to a structure comprising generating an accumulation result image for each class in units of pixels and deriving a probability value for an accumulation number based on the accumulation result image.
According to claim 1,
After step (e) above
generating a binary image through an image processing process to assign a new label value to a pixel of the image as a result of the damage detection; and
The damage detection method of a structure comprising the step of multiplying the final classification result image by the binary image in pixel units to generate a final damage detection result image.
In the structure damage detection system using multi-resolution image generation,
a memory storing a program providing a method for detecting damage to a structure; and
A processor for executing a program stored in the memory;
According to the execution of the program, the processor,
Receives an image of a structure photographed by a vision camera, generates a high-resolution image that satisfies a minimum standard value from the received image, divides the high-resolution image into multi-resolution images of a first scale to an Nth scale, and damage detection network A damage detection result image for each scale is generated through, each damage detection result image is stacked on the damage detection result image of the first scale, and a statistical value for each class for damage classification in pixel units is generated based on the stacking result. and detecting damage of the structure to derive a final classification result image by determining a class of the highest statistical value as a final label of a pixel as a result of the detection of the high-resolution image.
According to claim 6,
The processor
When generating the high-resolution image,
From the received image, ?汰? Extract shallow features,
Extracting a deep feature from the received image;
The extracted ?汰? Upscale the feature data of the image constructed using features and deep features to a high-resolution size,
A structure damage detection system that generates a high-resolution image from feature data of the upscaled image.
According to claim 6,
The processor
When generating an image as a result of the damage detection,
Generating the high-resolution image into a multi-resolution image of a first scale to an N-th scale based on a bicubic interpolation algorithm;
Inputting the multi-resolution image to a deep convolution neural network (DCNN), extracting low-level features, accumulating feature maps generated for each resolution to generate a first feature map,
A second feature map is generated by accumulating the first feature map and the low level features, and features of the second feature map are classified into damage labels in units of pixels for each scale;
And generating the damage detection result image through an upsampling layer as a damage detection result for each learning class of the damage detection network.
According to claim 6,
The processor
When calculating the statistical value for each class,
Upsampling the damage detection result image generated for each scale to the same size as the maximum resolution,
Stacking N damage detection result images upsampled to the same resolution on the damage detection result image of the first scale,
A structure damage detection system comprising: generating an accumulation result image for each class in units of pixels, and deriving a probability value for an accumulation number based on the accumulation result image.
According to claim 6,
The processor
Creating a binary image through an image processing process to assign a new label value to a pixel of the image as a result of the damage detection;
Wherein the final classification result image is multiplied by the binary image in pixel units to generate a final damage detection result image.

Images