KR20190142626A - System and method for autonomous crack evaluation of structure using hybrid image scanning - Google Patents

Abstract

Provided are an automated structure crack evaluation system for detecting a crack on a structure and a method thereof. The system includes: an excitation device emitting a continuous wave line laser to a target structure; a sensing device including a thermal image camera creating thermal images by measuring a heat wave radiated from the structure and a vision camera creating realistic images by photographing the exterior of the structure; and a control device creating a thermal image and an actual image, which are spatially and temporally integrated, in regard to the thermal images and the realistic images; creating a phase image based on the spatially and temporally integrated thermal image, detecting a crack by conducting a deep learning process on the spatially and temporally integrated realistic image through an artificial neural network having learnt the realistic images, creating a thermal image including crack information by conducting a deep learning process through the artificial neural network after specifying a crack area specified in the realistic image also in the phase image, and a final image in which only the crack is visualized by mapping the thermal image with the realistic image including the crack information. Therefore, the present invention is capable of examining a crack on a large structure nondestructively and without coming in contact with the structure.

Description

Automated structure cracking evaluation system based on hybrid image scanning and its method {SYSTEM AND METHOD FOR AUTONOMOUS CRACK EVALUATION OF STRUCTURE USING HYBRID IMAGE SCANNING}

The present invention relates to an automated crack evaluation system and method for detecting cracks in a structure in a non-contact and non-destructive manner through hybrid image scanning.

Conventional methods for monitoring cracks in structures include visual inspection, vision-based crack detection, ultrasonic, passive IR thermography and active thermal. Active IR thermography and the like were used.

Visual inspection is the method by which specialists access the structure directly and measure the cracks. According to the visual inspection technique, it is less reliable than other methods because the human subject is involved in the crack evaluation, and it is difficult and cumbersome to evaluate when human access to the structure is impossible.

Vision-based damage detection is a method of estimating cracks by taking images using a vision camera. According to the vision-based damage detection technique, due to the limitation of the filed of view (FOV), it is difficult to evaluate large structures, it is not easy to quantify the cracks, and especially the false report rate according to the illumination and the surrounding environment. This has the disadvantage of being high.

Ultrasonic technique is a method of generating and measuring ultrasonic waves and evaluating cracks. However, the ultrasonic technique mainly uses a sensor attached to the structure, which is not suitable for performing the overall crack evaluation of a large structure, and has a disadvantage of complicated signal analysis.

In addition, the passive thermal imaging technique and the active thermal imaging technique are methods for measuring thermal radiation occurring naturally in a structure or thermal radiation occurring in a structure to which a heat source is applied. These techniques are evaluated as effective crack detection methods, but the heat source is limited for large structures, and the limitation of FOV makes it difficult to evaluate large structures. In addition, there is a problem that the misalignment can be generated by evaluating the surface non-uniformity point due to excessive precision.

Therefore, there is a need for a monitoring method that can effectively measure and inspect damage even for large structures.

Korea Patent No. 10-1718310 (Invention name: Vibration-based structure damage detection system and method using the drone)

One embodiment of the present invention is to solve the above-described problems of the prior art, it is possible to visualize and quantify the crack of the structure using a non-contact / non-destructive scanning hybrid image, and to dip the obtained hybrid image It is an object of the present invention to provide an automated structure cracking evaluation system and method for automating structure cracking evaluation by applying a running process.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

According to any one of the above-described means for solving the problems of the present invention, by processing a hybrid image scanning using an unmanned aerial vehicle (UAV), it is possible to non-destructive and non-contact inspection of the crack of the large structure without the access of manpower.

In addition, according to any one of the problem solving means of the present invention, it is possible to minimize the error rate of the crack evaluation through the measurement and analysis of the hybrid image (ie, thermal image and real image image).

In addition, according to any one of the problem solving means of the present invention, it is possible to overcome the limitation of the viewing angle of the hybrid image, and to visualize and quantify the damage of the structure from the real-time measurement data by scanning a wide area of interest as an inspection range in a short time. have.

In addition, according to any one of the problem solving means of the present invention, by applying the deep learning process on the hybrid image scanning results, it is possible to automatically perform accurate crack evaluation without subjective intervention of manpower.

1 is a schematic view for explaining the overall crack detection process through the automated structure crack evaluation system according to an embodiment of the present invention.
2A is a configuration diagram of a crack evaluation system according to an embodiment of the present invention.
2B is a diagram illustrating a hybrid image scanning process according to an embodiment of the present invention.
3 is a view for explaining a distortion correction method using a correction marker applied to an embodiment of the present invention.
4 is a diagram for describing a process of setting an image analysis region according to an exemplary embodiment.
FIG. 5 is a diagram illustrating an image reconstruction process through space-time coordinate axis transformation according to an embodiment of the present invention.
6 is a view for explaining a process of crack visualization image processing according to an embodiment of the present invention.
7 is a conceptual diagram of a deep learning process applied to the automated crack assessment process according to an embodiment of the present invention.
8 is a view for explaining a crack evaluation automated processing according to an embodiment of the present invention.
9 is a diagram for describing a decision making process through image matching according to an embodiment of the present invention.
10 shows concrete specimens with simulated cracks.
11 illustrates a result of distortion correction of a raw thermal image captured by an IR camera and a raw real image captured by a vision camera.
12 illustrates a result of processing a space-time coordinate axis transformation on a distortion-corrected image.
13 illustrates a noise canceled phase image obtained from the space-time integrated thermal image of FIG. 12.
FIG. 14 shows a crack image subjected to the deep CNN process for the space-time integrated real image image of FIG. 12.
FIG. 15 illustrates a decision making process through image matching between a phase image according to the thermal image of FIG. 13 and a crack image according to the real image image of FIG. 14.
FIG. 16 illustrates a resultant image of crack detection obtained according to the image matching of FIG. 15.
17 is a flowchart illustrating an automated structure cracking evaluation method based on hybrid image scanning according to an embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in the drawings, and like reference numerals designate like parts throughout the specification. In addition, while describing with reference to the drawings, even if the configuration shown by the same name may be different according to the drawing number, the drawing number is just described for convenience of description and the concept, features, functions of each configuration by the corresponding reference number Or the effect is not limited to interpretation.

Throughout the specification, when a part is "connected" to another part, this includes not only "directly connected" but also "electrically connected" with another element in between. . In addition, when a part is said to "include" a certain component, which means that it may further include other components, except to exclude other components unless specifically stated otherwise, one or more other It is to be understood that the present invention does not exclude the possibility of the presence or the addition of features, numbers, steps, operations, components, parts, or combinations thereof.

In the present specification, the term 'part' or 'module' includes a unit realized by hardware or software, and a unit realized using both, and one unit is realized by using two or more pieces of hardware. Two or more units may be implemented by one hardware.

1 is a schematic view for explaining the overall crack detection process through the automated structure crack evaluation system according to an embodiment of the present invention.

As shown in FIG. 1, the automated structure cracking evaluation system 100 according to an exemplary embodiment of the present invention is mounted on an unmanned aerial vehicle (UAV) such as a drone or the like, and is hard to be accessed by a person. The structure can be scanned and used to detect and evaluate cracks. In addition, the automated structure crack evaluation system 100 may be implemented by being mounted on an unmanned robot device such as a climbing robot.

The automated structure crack evaluation system 100 is equipped with a system for processing heterogeneous image scanning (hereinafter, referred to as a 'hybrid image scanning device') in a UAV, and acquires crack data of a large structure in a non-destructive / non-contact manner without human access. . The automated structure crack evaluation system 100 matches the two images through distortion correction and data processing on the obtained hybrid image. In addition, the automated structure crack evaluation system 100 performs an automated crack detection process through a deep learning process, thereby visualizing and quantifying cracks even for a large structure and performing crack detection at a lower false rate. Inform the user of structural damage information (ie, structure cracking information) accordingly.

In detail, the automated structure cracking evaluation system 100 scans a region of interest (ROI) on a target structure through a thermo-graphic camera and a vision camera, and heterogeneous as a scanning result. Image data (ie, thermal image and real image data) is acquired. For reference, in the present invention, an infrared (IR) camera is used as the thermal imaging camera. In addition, the heterogeneous image data will be referred to as hybrid image hereinafter.

In addition, the automated structure cracking evaluation system 100 processes a distortion calibration process to match the obtained hybrid image (ie, thermal image and real image), and then time-spatial-integrated. coordinate coordinate, TSI coordinate transform) to integrate the entire region of interest. This allows spatially integrating data that changes in real time due to the limited field of view (FOV) of the thermal imaging camera and vision camera, enabling detailed crack assessment and at the same time having a uniform thermal wave across the region of interest. You can extract aspects.

The automated structure crack evaluation system 100 calculates an instantaneous phase angle for each pixel, and processes the spatial derivative in the scanning direction, thereby obtaining a precise crack image.

In addition, the automated structure crack evaluation system 100 performs a deep learning based deep convolutional neural network process (Deep CNN process) to automate crack evaluation. Through this, the automated structure crack evaluation system 100 finds the features of the image by itself without expert intervention, and automatically classifies them to perform crack evaluation. In this case, by using a thermal image and a real image image together, crack detection can be minimized.

2A and 2B, the configuration and operation of the automated structure crack evaluation system 100 based on hybrid image scanning will be described in more detail.

2A is a configuration diagram of a crack evaluation system according to an embodiment of the present invention, and FIG. 2B is a view for explaining a hybrid image scanning process according to an embodiment of the present invention.

As shown in FIG. 2A, the automated structure crack evaluation system 100 includes an excitation device 110 for irradiating a continuous-wave line laser to a crack detection target structure, and an IR camera of the target structure. And a sensing device 120 that scans through a vision camera to obtain thermal images and real images, and a control device that detects cracks in a target structure based on the obtained hybrid images (ie, thermal images and real images). 130.

The automated structure cracking evaluation system 100 according to an embodiment of the present invention is a hybrid image scanning device composed of an excitation device 110 and a sensing device 120, and an automated structure cracking evaluation device composed of a control device 130. Can be implemented as'. At this time, the automated structure crack evaluation system 100 may be implemented as a single unit having the excitation device 110, the sensing device 120 and the control device 130. In addition, the automated structure crack evaluation system 100 may be implemented by separating the hybrid image scanning device and the automated structure crack evaluation device. As such, when the hybrid image scanning device and the automated structure cracking evaluation device are separated, the hybrid image scanning device may be mounted on the UAV. The hybrid image scanning device and the automated structure cracking evaluation device may each include a communication unit for wired / wireless communication so as to transmit and receive data or signals with each other.

The excitation device 110 includes a continuous-wave line laser (CW line laser) 111, a line beam generator 112, a collimator lens 113 and a focusing lens. lens 114).

When the point laser beam is oscillated in the continuous wave line laser 111 of the excitation device 110, it passes through the linear beam generator 112 and is transformed into a linear laser beam, and then heat wave to the target structure. (thermal wave) is generated. In this case, the linear laser beam may pass through the collimator lens 113 and the focus lens 114 to be irradiated to the target structure. For reference, the excitation device 110 may be a kind of heating unit that generates a heat wave.

In addition, the excitation device 110 may irradiate the continuous wave line laser to the target structure by moving the set path under the control of the control device 130. That is, the continuous wave line laser may be irradiated by moving a path along the set scanning direction.

The sensing device 120 includes a thermal imaging camera (ie, an IR camera) 121 and a vision camera 122.

The sensing device 120 is operated simultaneously with the excitation device 110 to transfer the thermal image data sensed through the thermal imaging camera 121 and the real image data captured by the vision camera 122 to the control device 130. to provide.

In this case, the sensing device 120 senses data along the scanning path under the control of the control device 130 and generates an IR image and a real image image according to the sensed data. That is, the sensing device 120 corresponds to the position on the target structure to which the excitation device 110 irradiates the continuous wave line laser beam and acquires hybrid images.

Referring to FIG. 2B, the excitation device 110 irradiates the continuous wave line laser beam to the target structure 201, while the sensing device 120 moves the thermal imaging camera 121 and the vision camera 122 in one direction in parallel. (I.e., scanning direction) to obtain thermal image and real image images by photographing. As in FIG. 2B, the thermal imaging camera 121 and the vision camera 122 simultaneously acquire data in the same region of interest.

The thermal imaging camera 121 measures thermal waves generated from the target structure to generate thermal image data that simulates the thermal wave shape, and the vision camera 122 generates actual image data by photographing the appearance of the target structure. That is, the heat wave generated in the target structure may be obtained through the thermal imaging camera 121, and the surface damage state of the target structure may be obtained through the vision camera 122. The thermal images and real images thus obtained are represented by I ^R which is a raw IR image and V ^{R which} is a raw vision image, respectively.

The control device 130 includes a communication unit 131, a memory 132, and a processor 133.

The communication unit 131 communicates with the excitation device 110 and the sensing device 120 to transmit and receive signals and data, respectively, and receives the hybrid image from the sensing device 120 and transmits the hybrid image to the processor 133.

In addition, the communication unit 131 may communicate with an external device (not shown) interlocked in advance, and may transmit a crack detection result or a visualized crack image to an external device under the control of the processor 133.

The structure crack detection program is stored in the memory 132. This structure crack detection program is executed by the processor 133 to detect the crack detection of the structure based on the hybrid image scanning, but to process the automated crack evaluation by applying a deep learning process for the hybrid image.

In addition to the structure crack detection program, the memory 132 further stores a control program for controlling the overall operation of the automated structure crack evaluation system 100. Such a control program may be linked to the structure crack detection program or may be implemented as a single program.

The memory 132 refers to a nonvolatile storage device that maintains stored information even when power is not supplied, or a volatile storage device that requires power to maintain stored information.

The processor 133 controls the overall operation of the automated structure crack assessment system 100. To this end, the processor 133 may be implemented to include at least one processing unit (CPU, micro-processor, DSP, etc.), random access memory (RAM), read-only memory (ROM), or the like, and the memory 132. The program stored in the C) may be read into the RAM and executed through at least one processing unit. In addition, according to an embodiment, the term “processor” may be interpreted to have the same meaning as the terms “controller”, “operating device”, “control unit”, and the like.

At this time, the processor 133 processes the following operation as the structure crack detection program is executed.

The processor 133 transmits the control and synchronization signal to the device 110 having the control and synchronization signal, and simultaneously transmits the control and synchronization signal corresponding to the signal transmitted to the device 110 to the sensing device 120. Accordingly, the excitation device 110 irradiates the continuous wave line laser beam to the target structure based on the control and synchronization signal. In parallel with this, the sensing device 120 measures the heat wave radiated from the target structure through the thermal imaging camera 121 and photographs the exterior of the target structure through the vision camera 122.

The processor 133 receives the hybrid image from the sensing device 120 and processes data processing for the hybrid image to extract and visualize information about the damaged area (ie, the crack generation area).

Hereinafter, a process in which the processor 133 performs distortion correction processing, image reconstruction processing, and data processing for crack visualization on the hybrid image will be described with reference to FIGS. 3 and 6.

First, a process of processing distortion correction on the hybrid image with the processor 133 will be described with reference to FIG. 3.

3 is a view for explaining a distortion correction method using a correction marker applied to an embodiment of the present invention.

Hybrid images acquired through the thermal imaging camera 121 and the vision camera 122 may generate resolution differences and lens system distortions. In order to match and quantify this, the processor 133 performs distortion correction on the hybrid image.

Since the camera image is obtained by projecting a three-dimensional space onto a two-dimensional image plane, the camera image has a transformation relationship between three-dimensional space coordinates and two-dimensional image coordinates as shown in Equation 1 below. In this case, Equation 1 is expressed based on a pin-hole camera modeling technique representing a correlation between the 3D real world and the 2D image plane.

<Equation 1>

In Equation 1, S represents an arbitrary scale factor, [x, y, z, 1] is an image coordiante system, and [X, Y, Z, 1] is a world coordinate system. The coordinates of the three-dimensional vertices of the world coordinate system. Where R and t are rotation and transformation matrices, respectively. [R | t] represents an rotation parameter for converting the world coordinate system into a camera coordinate system with an extrinsic parameter, and A is an intrinsic parameter of the camera.

The pin-hole camera model simplifies the geometric projection relationship between 3D space and 2D image planes. In this case, a distortion correction process is performed using a calibration marker, and a homography matrix H may be defined as shown in Equation 2 below to project a 3D space into a 2D plane.

<Equation 2>

In Equation 2, λ is an arbitrary scalar value, and r ₁ and r ₂ are each elements of R.

Given a calibration marker image as in FIG. 3, H can be estimated according to a maximum likelihood criterion. In this case, when r ₁ and r ₂ are orthonormal, they may be represented by the following equations (3) and (4).

<Equation 3>

<Equation 4>

If the closed-form solution of A in Equations 3 and 4 is known, external parameters can be obtained as in Equation 5 below.

<Equation 5>

As such, after acquiring internal and external parameters for the thermal imaging camera 121 and the vision camera 122 using the calibration marker, the I ^R and V ^R images are distorted through the distortion correction process of Equation 1 above. It is converted to I ^C which is distortion-calibrated IR images and V ^C image which is distortion-calibrated vision images. 3 illustrates a process of converting an I ^R and V ^R image into an I ^C and V ^C image through a distortion correction process.

Next, the process of reconstructing the distortion-corrected hybrid image by the processor 133 will be described with reference to FIGS. 4 and 5.

4 is a view for explaining a process of setting an image analysis region according to an embodiment of the present invention, Figure 5 is a view for explaining a process of reconstructing the image through the time-space coordinate axis transformation according to an embodiment of the present invention to be.

The thermal images and the real images received from the sensing device 120 each include a process in which the heat wave and the appearance change with time as the FOV is moved to the scanning path while the FOV is fixed. Thus, hybrid images need to be integrated and reconstructed to be suitable for damage detection.

Accordingly, the processor 133 generates an integrated image by performing a TSI coordinate transformation on the distortion-corrected images through Equations 1 to 5 above.

First, the processor 133 processes an image reconstruction by setting a region of interest and transforming a space-time coordinate axis.

In detail, image reconstruction by transforming the space-time coordinate axis enables to check the entire ROI beyond the limit of the viewing angle of the camera. However, before performing the space-time coordinate axis transformation, it is necessary to designate a region of interest (ROI) of the hybrid image.

Since the object structure is fixed and the hybrid image scanning device moves while maintaining a distance from the object structure, the physical area contained in the viewing angle is defined. Therefore, as the hybrid image scanning device (ie, the excitation device and the sensing device) moves with time, the physical area of the target structure is acquired as a heterogeneous image.

Referring to FIG. 4, assuming that the hybrid image scanning apparatus scans along the horizontal direction ('x axis' in FIG. 4), the distribution of laser heat waves passing through the linear beam generator is a Gaussian distribution within the region of interest. Similar to Accordingly, the midpoint of the heat wave in the obtained thermal image can be obtained using μ (x) of the Gaussian distribution. In this case, an image of the ROI may be acquired by setting the affected boundary as the Gaussian distribution confidence interval of 95%. In other words, the region of interest refers only to the region affected by the linear laser beam, which can be expressed by Equation 6 below.

<Equation 6>

And R ^ROI is of interest in the thermal image in Equation 6, h is a height of the region of interest, l _center is the center point of the heat waves, x is space, the horizontal area of the R ^ROI, y is an area perpendicular to the area of R ^ROI , t means the time domain of the R ^ROI . According to the corresponding R ^ROI image, the ^ROI of the real image image may also be equally represented.

Here, the hybrid image is transformed into an I ^ROI , which is an integrated IR image of the ^ROI , through a space-time coordinate axis transformation, and the real image is an integrated vision image of the ^ROI . Is converted to V ^ROI .

Referring to FIG. 5, the space-time coordinate axis transformation is based on a physical phenomenon in which a point of a specific space is heated and cooled by laser excitation.

Accordingly, the x-axis data of the I ^C image may be regarded as a thermal change in the time domain at a specific point of the viewing angle, and the t-axis data may be regarded as a thermal change in the spatial area at a specific point in time. Therefore, I ^C is converted into an I ^ROI image through Equation 7 below.

 <Equation 7>

이며, v는 스캐닝 속도이다. 또한 위 첨자 *은 변형된 좌표를 의미한다. 이와 같은 수학식 7의 변환을 통해 전체 관심 영역에 동시에 열 가진 및 냉각 사이클 (heating and cooling cycle)의 효과를 확인할 수 있다.At this time,

V is the scanning speed. The superscript * also means transformed coordinates. Through the conversion of Equation 7, it is possible to confirm the effects of heating and cooling cycles on the entire region of interest simultaneously.

In addition, like the I ^C image, the V ^C image is converted into a V ^ROI image through Equation 8 below. In this case, the real image is integrated with only the spatial region since there is no laser excitation.

<Equation 8>

 Next, a process of data processing for visualizing the crack of the target structure by the processor 133 will be described with reference to FIG. 6.

6 is a view for explaining a process of crack visualization image processing according to an embodiment of the present invention.

After reconstructing the image, the processor 133 performs a phase mapping process for crack visualization. The phase mapping process has the effect of equalizing for each pixel, allowing the identification of microcracks occluded by differences in amplitude when large cracks exist in an image. That is, the phase mapping process is performed to prevent micro cracks from being undetected by large cracks by leveling for each pixel.

Referring to FIG. 6A, a Hilbert transform may be performed on a time axis t ^* of each I ^ROI image to acquire data including both real and complex values for each pixel. It can be expressed as in Equation (9).

<Equation 9>

In Equation 9, P is the cauchy principle value, and τ represents the parallel movement.

Referring to (b) of FIG. 6, the instantaneous phase angle is calculated by dividing each pixel into real and imaginary parts for the Hilbert transform data as described above, which can be expressed by Equation 10 below. have.

<Equation 10>

In Equation 10, Re and Im represent a real part and an imaginary part, respectively.

Referring to FIG. 6C, each image representing the instantaneous phase angle is added to the t ^* axis to construct a phase mapping image, which may be represented by Equation 11 below.

<Equation 11>

Through the above steps, a phase image can be obtained.

On the other hand, in the phase image, there may still exist point noise generated by increasing the value of the linear noise in the scanning direction generated by the scanning and the point noise generated during the leveling process. Therefore, the processor 133 may further perform the noise removing process.

Referring to (d) of FIG. 6, only intact cracking elements may be left by removing noise by performing a spatial derivative in the scanning direction x ^* through Equation 12 below. Accordingly, an I ^P image, which is a signal-processed IR image, may be obtained such that only crack elements remain.

<Equation 12>

Hereinafter, a process of performing the crack evaluation automation process by the processor 133 will be described with reference to FIGS. 7 to 9.

7 is a conceptual diagram of a deep learning process applied to an automated crack evaluation process according to an embodiment of the present invention, and FIG. 8 is a view for explaining a crack evaluation automated process according to an embodiment of the present invention.

The processor 133 performs a deep learning process, which is a kind of artificial neural network, for automatic crack evaluation. In an embodiment of the present invention, a deep CNN process as shown in FIG. 7 may be used as a deep learning process. In FIG. 7, it is shown that Googlenet (GoogLeNet), which is a known learning model as an example of the deep CNN process, is not limited thereto.

Referring to FIG. 7, the artificial neural network for image evaluation for crack evaluation includes 'Fully-connected layer' and 'Softmax layer', which are final stages of layers of artificial neural networks that have been previously formed through transfer learning. In addition, the 'Classification layer' can be rebuilt by changing the crack detection.

At this time, the processor 133 learns a sufficient amount of real image images so that the crack image and the non-crack image, and images such as shadows, vegetation, and moss, which may be mistaken for the crack, are learned in the artificial neural network. Using the artificial neural network for crack detection, the deep CNN process is performed on the V ^ROI image.

Referring to FIG. 8, a convolution mask size is determined by optimizing a crack size, and the processor 133 performs a deep CNN process with at least 16 convolution masks within a similar range. The processor 133 overlaps the result to obtain a reliability map for the crack detection reliability. Then, the processor 133 performs an image processing process to extract exactly the cracks of the composite product mask area detected as the cracks based on the obtained reliability map. In detail, the processor 133 applies a median filter to the detected product mask region. In addition, the processor 133 may perform threshold processing based on an extreme value distribution (EVS) along a Weibull distribution, in order to process point noise caused by a heterogeneity of a structure (eg, a concrete) surface. (tthresholding process). Through this, a deep CNN process is performed on the V ^ROI image to obtain V ^D , which is a deep CNN processed vision image showing only crack information.

The V ^D image contains the crack information of the target structure, but there is a possibility of false alarm that misrecognizes the crack by the surface condition, roughness, dust, etc. of the structure. In order to eliminate the possibility of misinformation on the V ^D image, the processor 133 performs an image matching process to increase the reliability of crack evaluation.

At this time, the processor 133 increases the reliability of crack evaluation through image matching between V ^D and I ^P.

9 is a diagram for describing a decision making process through image matching according to an embodiment of the present invention.

Referring to (a) process of Figure 9, the processor 133 is the same position and size as the cracks each one convolution mask V ^D image in the specified and, I ^P image with the each successive crack being detected at V ^D images Select masks. The masks selected from the I ^P images are respectively resized to a predetermined size (for example, 224x224x3) to be suitable for deep CNN processing, and then the mask images are set as I ^M images.

Referring to the processes of FIGS. 9B and 9C, the processor 133 performs a deep CNN process (ie, a deep CNN process previously trained based on V ^ROI ) on an I ^M image. depending on the results cracking image information rather than the infrared image collect only I ^D crack the (deep CNN processed IR image) it contains the causes excluded.

Referring to the process of FIG. 9 (d), the processor 133 adjusts the I ^D images collected through the above process back to the original size and maps the final result image to the V ^D image. Create This can increase the crack recognition rate.

Hereinafter, the crack detection effect of the crack evaluation system 100 according to an embodiment of the present invention will be specifically described with reference to FIGS. 10 to 16. However, the present invention is not limited by the following examples.

EXAMPLE

In this embodiment, the concrete specimen, which is the target structure, is fixed to the wall, and the hybrid image scanning device (ie continuous wave line laser, thermal imaging camera, and vision camera) moves horizontally along the scanning jig. It was set to move at a speed of 23 mm / s. That is, in the present embodiment, the role of the UAV is substituted by using the scanning jig.

In addition, in the present embodiment, a 950 nm wavelength laser beam is oscillated by the continuous wave laser, and the dot laser beam is transformed into a line laser beam through a linear beam generator so that heat waves are excited on the specimen. At this time, the size of the line laser beam is 5x200mm ² , the intensity of the laser excited on the specimen is approximately 25mW / mm ² , the thermal conductivity of the concrete specimen is 0.8Wm ^-1 k ^-1 .

In this embodiment, the vision camera acquires data at a frame rate of 30 Hz, but the pixel resolution is set to be 3264x2448. In addition, the thermal imaging camera acquires data at a frame rate of 30 Hz, but sets the spectral range to 3 μm to 5 μm and the pixel resolution to 640 × 512. The data was acquired for 22 seconds using a vision camera and a thermal imaging camera. At this time, the hybrid image scanning device is 700mm away from the surface of the specimen.

In addition, in this embodiment, the entire system is operated by a control computer, which corresponds to the control device.

For reference, in this embodiment, the indoor verification experiment was performed using the scanning jig, but the weight of the excitation device, the sensing device, and the control device used in the experimental setup was added so that the appropriate weight (approximately 3 kg) was mounted so that the actual UAV was mounted. Field experiments are also possible.

10 shows concrete specimens with simulated cracks.

Specimen used in this example has a concrete mixing ratio as shown in Table 1 below, the size is 1000X500X100 mm ³ , the compressive strength is 103MPa, it was produced using three kinds of cement.

TABLE 1

Referring to FIG. 10, within the ROI on a specimen of size 750 x 240 mm ² , macro cracks (≥500 μm), micro cracks (<500 μm), and false false alarms are identified. A fake crack of 1 mm width drawn with a pen to simulate was simulated.

First, the result of processing image distortion correction on a hybrid image obtained by heterogeneous scanning of the specimen will be described.

11 illustrates a result of distortion correction of a raw thermal image captured by an IR camera and a raw real image captured by a vision camera.

In Fig. 11, the left side of each arrow is an I ^R and V ^R image acquired after 10 seconds of laser excitation, and the right side of each arrow is an I ^C and V ^C image obtained by distortion correction processing of the I ^R and V ^R images, respectively. In this case, the image distortion of the real image image was successfully corrected, but the difference between the before and after distortion correction of the thermal image is not significant.

Next, the results of image reconstruction through space-time coordinate axis transformation for the distortion-corrected image will be described.

12 illustrates a result of processing a space-time coordinate axis transformation on a distortion-corrected image.

The I ^C and V ^C images as shown in FIG. 11 are transformed into I ^ROI and V ^ROI by space-time coordinate transformation through Equations 7 and 8, respectively, as described above. 12A and 12B are I ^ROI images and V ^ROI images acquired 1 second after the laser excitation, respectively. This I ^ROI image can confirm the heating and cooling effect of the ROI over time, and the V ^ROI image is represented as a single image because it is not affected by laser excitation.

Next, data processing for visualizing cracks for detailed crack evaluation will be described.

FIG. 13 illustrates a noise canceled phase image obtained from the space-time integrated thermal image of FIG. 12.

For detailed crack evaluation, the instantaneous phase angle by Hilbert transform is calculated for each pixel of the I ^ROI image through Equations 9 and 10 described above. In addition, the phase mapping image may be obtained by accumulating data in the t ^* -axis direction based on the calculated phase angle according to Equation 11 described above. Then, in order to remove the noise component and visualize only the crack information, spatial differentiation is performed in the scanning direction x ^* according to Equation 12 described above. As a result, as shown in FIG. 13, an I ^P image capable of confirming cracks in both macro and micro units without noise components can be obtained.

Next, the automated crack evaluation process for reliable crack detection will be described.

FIG. 14 shows a crack image subjected to the deep CNN process for the space-time integrated real image image of FIG. 12.

Specifically, the crack detection in the V ^ROI image was performed through a deep CNN process using an artificial neural network that previously trained real image crack images.

For reference, the artificial neural network that reconstructed the existing Google Net (GoogLeNet) for crack detection through transfer learning was utilized. In order to construct the artificial neural network for crack detection, 200 pieces of crack images and no crack images were collected, and 200,000 pieces were learned through augmentation and segmentation through rotation and symmetry. In addition, in order to reduce errors, the deep CNN process was performed three times for each convolution mask size, and a reliability map was constructed using all the data. That after the following, a noise removal process by mapping the V ^ROI image to obtain an image, such as V ^D in Fig.

In this case, the performance of the deep CNN process may be evaluated by calculating reliability indices such as precision and recall, and may be represented by Equations 13 and 14 below.

<Equation 13>

<Equation 14>

In Equations 13 and 14, Tp, Fp, and Fn represent 'True positive', 'False Positive', and 'False negative', respectively, and the precision and recall values were calculated as 66.64% and 91.93%, respectively. At this time, the precision value is relatively low because the fake crack is recognized as the actual crack.

FIG. 15 illustrates a decision making process through image matching between a phase image according to the thermal image of FIG. 13 and a crack image according to the real image image of FIG. 14.

FIG. 16 illustrates a resultant image of crack detection obtained according to the image matching of FIG. 15.

FIG. 15 (a) shows that the continuous crack among the cracks detected in the V ^D image of FIG. 14 was selected as one mask. And Figure 15 (b) showed that masks selected in the same position as the selected mask to V ^D of (a) with respect to the I ^P in Fig. In this case, FIG. 15B illustrates an I ^P image to which an image I ^D determined as a crack is mapped.

In detail, images selected as masks in I ^P of FIG. 13 are sized to a size (that is, 224 x 224 x 3) set for crack detection through an artificial neural network, and then classified into I ^M images. These I ^M images undergo a deep CNN process, and only the images determined to be cracks are stored as I ^D. The I ^Ds are then resized to their original size and mapped to I ^P. Referring to FIG. 15, the masks indicated by dotted yellow lines in (a) and (b), respectively, are set according to fake cracks, and in I ^{P to} which the I ^D image of (b) is mapped, fake masks in the masks. It can be confirmed that this was not detected.

Next, I ^D converted to its original size is mapped to V ^D of FIG. 14. Through this, a final image as shown in FIG. 16 may be obtained.

In order to compare the crack detection performance (ie, reliability) between V ^D of FIG. 14 and the final image of FIG. 16, precision and recall are shown in Table 2 below.

TABLE 2

When using the hybrid image as described above, it can be seen that the two indexes for determining the reliability are clearly improved.

Hereinafter, an automated structure crack evaluation method based on hybrid image scanning through the automated structure crack evaluation system 100 according to an embodiment of the present invention will be described with reference to FIG. 17.

17 is a flowchart illustrating an automated structure cracking evaluation method based on hybrid image scanning according to an embodiment of the present invention.

First, the hybrid image scanning on the target structure is processed (S110).

Specifically, the continuous wave line laser is irradiated to the target structure through the excitation device 110, but in parallel with the sensing device 120 (ie, the thermal imaging camera and the vision camera) synchronized with the excitation device 110. The raw thermal image data generated by measuring the heat waves radiated from the structure and the raw real image data of the appearance of the target structure are acquired. At this time, the raw thermal image data (ie, the thermal image) and the raw actual image data (ie, the virtual image) are provided to the control device 130.

Next, data processing is performed to reconstruct the raw thermal image and the raw real image to be suitable for crack detection, respectively (S120).

Specifically, the controller 130 performs image distortion correction processing on each of the raw thermal image image I ^R and the raw real image image V ^R , thereby correcting the distortion-corrected thermal image image I ^C and distortion correction. The real image image V ^C is obtained (S121).

At this time, the image distortion correction process is a process of obtaining a two-dimensional homography between the raw thermal image and the raw real image image and the calibration markers, calculating a camera internal and external parameters based on the two-dimensional homography, and the internal and external parameters The method may include obtaining a distortion-corrected thermal image and a real image image by applying the parameter to a mathematical equation (eg, Equation 1) in which a correlation between the 3D real world and the 2D image plane is modeled.

And the control device 130 is the distortion correction with the infrared image (I ^C) and by performing a space-time integration coordinate conversion for the distortion correction real image pictures (V ^C), with each space-time integrated thermal image (I ^ROI) The space-time integrated real image image V ^ROI is obtained (S122).

In this case, before performing the spatiotemporal coordinate coordinate transformation, an analysis region (ie, an ROI) may be set. For example, a gaussian distribution along the heat wave distribution of the line laser beam irradiated onto the target structure may be matched to the y-axis of the FOV to set an analysis region having a predetermined confidence interval of the Gaussian distribution.

Next, the control device 130 performs data processing for crack visualization on the spatiotemporal integrated thermal image I ^ROI to obtain a thermal image I ^P including crack information (S123).

In this case, a phase mapping process is performed based on the space-time integrated thermal image I ^ROI to generate a phase image. The phase mapping process performs a Hilbert transformation on the time axis t ^* of the spatiotemporal integrated thermal image (I ^ROI ) containing the cracks, calculates the instantaneous phase angle of each pixel, and then instantaneous phases on the time axis. Include the process of adding angles.

In addition, a process of removing noise included in the phase image may be further performed. In exemplary embodiments, the noise removing process may include spatially differentiating a phase image in a scanning direction.

After the processing step (S120) of the data processing including the above processes, the control device 130 performs a crack evaluation automation process for the image in which the crack is visualized (S130).

Specifically, the control device 130 through the artificial neural network learned in advance the real image image for crack evaluation (for example, crack image, non-crack image, image containing object information that can be mistaken crack), the (S122) Performs a deep learning process (ie, a deep CNN process) on the spatiotemporal integrated real image image V ^ROI obtained in step), and as a result, obtains a real image image V ^D including crack information ( S131).

In this case, the composite product mask is determined by optimizing the crack size in the deep CNN process, and the deep CNN process is performed using a plurality of composite product masks within a similar range. The results are superimposed to obtain a reliability map for crack detection reliability.

Then, based on the reliability map, image processing is performed to extract exactly cracks from the composite product mask region detected as cracks. The image processing process applies a median filter to the detected product mask area and performs critical processing based on the extreme value distribution (EVS) according to the Weibull distribution to produce a real image image (V ^D ) showing only crack information. It may include the process of obtaining.

Next, in order to remove the fake crack information included in the real image image V ^D including the crack information, the control device 130 includes the thermal image I including the crack information obtained in step S123. Decision processing is performed through image matching with ^P ) (S132).

At this time, the decision processing through image matching may designate a continuous crack as a composite product mask on the real image image V ^D including the crack information and the thermal image image I ^P including the crack information. The process of selecting a mask having the same position and size as that of the mask, and adjusting the masks selected on the thermal image image I ^P including the crack information to the preset image size for deep learning, respectively, to the mask image I ^M. The process of setting, the process of performing a deep learning process (ie, a deep CNN process) through the artificial neural network learned from the real image image for the crack evaluation by inputting the mask image (I ^M ), as a result of collecting only the crack image The process of acquiring a thermal image (I ^D ) showing only the crack information except for the image that is not a crack, and reconstructing the thermal image (I ^D ) showing only the crack information to the original size The method may include obtaining a final result image by mapping to the real image image V ^D including the column information. Through this, the reliability of crack evaluation can be greatly increased.

After the crack evaluation automation processing step (S130), the control device 130 outputs the final result image in which the crack information is visualized (S140).

In this case, the automated structure cracking evaluation system 100 may further include an output device (not shown) by itself, or the automated structure cracking evaluation device may be linked with an external output device (not shown). In this case, the control device 130 may not only output the final result image through the output device, but also output the images acquired for each processing step described above. In addition, the control device 130 may provide corresponding image data by wired / wireless communication through the communication unit 131 to an external device requesting the final result image.

The crack detection method based on hybrid image scanning according to the embodiment of the present invention described above may be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Such recording media include computer readable media, and computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, computer readable media includes computer storage media, and computer storage media are volatile and nonvolatile implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Both removable and non-removable media.

The foregoing description of the present invention is intended for illustration, and a person of ordinary skill in the art may understand that the present invention can be easily modified in other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

In addition, while the methods and systems of the present invention have been described in connection with specific embodiments, some or all of their components or operations may be implemented using a computer system having a general purpose hardware architecture.

The scope of the present invention is shown by the following claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention. .

100: automation structure crack evaluation system 110: the excitation device
120: sensing device 130: control device

Claims (15)

In automated structure cracking evaluation system based on hybrid image scanning,
An excitation device for irradiating a continuous wave line laser to a target structure;
A thermal imaging camera for measuring thermal waves radiated from the target structure to generate thermal images, and a vision camera for photographing the exterior of the target structure to generate real image images. A sensing device synchronized with the excitation device and moving in a preset scanning direction; And
Distortion correction processing and spatiotemporal coordinate coordinate transformation processing are performed on the thermal images and the real image images to generate a spatiotemporal thermal image and a real image, and a phase mapping process on the spatiotemporal thermal image. To generate a phase image, perform a deep learning process on the space-time integrated real image image through the artificial neural network with which the real image images for crack evaluation were learned in advance, and detect the crack, A crack region is specified in the image image, and a region corresponding to the same position and size as the specified crack region is specified in the phase image, and a deep learning process is performed through the artificial neural network for the image for each region specified in the phase image. To generate a thermal image containing crack information, the crack information included A control device for mapping the thermal image to the real image image in which the crack is detected to generate a final image in which only the crack is visualized,
The real image image for crack evaluation, at least one of a crack image, a non-crack image, the image containing at least one object information mistaken for a crack, automated structure crack evaluation system. The method of claim 1,
The control device,
A deep convolutional neural network process is performed on the space-time integrated real image image,
Crack detection reliability maps by overlapping the results of performing the deep convolutional neural network process with a plurality of convolution masks by applying a predetermined convolution mask size to the space-time integrated real image. ),
A crack is detected by applying a median filter to a composite product mask region detected as including a crack based on the reliability map, and a thresholding process based on an extreme value distribution (EVS). To remove the point noise by performing an automated structure cracking evaluation system. The method of claim 1,
The control device,
As a deep learning process, a deep convolutional neural network process is performed.
Each of the cracks detected in the space-time integrated real image is designated by a composite product mask for each continuous crack,
Selecting a mask having a position and a size corresponding to the composite product mask in the phase image,
Generate a mask image in which the selected mask is resized to an image size preset for the deep convolutional neural network process,
Extracting the mask images in which the crack is detected by performing the deep convolutional neural network process using the mask image as an input,
Resizing the extracted mask image to its original size to generate a thermal image including the crack information,
Automated structure cracking evaluation system to remove the false crack by mapping the thermal image including the crack information to the real image image of the crack detected. The method of claim 1,
The excitation device and the sensing device is mounted on an unmanned aerial vehicle or an unmanned climbing robot, automated structure cracking evaluation system. In the automated structure cracking evaluation apparatus of automated structure cracking evaluation system based on hybrid image scanning,
An excitation device for irradiating a continuous wave line laser to a target structure, a thermal imaging camera for generating raw thermal image images by measuring the heat wave radiated from the target structure, and a vision for generating the raw real image images by photographing the exterior of the target structure A raw thermal image and a raw seal for the target structure from a hybrid image scanning device including a camera, wherein the thermal imaging camera and the vision camera each include a sensing device that is synchronized with the excitation device and moves in a preset scanning direction. A communication unit for receiving an image image;
A memory storing a structure crack detection program based on hybrid image scanning; And
A processor for executing a program stored in the memory;
The processor is configured to execute the structure crack detection program.
Distortion-correction processing and spatiotemporal coordinate coordinate transformation processing are performed on the raw thermal image images and the raw real image images to generate a space-time integrated thermal image and a real image image;
Performing a phase mapping process on the space-time integrated thermal image to generate a phase image,
Crack detection is performed by performing a deep learning process on the real-time image integrated with the space-time through the artificial neural network where the real-image images for crack evaluation have been learned in advance
The crack region is specified in the real image image in which the crack is detected, and the region corresponding to the same position and size as the specified crack region is specified in the phase image, and the image for each region specified in the phase image is selected from the artificial neural network. Performs a deep learning process to create a thermal image with crack information,
Mapping the thermal image including the crack information to the real image in which the crack is detected to generate a final image in which only the crack is visualized;
The real image image for crack evaluation, at least one of a crack image, a non-crack image, the image containing at least one object information mistaken for a crack, automated structure crack evaluation apparatus. The method of claim 5,
The processor,
A deep convolutional neural network process is performed on the space-time integrated real image image,
Crack detection reliability maps by overlapping the results of performing the deep convolutional neural network process with a plurality of convolution masks by applying a predetermined convolution mask size to the space-time integrated real image. ),
A crack is detected by applying a median filter to a composite product mask region detected as including a crack based on the reliability map, and a thresholding process based on an extreme value distribution (EVS). To eliminate the point noise by performing the automated structure cracking evaluation apparatus. The method of claim 5,
The processor,
As a deep learning process, a deep convolutional neural network process is performed.
Each of the cracks detected in the space-time integrated real image is designated by a composite product mask for each continuous crack,
Selecting a mask having a position and a size corresponding to the composite product mask in the phase image,
Generate a mask image in which the selected mask is resized to an image size preset for the deep convolutional neural network process,
Extracting the mask images in which the crack is detected by performing the deep convolutional neural network process using the mask image as an input,
Resizing the extracted mask image to its original size to generate a thermal image including the crack information,
Automated structure cracking evaluation apparatus to remove the false crack by mapping the thermal image including the crack information to the real image image of the crack detected. The method of claim 5,
The processor,
Estimate a two-dimensional homography matrix between the raw thermal images and the raw real images and a calibration marker,
Calculating camera internal and external parameters based on the two-dimensional homography matrix,
Applying the internal and external parameters to a correlation model between a predetermined three-dimensional real world and a two-dimensional image plane to convert the raw thermal images and the raw real images into distortion-corrected images, respectively. Crack evaluation device. The method of claim 5,
The processor,
Setting an analysis region having a predetermined confidence interval for the Gaussian distribution followed by the heat wave distribution of the continuous wave line laser irradiated to the target structure,
After applying the analysis region to each of the thermal image and the real image image according to the distortion correction processing, and performing the spatiotemporal coordinate coordinate transformation, transformed into the spatiotemporal image and real image image for the analysis region Will, automated structure crack evaluation device. The method of claim 5,
The processor,
Performing a Hilbert transform on the time axis of the space-time integrated thermal image to obtain real and complex values for each pixel,
Calculating an instantaneous phase angle with respect to the real part and the imaginary part of each pixel,
And adding the calculated instantaneous phase angle with respect to the time axis to generate a phase image. The method of claim 10,
The processor,
Spatially differentiating in the scanning direction with respect to the phase image to remove noise. In the automated structure crack evaluation method based on hybrid image scanning through an automated structure crack evaluation system,
Irradiating a continuous wave line laser to a target structure through an excitation device;
Raw thermal images and raw real image images of external appearances of the target structures are measured by measuring thermal waves radiated from the target structure through a thermal imaging camera and a vision camera synchronized with the excitation device and moving in a predetermined scanning direction. Generating them;
Generating a space-time integrated thermal image and a real image by performing a distortion correction process and a space-time integrated coordinate transformation process on the raw thermal image images and the raw real image images;
Generating a phase image by performing a phase mapping process on the space-time integrated thermal image;
Detecting a crack by performing a deep learning process on the space-time integrated real image image through an artificial neural network with which the real image images for crack evaluation have been previously learned;
The crack region is specified in the real image image in which the crack is detected, and a region corresponding to the same position and size as the specified crack region is specified in the phase image, and the image for each region specified in the phase image is determined by the artificial neural network. Performing a deep learning process to generate a thermal image including crack information; And
Mapping the thermal image image including the crack information to the real image image in which the crack is detected to generate a final image in which only the crack is visualized;
The real image image for crack evaluation, crack image, non-crack image, at least one of the image containing at least one object information mistaken for a crack, automated structure crack evaluation method. The method of claim 12,
The deep learning process may include generating a real image image including crack information.
A deep convolutional neural network process is performed on the space-time integrated real image image,
Crack detection reliability maps by overlapping the results of performing the deep convolutional neural network process with a plurality of convolution masks by applying a predetermined convolution mask size to the space-time integrated real image. ),
A crack is detected by applying a median filter to a composite product mask region detected as including a crack based on the reliability map, and a thresholding process based on an extreme value distribution (EVS). To remove the point noise by performing the automated structure crack evaluation method. The method of claim 12,
The deep learning process may be performed through the neural network to generate a thermal image including crack information.
As a deep learning process, a deep convolutional neural network process is performed.
Each of the cracks detected in the real image image in which the cracks are detected is designated as a composite product mask for each consecutive crack,
Selecting a mask having a position and a size corresponding to the composite product mask in the phase image,
Generate a mask image in which the selected mask is resized to an image size preset for the deep convolutional neural network process,
Extracting the mask images in which the crack is detected by performing the deep convolutional neural network process using the mask image as an input,
Resizing the extracted mask image to its original size to generate a thermal image including the crack information,
Automated structure crack evaluation method for removing the false crack by mapping the thermal image including the crack information to the real image image of the crack detected. A computer-readable recording medium having recorded thereon a program for implementing the method of claim 12.

Images