KR102355997B1 - method for monitoring the soundness of concrete structure based on smart glass - Google Patents

Abstract

The present invention relates to a smart glass-based concrete structure soundness monitoring method, which monitors the soundness of a concrete structure by constructing a digital exterior survey network map (image) containing the locations of multiple damaged areas of the concrete structure based on smart glass, and performs processing with deep learning technology. The smart glass-based concrete structure soundness monitoring method according to the present invention comprises the steps of: (a) capturing images of each of multiple damaged areas of the concrete structure using a camera attached to the smart glass; (b) performing image pre-processing on each of the multiple damaged images; (c) performing local image registration using affine transformation based on mutual information amount for each image of multiple damaged areas which have undergone image preprocessing; (d) equalizing the brightness values of overlapping regions in the image registration process using histogram equalization; and (e) building a digital exterior survey network map by combining the images of each multiple damage area which has undergone (d) step. The present invention can more accurately analyze the type of fault.

Description

Smart glass-based concrete structure soundness monitoring method {method for monitoring the soundness of concrete structure based on smart glass}

The present invention relates to a smart glass-based concrete structure health monitoring method. In particular, a digital exterior survey network map (video) containing the locations of multiple damaged areas of a concrete structure based on smart glass is constructed and then it is used with deep learning technology. It relates to a smart glass-based concrete structure health monitoring method that processes and monitors the health of the concrete structure.

As is well known, the safety diagnosis of concrete structures such as bridges or buildings can be performed by an inspector climbing on the structure and visually checking the damage or deterioration that has been newly or previously created in the structure.

For this purpose, the inspector should be able to check whether the damage or deterioration is new or existing, and whether additional growth has occurred in the case of existing damage or deterioration, while visually checking the damage or deterioration one by one by approaching the concrete structure.

However, in the prior art, an inspector brings a paper drawing of a massive structure to mark damage or deterioration information, and if necessary, uses a writing instrument to add a description, photograph the damaged or deteriorated area with a camera, and in addition to using a mobile communication terminal Since the work has been carried out in a way that reports on site conditions or requests additional information, the risk of safety accidents has always existed.

In addition, there was a problem in that the location, type and size of damage or deterioration was inaccurate because it was made by the subject of the inspector.

In consideration of this, a smart glass safety diagnosis system and a safety diagnosis method are proposed in the following prior art 1, and FIG. 1 is a system configuration diagram thereof. As shown in FIG. 1 , the smart glass safety diagnosis system 10 of the prior art 1 includes a smart glass 100 , a portable terminal 200 , a server 300 , and a database 400 .

The smart glass 100 is a device in the form of glasses that is equipped with a camera, an interface device (eg, a microphone) capable of inputting commands, a wireless communication device, and an augmented reality (AR) function to perform a see-through function and a computer function. can be worn by the user.

The portable terminal 200 is a portable terminal having communication and computer functions, and a smart phone, a tablet PC, etc. may be used. The digital drawing 210 of the structure under diagnosis may be always output to the mobile terminal 200, and if there is damage or deterioration that has been discovered and managed in the structure, the marker 220 has already been created and displayed. can

The marker 220 is preferably a QR code 221 , and the marker 220 as the QR code 221 is the structure, the location of damage and deterioration in the structure, the shape and length of the damage and deterioration in the drawing 210 on the drawing 210 . , discovery time, occurrence estimation time, etc., all information about damage and deterioration may be covered and included. may be displayed on the smart glass 100 .

The smart glass 100 and the mobile terminal 200 may be used in the field as devices that the inspector directly wears, carries, and brings when inspecting the structure.

On the other hand, the server 300 and the database 400 may be located in a remote central control center or management office. The server 300 is preferably a server 300 equipped with an artificial intelligence function, and can communicate with the smart glass 100 and the portable terminal 200 through a wireless network, so that the smart glass 100 and For the damage and deterioration transmitted by the mobile terminal 200 in the form of an image, the growth rate is read, and for the newly discovered and detected signs of damage and deterioration, the damage or deterioration is read, and the risk of damage and deterioration is determined. It can be performed including the function of deciding whether to notify the relevant authorities.

In the above configuration, when the inspector detects signs of new damage to the concrete structure with the naked eye, the suspect image is taken through the smart glass camera and transmitted to the server, which then processes the suspicious image with artificial intelligence to process the damage Determining whether there is or determining the type of damage may be performed.

However, according to the above-mentioned prior art 1, there is no pre-processing process for the image taken (captured) by the smart glass, and in the case of multiple injuries, it is very important to accurately identify the location between each damaged area is handled and processed separately, so the location of damage cannot be accurately specified on the drawing, and the result is inaccurate when diagnosing the safety of the structure through this.

Prior Art 1: Registered Patent Publication No. 10-2195890 (Title of Invention: Smart Glass Safety Diagnosis System and Safety Diagnosis Method)

Prior art 2: Registered Patent Publication No. 10-21413520 (Title of the invention: Crack width evaluation method in concrete structure using image processing technique)

Prior art 3: Registered Patent Publication No. 10-1922831 (Title of the invention: image analysis apparatus and image analysis method for determining the state of concrete)

The present invention has been devised to solve the above-mentioned problems, and after constructing a digital exterior survey network map (image) containing the locations of multiple damaged parts of a concrete structure based on smart glass, and processing it with deep learning technology, the concrete An object of the present invention is to provide a smart glass-based concrete structure health monitoring method that monitors the health of the structure.

The smart glass-based concrete structure health monitoring method of the present invention for achieving the above object includes the steps of: (a) capturing images of multiple damaged areas of a concrete structure using a camera attached to the smart glass; (b) performing image pre-processing on each of the multiple damaged images; (c) performing local image registration using affine transformation based on mutual information amount for each image of multiple damaged regions that have undergone image preprocessing; Steps (d) of equalizing the brightness value of the overlapping area in the image registration process using histogram equalization and (e) of constructing a digital exterior survey network by combining the images of multiple damaged areas that have passed through steps (d) made including

In the above configuration, the image pre-processing in step (b) includes the process of detecting a boundary line of multiple damaged sites in each multiple damaged site image, removing noise using a filter, and improving image resolution.

The boundary line detection is performed by applying a differential operator to the distribution of brightness values corresponding to the x-axis and y-axis of each multi-damage site image, which is a two-dimensional image, and confirms the point at which the value changes rapidly.

Image resolution enhancement is performed by a power law, sharpening effect filtering, or interpolation.

This includes the step (f) of automatically classifying multiple damage cases of the concrete structure by applying a deep learning-based algorithm to the digital exterior survey network diagram.

According to the smart glass-based concrete structure health monitoring method of the present invention, after capturing images of multiple damaged areas of a concrete structure based on the smart glass, the types of defects, such as cracks, peeling, exfoliation, and layer separation, are based on this. Because it analyzes , white matter, rebar exposure, etc., objective analysis is possible.

In addition, since multiple (multiple) damaged site images are integrated into one image through image registration technology to construct a digital exterior survey network, the location between each damaged area image can be accurately identified, and through this, the type of defect can be identified. can be analyzed more accurately.

In addition, various external factors such as illuminance, noise, and distortion that affect the image quality exist in each damaged area image captured with smart glass. It is possible to construct a high-quality digital exterior survey network by processing using

Smart Glass Safety Diagnosis System Configuration Diagram of Prior Art 1.
Figure 2 is a flow chart for explaining the smart glass-based concrete structure health monitoring method of the present invention.
3 is a view for explaining a boundary line detection process in the smart glass-based concrete structure health monitoring method of the present invention.
4 is a view for explaining a noise removal process through filtering in the smart glass-based concrete structure health monitoring method of the present invention.
5 is a view for explaining an image resolution improvement process in the smart glass-based concrete structure health monitoring method of the present invention.
6A and 6B show the smoothing result and high-frequency filter (HPF) performed using a low-frequency filter (LPF) in the process of noise removal and resolution improvement through frequency analysis in the smart glass-based concrete structure health monitoring method of the present invention, respectively. An example drawing showing an image boundary line detected using
7 is a view for explaining the process of building a digital exterior survey network diagram in the smart glass-based concrete structure health monitoring method of the present invention.
8 is a schematic diagram of a classification model development using high-dimensional features extracted from a convolution filter-based autoencoder learning network in the smart glass-based concrete structure health monitoring method of the present invention.

Hereinafter, a preferred embodiment of the smart glass-based concrete structure health monitoring method of the present invention will be described in detail with reference to the accompanying drawings.

2 is a flowchart for explaining the smart glass-based concrete structure health monitoring method of the present invention, unless otherwise stated, it is revealed that the smart glass CPU is the main body. As shown in Fig. 2, according to the smart glass-based concrete structure health monitoring method of the present invention, the inspector captures an image of the damaged area in the concrete structure to be inspected through the smart glass camera while wearing the smart glass ( photographing) (step S10).

Here, if multiple (multiple) damage occurs in a certain area, images of each damaged area are captured by overlapping a certain portion.

Next, data preprocessing is performed on each of the multiple damage images captured in this way. For example, in the case of an image of damage to an exterior wall of a building, since there are various external factors such as illumination, noise, or distortion that affect the quality of the image In order to remove these factors, image preprocessing such as computer vision-based boundary detection, noise removal, and resolution improvement is required.

To this end, in step S20, a boundary line is first detected for each image of multiple damaged areas. For example, by using a differential operator, that is, by applying a differential operator to the distribution of brightness values corresponding to the x-axis and y-axis of the two-dimensional image. The boundary line is detected by checking the point where the value changes rapidly.

3 is a diagram for explaining the boundary line detection process in the smart glass-based concrete structure health monitoring method of the present invention, and various differential operators, for example, Prewitt, Sobel, and Roberts with respect to the x-axis and y-axis of the example image image. The boundary line detected by applying the operator is indicated.

를 이용하여 계산하여 새로운 영상 g를 생성하는데, 잡음 제거를 위한 평활화(Smoothing) 작업은 주로 평균 필터나 가우시안 필터를 이용하여 수행한다.Returning to FIG. 2 again, in step S30, noise removal using a filter is performed on each image of the damaged area. Filtering is a specific function, that is, a kernel by applying a specific function to a given pixel and its surrounding values. refers to the process of processing. At this time, the convolution between the given image h and the filter f

A new image g is generated by calculating using

4 is a diagram for explaining a noise removal process through filtering in the smart glass-based concrete structure health monitoring method of the present invention, and for explaining an image smoothing process using an average filter and a Gaussian filter It is a drawing.

Returning to FIG. 2 again, the resolution improvement process is performed in step S40. First, as a representative method of improving the image contrast ratio, there are a method using a power law and a method using a filter. If it looks too dark, the sharpness can be improved by applying a power series function to the image. Even if the sharpening filter method is used as well as the power series method, the sharpness of the blurry part of the image can be improved.

5 is a diagram for explaining the image resolution improvement process in the smart glass-based concrete structure health monitoring method of the present invention, (A) is a power series method, (B) is a sharpening effect filtering method, (C) using an interpolation method It is an example drawing showing the result of image resolution improvement.

On the other hand, spatial frequency analysis refers to a method of analyzing an image by changing the dimension of an image from space to frequency. As an example, Inverse Fourier Transform, which transforms spatial-dimensional image data into frequency-dimensional data through Fourier Transform, processes information corresponding to high and low frequencies, and then transforms it back into spatial dimensions Go through the process. In this case, since the high-frequency component mainly means a sharp change in the boundary line, noise, and brightness value, it can be removed by using a low-pass filter (LPF). If you want to detect the edge of the image, you can perform the operation using a high-pass filter (HPF).

6A and 6B show the smoothing result and high frequency filter (HPF) performed using a low frequency filter (LPF) in the process of noise removal and resolution improvement through frequency analysis in the smart glass-based concrete structure health monitoring method of the present invention, respectively. It is an example diagram showing the image boundary line detected using

On the other hand, when taking images of damaged parts of concrete structures, it is difficult to simultaneously grasp images of multiple damaged parts across the entire area due to the limitation of the Field of View (FOV) of the smart glass camera. In addition, it is very important to understand the entire absence of a facility through an exterior survey network diagram in order to express the type and location of defects on the design drawings and abbreviated drawings during safety inspection or safety diagnosis of facilities.

Accordingly, it is necessary to construct a digital exterior survey network diagram for the damaged area image over the entire area of the concrete structure by using the image matching technique that connects the divided images of the damaged area.

7 is a view for explaining the process of constructing a digital exterior survey network diagram in the smart glass-based concrete structure health monitoring method of the present invention.

As shown in FIG. 7, in the present invention, in step S50 of FIG. 2, local image registration is performed based on the amount of mutual information for the multiple damaged images of the concrete structure. Registration enables mapping in a common space based on Affine Transformation for regions sharing spatial information between individual damaged images.

On the other hand, as described above, the brightness of the common shared (overlapping) space (the part where the red dotted line overlaps in FIG. 7) between the damaged area images obtained through local image registration may be different. In step S60 of FIG. The image contrast ratio can be improved by performing brightness value equalization of the overlapping region using histogram equalization.

Next, in step S70 of FIG. 2, the entire digital exterior survey network is constructed by combining each locally matched damaged area image into one, whereby the location of multiple damaged areas in concrete is accurately identified, and through this in the future, the type of defect, etc. can be analyzed more accurately.

Finally, in step S80, a deep learning-based algorithm is applied to the digital exterior survey network obtained in this way to automatically classify multiple damage cases of the concrete structure so that each damage case can be appropriately dealt with.

8 is a schematic diagram of a classification model development using high-dimensional features extracted from a convolution filter-based autoencoder learning network in the smart glass-based concrete structure health monitoring method of the present invention. As shown in Fig. 8, the learning network using an autoencoder based on a convolution filter is based on the characteristics that can be converted from an input image to an image emphasizing a desired feature, each of concrete damage cases. It has the advantage of being able to learn by capturing features. This advantage can extract high-level features from deep learning learning, and this can be used as information representing each damage case.

Next, a multi-class classification model is trained using an existing machine learning classifier based on the high-dimensional features of the extracted concrete damage cases, for example, a support vector machine (SVM), The performance is evaluated using a learning method of a classification model such as a random forest (RF, Randm Forest) and k-nearest neighbor algorithm. Performance evaluation for multi-class may be performed by comparing using sensitivity, specificity, accuracy, and confusion matrix.

Above, a preferred embodiment of the smart glass-based concrete structure health monitoring method of the present invention has been described in detail with reference to the accompanying drawings, but this is merely an example, and various modifications and changes within the scope of the technical spirit of the present invention It will be possible. Accordingly, the scope of the present invention should be defined by the description of the following claims.

For example, in this specification, the singular also includes the plural unless the phrase specifically dictates otherwise. As used herein, "comprises" and/or "comprising" does not exclude the presence or addition of one or more other components, steps, acts and/or elements in a recited element, step, operation and/or element. .

As used herein, an “embodiment,” or the like, is not to be construed as an advantage that any aspect or design described is preferred or advantageous over other aspects or designs.

Also, the term 'or' means an inclusive or rather than an exclusive or.

Also, the expressions "a" and "a", "a" and "a", as used in this specification and the claims, are to be construed to mean "one or more" in general, unless stated otherwise, or unless it is clear from the context that it relates to the singular form.

Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used with the meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless clearly defined in particular.

Meanwhile, in describing the present invention, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

Claims (5)

delete It is performed by the CPU of Smart Glass,
(a) overlapping each of the multiple damaged site images of the concrete structure by using a camera attached to the smart glass;
(b) performing image pre-processing on each multi-damage image for computer vision-based boundary detection, noise removal, and resolution improvement;
Local image registration is performed based on the amount of mutual information for each image of multiple damaged areas that have undergone image preprocessing, and the image registration is an affine transformation ( (c) performing local image registration that enables mapping in a common space based on affine transformation;
(d) improving the image contrast ratio by equalizing the brightness values of the overlapping regions in the image registration process using histogram equalization;
Step (e) of constructing a digital appearance survey network by combining the images of each multiple damage site that has undergone (d) step, and
(f) automatically classifying multiple damage cases of the concrete structure by applying a deep learning-based algorithm to the digital exterior survey network diagram;
The image preprocessing in step (b) is performed by applying a differential operator to the distribution of brightness values corresponding to the x-axis and y-axis in each multi-injured part image, which is a two-dimensional image, and confirms the point at which the value changes rapidly. Detects the boundary of multiple damaged areas, applies a specific function kernel to a given pixel and its surrounding values, and removes noise using a filter that processes the pixel information. Power Law, sharpening effect filtering Or it is made including the process of improving the resolution of the image performed by the interpolation method,
The resolution improvement process is the Inverse Fourier Transform, which transforms the spatial dimension image data into the frequency dimension through Fourier Transform, processes the information corresponding to high and low frequencies, and then transforms it back to the spatial dimension. This is done through a process. Since high-frequency components mainly mean abrupt changes in boundary lines, noise, and brightness values, a low-pass filter (LPF) is used to remove them. A smart glass-based concrete structure health monitoring method that performs work using -Pass Filter; HPF). delete delete delete

Images