JP7517641B2 - Structure inspection and repair support system - Google Patents

Description

The present invention relates to a structure inspection and repair support system that provides information for inspecting and repairing structures such as concrete structures.

There are many roads and other infrastructure structures in Japan that were constructed during the period of high economic growth, and the proportion of aging structures in the total is rapidly increasing. Appropriate inspection and repair are essential to maintain and manage aging structures. When damaged areas are discovered, experts conduct visual inspections to determine the damage rank and decide on a repair policy. Rapid and accurate assessment is essential, especially for damaged areas that require urgent repair. However, there is currently an issue of a shortage of engineers due to factors such as a declining birthrate and aging population, and efficient maintenance and management is required.

In light of these circumstances, Japanese Patent No. 6347384 (Patent Document 1) discloses a system in which, when inspection data such as deformation images acquired during bridge inspections are registered in the system, past inspection data is searched for and similar cases are presented. Specifically, in this system, a co-occurrence information storage unit stores co-occurrence information, which is the probability of co-occurrence with deformation information of damaged parts, and a list of recommendation rankings based on the deformation information to be processed and the co-occurrence conditions received in advance is referenced to the co-occurrence information stored in the co-occurrence information storage unit to generate a list of recommendation rankings based on co-occurrence, and an image similarity list is generated by executing a process of determining the image similarity between the deformation image information in the deformation information to be processed and the deformation image information in the list of recommendation rankings based on the generated co-occurrence, and a list of recommendation rankings based on co-occurrence and the image similarity list of the deformation information that is highly similar to the deformation information to be processed is generated. Here, the co-occurrence information is information indicating the co-occurrence probability P(x|y) of another event x occurring under the condition that an event y occurs. This system then accepts the selection of the newly registered unconfirmed damage information (query abnormality information) to be processed. This selection may be made by accepting input of a selection operation by the user, or may be automatically accepted by some method.

Also, Japanese Patent No. 5443055 (Patent Document 2) discloses a concrete diagnosis system that diagnoses the degree of deterioration of reinforced concrete based on the concentration of deterioration factors on the surface of the reinforced concrete, the cover of the reinforced concrete, and the number of years of service of the reinforced concrete. In this system, an area to be diagnosed is set on a previously captured image of the target reinforced concrete structure to be diagnosed, and the set area is divided into meshes. For each mesh of the divided area, the concentration of deterioration factors on the surface of the reinforced concrete and the cover of the reinforced concrete are input and set, and the number of years of service of the reinforced concrete is input and set. The deterioration level is calculated for each mesh based on the input values, and each mesh is color-coded according to the degree of deterioration calculated for each mesh and displayed on the display unit.

Furthermore, Japanese Patent No. 6042313 (Patent Document 3) discloses a deterioration detection device that automatically detects deteriorated parts on the surface of a concrete structure from an image of the concrete structure by image processing. This conventional device extracts different deteriorated part candidate pixels by detecting edges in different directions based on the image of the concrete structure, generates a noise-removed image by calculating the logical product of the extracted deteriorated part candidate pixels, and detects deteriorated parts from the noise-removed image.

Patent No. 6347384 Patent No. 5443055 Patent No. 6042313

However, in the invention of Patent Document 1, the undetermined damaged parts are identified by the inspector from the images he or she has taken, so the inspector must be highly skilled in identifying the undetermined damaged parts. Also in the invention of Patent Document 2, the inspector must identify the damaged parts. In the invention of Patent Document 3, deteriorated parts that have appeared on the surface of a concrete structure are automatically detected by image processing, but this automatic detection does not utilize the knowledge of an expert, and the inspector cannot know the inspection area (area of interest) of the deteriorated parts that he or she should actually inspect.

The objective of the present invention is to provide a structure inspection and repair support system that can automatically extract and display an appropriate inspection area (area of interest) for inspection from an image including damaged areas captured by an inspector.

Another object of the present invention is to provide a structure inspection and repair support system that searches for past damage images similar to images of damaged areas in an extracted inspection area (area of interest) from image features and presents the results, thereby enabling appropriate support for future maintenance and management of the damage.

Another object of the present invention is to provide a structure inspection and repair support system that can improve search accuracy by using image features that combine shape features and color features.

The structure inspection and repair support system of the present invention includes an image data acquisition unit, a data storage unit, a region of interest determination unit, and a display data creation unit. The image data acquisition unit acquires image data including one or more target damaged parts, obtained by photographing a damaged part in a concrete structure to be inspected and the vicinity of the damaged part with a photographing device. The data storage unit accumulates image data of the multiple damaged parts obtained by past inspections and damage data of the multiple damaged parts including at least data of the inspection range corresponding to the image data of the multiple damaged parts. The region of interest determination unit receives the image data including one or more target damaged parts as input, and determines the inspection range to be inspected by an inspector for one or more target damaged parts as a region of interest using a learning model created using image data including at least the multiple damaged parts obtained by past inspections and data including the inspection range of the multiple damaged parts determined by an expert in inspection or repair work corresponding to the image data including the multiple damaged parts as teacher data. Then, the display data creation unit displays a display image in which the region of interest is displayed on the image of one or more target damaged parts on the display screen of the image display device.

According to the present invention, a learning model is created using data including the inspection ranges of multiple damaged areas determined by an expert in inspection or repair work as training data, and the inspection range that should be inspected by an inspector for one or more target damaged areas is determined as an area of interest, so that an appropriate inspection range can be automatically presented to the inspector.

The region of interest determination unit includes a learning model constructed to estimate the probability of the presence or absence of damage in and around the damaged portion as output using image data including multiple damaged portions obtained by past inspections and data including the inspection range of the multiple damaged portions determined by an expert in inspection or repair work corresponding to the image data including the multiple damaged portions as teacher data, and a probability region determination unit that determines, as the region of interest, a region including a portion where the probability of the presence or absence of damage in the target damaged portion and the surrounding region of the target damaged portion output from the learning model using image data including the target damaged portion as input, exceeds a predetermined value. According to the present invention, the learning model outputs the probability of the presence or absence of damage, and the region of interest is determined to be a region including a portion where the probability of the presence or absence of damage exceeds a predetermined value, so that the region of interest can be determined with high accuracy.

The display data creation unit creates, as a display of the region of interest, a display showing the contour shape of the area to be inspected by the inspector or a display showing a rectangular shape surrounding the contour. If it is a contour shape, the inspection range can be made small. Also, if it is a rectangular shape, the inspector's inspection work will be easier.

The system may further include a region of interest correction unit that stores data obtained by manually correcting the region of interest determined by the region of interest determination unit in the data storage unit as data on the inspection range by an inspector. In this way, it becomes possible to correct the inspection results of a newly inspected damaged area and reflect them in subsequent inspections.

The image display device may further include an image feature amount calculation unit that calculates image feature amounts of an image including one or more target damaged parts and image feature amounts of an image including multiple damaged parts from image data including one or more target damaged parts and image data including multiple damaged parts, and a similar image selection unit that selects multiple images including multiple damaged parts similar to the image including one or more target damaged parts from the calculation result of the image feature amount calculation unit. The display data creation unit preferably creates display data that selectably displays the images of the multiple damaged parts selected by the similar image selection unit on the display screen of the image display device. When the images of the multiple damaged parts similar to the image of the target damaged part are selectably displayed based on the image feature amount, it is possible to find past similar data necessary for repairing the target damaged part, including the inspector's experience, from the data of the multiple past damaged parts selected based on the image feature amount. In this case, the data storage unit stores repair history information of the multiple damaged parts corresponding to image data including the multiple damaged parts obtained by performing inspection in the past, and the display data creation unit preferably creates display data that displays repair history information corresponding to the image of the damaged part selected from the image including the multiple damaged parts displayed on the display screen on the display screen. In this way, the inspector is more likely to obtain appropriate repair history information. The data storage unit preferably stores damage class data that represents the degree of damage of a plurality of damaged parts by a damage class corresponding to image data including the plurality of damaged parts obtained by past inspections, and the display data creation unit preferably creates display data that displays the damage class on the display screen together with the images of the plurality of damaged parts. In this way, the inspector can use the damage class as a guide to select an image of a past damaged part that is similar to the target damaged part.

The image features calculated by the image feature calculation unit are a combination of shape features and color features.

When the image data of the target damaged part and the image data of the multiple damaged parts are RGB image data, the image feature calculation unit converts the RGB color space of the RGB image data into the L*a*b* color system, separates only the a*b* component from the L*a*b* components from the data converted into the L*a*b* color system, quantizes the separated a*b* component using a quantization model, classifies the quantized color image obtained by quantization into two types, color area I and color boundary B, and preferably extracts the occurrence frequency of each quantized color for the classified color area I and color boundary B as a color feature (quantized BIC feature). When color features are extracted in this way, as long as the image is taken with a certain degree of recognizable brightness, it is possible to extract features that focus on color differences based on the data, regardless of changes in the brightness of the image. Therefore, the color features make it possible to extract features from image data without being affected by shooting conditions such as whether the image was taken on a sunny or cloudy day.

The image feature calculation unit may also convert the RGB image data into grayscale image data, remove noise from the converted grayscale image data, extract contours from the noise-removed grayscale image data, and extract HLAC features as shape features from the contour-extracted grayscale image data. Extracting shape features in this manner makes it easy to extract shape features.

It is preferable that the image feature calculation unit concatenates the shape feature and the color feature, normalizes the concatenated feature, and outputs it as the image feature. If the normalized image feature is used, the image feature can be used regardless of the size of the image.

1 is a block diagram showing a configuration of a first embodiment of a structure inspection and repair support system according to the present invention; 13A to 13D are diagrams used to explain damage images and inspection ranges that were actually inspected in the past. 13A and 13B are diagrams showing shapes of a region of interest other than a rectangle. 10 is a flowchart showing steps of a damage image storage process in a data storage unit. 13 is a flowchart showing the processing steps in a region of interest determining unit and the processing steps for creating a learning model. 6 is a flowchart showing details of steps ST23 and ST24 of the learning process in FIG. 5 . 13A and 13B are diagrams showing examples of a correct rectangular image and a damage image inspection range. FIG. 13 is a diagram showing the first half of the convolution calculation process in an SSD. FIG. 13 is a diagram showing the latter half of the convolution calculation process in the SSD. FIG. 13 is a diagram used to explain different aspect ratios for each grid position of a filter list in an SSD. 6 is a flowchart showing details of step ST13 of the estimation process in FIG. 5 . FIG. 1A is a diagram showing an example of a target damage image, and FIG. 13 is a flowchart of a learning process for constructing a learning model using a CNN-based VGG model. A figure showing damage image data and the corresponding damage image inspection range for constructing a learning model consisting of a VGG model. 1 is a flowchart of region of interest estimation using a CNN-based learning model. FIG. 1 is a diagram illustrating an example of a network structure used in CNN calculation. 1 is a diagram showing, with images, the process of determining a region of interest using a CNN technique. FIG. 13 is a diagram showing a comparison between the SSD model and the VGG model as learning models. FIG. 11 is a block diagram showing a configuration of a second embodiment of the present invention. 5A to 5C are diagrams used to explain the operation of a region of interest correction unit. FIG. 13 is a block diagram showing a configuration of a third embodiment of the present invention. 10 is a flowchart showing steps of a display data creation process in a display data creation unit. FIG. 2 is a diagram showing an example of a display on a display screen of an image display device. 13A and 13B are diagrams used to explain an example of image repair history information of a damaged portion. FIG. 13 is a diagram showing an example of an algorithm when the image feature amount calculation unit is realized by software. FIG. 1A is a diagram showing a process for creating a color quantization model, and FIG. 1B is a diagram showing an algorithm for determining a quantized BIC feature. FIG. 1 is a diagram illustrating a concept of color information extraction. FIG. 1 is a diagram illustrating a concept of determining a representative color. 1 is a diagram used to explain an example in which color information of all pixels in a color-quantized image is classified into I (within a color region) and B (color boundary). FIG. FIG. 13 is a diagram illustrating an example of calculating quantized BIC features.

The following describes in detail an embodiment of the structure inspection and repair support system of the present invention with reference to the drawings.

[First embodiment]
FIG. 1 is a block diagram showing the configuration of a first embodiment of the structure inspection and repair support system of the present invention. The structure inspection and repair support system 1 of the first embodiment includes an image data acquisition unit 3, a data storage unit 5, a region of interest determination unit 7, a display data creation unit 9, and an image display device 11. In this embodiment, the main units from the image data acquisition unit 3 to the display data creation unit 9 are realized inside the computer by software installed in the computer. The image data acquisition unit 3 acquires image data including one or more target damaged parts a1 to a3, which are photographed by a photographing device such as a digital camera of a damaged part and the vicinity of the damaged part in an inspection target concrete structure as shown in FIG. 2(A). The damaged parts are rebars exposed from the concrete structure, cracks or lifts in the concrete, cracks in the steel structure, and the like.

The main part of the image data acquisition unit 3 is composed of a computer input port. In this embodiment, image data including one or more target damaged parts acquired by the image data acquisition unit 3 is input to the region of interest determination unit 7. The data storage unit 5 stores image data of multiple damaged parts obtained by past inspections and damage data of multiple damaged parts (inspection range actually inspected, damage location, repair history, damage ranking, etc.) including at least data of the inspection range corresponding to the image data of the multiple damaged parts. Figure 2 (C) shows an example of image data of a damaged part obtained by a past inspection. Figure 2 (D) shows image data in which the inspection range D1 actually inspected in the past is displayed on the damage image of Figure 2 (C).

Figure 4 is a flowchart showing the steps of the damage image accumulation process in the data accumulation unit 5. In step ST1, past damage image data is read. In step ST2, the inspection area D1 where an expert actually performed inspection or repair work is specified to be displayed on the damage image. Then, in step ST3, the damage image and inspection area data are accumulated in a damage image database constructed in the data accumulation unit 5.

The region of interest determination unit 7 receives image data including one or more target damaged parts a1-a3 as input, and determines the inspection range that an inspector should inspect for one or more target damaged parts as regions of interest b1-b3 [see FIG. 2(B)] using a learning model created using image data including at least multiple damaged parts obtained by past inspections and data including inspection ranges of multiple damaged parts previously determined by an expert in inspection or repair work as teacher data corresponding to the image data including the multiple damaged parts. The display data creation unit 9 creates display data that displays a display image in which the region of interest is displayed on the image of the target damaged part on the display screen of the image display device 11. In this embodiment, the display data creation unit 9 stores the image data, which is the display data, and the region of interest to be inspected in the data storage unit 5 as new data.

As shown in Figures 3(A) and (B), the areas of interest b4 and b5 displayed on the display screen of the image display device 11 may have a shape other than a rectangle. The areas of interest are the range of the area that an inspector should refer to (or confirm or visually confirm) when determining damaged parts during an inspection, and provide useful information to compensate for the experience of technicians with little inspection experience, and provide an objective indicator based on past data for experienced technicians with a lot of inspection experience.

Figure 5 is a flowchart showing the processing steps in the region of interest determination unit 7 and the processing steps for creating a learning model. First, in step ST11, image data including one or more target damaged parts a1 to a3 is read. Then, in step ST12, a learning model of the region of interest is read, in step ST13 the region of interest is estimated, and in step ST14 the region of interest is determined. Specifically, the learning model estimates the probability of the presence or absence of damage from within the image. Then, depending on the probability of the presence or absence of damage, a region of interest (inspection region) is determined based on whether or not it exceeds a threshold. This means that the likelihood of the region of interest location is calculated from within the target image input by the learning model.

In the process of creating a learning model, in step ST21, multiple accumulated image data of multiple damaged parts from the past are read out, and in step ST22, data including the inspection range of the multiple damaged parts previously determined by an expert in inspection or repair work is read out as teacher data corresponding to the image data including the multiple damaged parts obtained by performing a previous inspection. Then, in step ST23, a learning process is executed, and in step ST24, a learning model to be used for estimating the region of interest is generated. In step ST24, learning parameters of the region of interest learning model designed for determining the region of interest are determined.

[Determining Regions of Interest Using Learning Models]
Any method can be used to generate a learning model for determining a region of interest and estimate the region of interest using the learning model. For example, in this embodiment, a method using SSD (Single Shot multi-box Detector) and a method using VGG are used. These methods are described in detail in the following paper.

(SSD literature) Liu, Wei, et al. "SSD: Single Shot MultiBox Detector." arXiv preprint arXiv:1512.02325 (2015).
(VGG literature) K. Simonyan and A. Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recoginition. In Intl. Conf. on Learning Repre-sentations (ICLR), 2015.

The processing of the region of interest determination unit 7 uses the learning model for region of interest estimation generated by the region of interest estimation model generation process to estimate a region of interest for the acquired image data including the target damaged part and its vicinity read from the image data acquisition unit 3, and determines the inspection region that should be referred to (or confirmed or visually recognized) by the inspector when determining the damaged part within the damage image including the target damaged part and its vicinity.

Figure 6 is a flow chart showing the details of step ST23 of the learning process in Figure 5. In this learning process, first, a correct rectangular image is generated (ST231) using the multiple damage image data and the corresponding damage image inspection range that have been read. As shown in Figures 7 (A-1) and (A-2) and Figures 7 (B-1) and (B-2), the correct rectangular image is a binary image in which the rectangular area including the damage image inspection range stored in the data storage unit 5 is set to 1 and the other areas are set to 0. If the damage image inspection range is not rectangular, the correct rectangular image is a binary image in which the rectangular area including the range is set to 1 and the other areas are set to 0, as shown in Figures 7 (C-1) and (C-2) and Figures 7 (D-1) and (D-2).

[Processing in learning model using SSD model]
A data set of multiple damage images and corresponding correct rectangular images is used as training data to train a Single Shot Multi-box Detector (SSD) model. The SSD model is one of the deep learning models aimed at object detection in images, and is a method that operates quickly and accurately in general object detection. A Convolutional Neural Network (CNN) is used to generate feature maps with multiple different scales obtained in the process. For each position in these feature maps, the relative position and likelihood with respect to the target object are calculated while changing the aspect ratio.

In learning, the feature map output by repeatedly applying convolution operations in the SSD model to all training data input images (damage images in this example) is divided onto a grid, and for each position, the overlap between candidate regions of interest with different aspect ratios (ratio of length to width) and the correct answer box calculated from the corresponding correct answer rectangular image is calculated. Learning proceeds while adjusting the parameters of the SSD model to increase the overlap, and the SSD model after a predetermined number of repetitions is used as the learning model for estimating the region of interest.

The convolution operation is processed in the order of Fig. 8 and Fig. 9. Convolution, batch normalization, activation, and maximum pooling are applied to the input image in sequence. The convolution operation in Fig. 8 is expressed as conv[layer number 1]_[layer number 2 (if any)]_[number of filters], and VGG Output represents the output result of the convolution operation in Fig. 8. (For example, in Fig. 8, conv1_16 represents the first convolution layer with 16 filters). The output obtained in Fig. 8 is input to the process shown in Fig. 9. In Fig. 9, convolution is applied to the input as in Fig. 8, but the output results in each layer are saved in the filter list fs. In other words, the filter list stores the sets of filter images after the convolution operation output from the 4th, 7th, 8th, 9th, 10th, and 11th convolution layers. Each of these filters is divided into a grid as shown in Fig. 10, and the probability of the presence or absence of a region of interest is estimated based on the aspect ratio set in advance for each grid position.

Figure 11 shows details of step ST13 of the estimation process in Figure 5. In the estimation process in Figure 11, a damage image is input to a learning model for estimation, and a feature map output by repeatedly applying convolution operations in the learning model (SSD model) is divided into a grid, and the likelihood of candidate regions of interest with different aspect ratios (ratio of length to width) is calculated for each position. Based on a specified likelihood threshold, candidates with a likelihood equal to or greater than the threshold are narrowed down to be determined as the region of interest. As shown in Figure 12 (A), for a target damage image containing two damaged areas, the image display device 11 displays the results of estimating the two regions of interest shown in Figure 12 (B). The estimation result in Figure 12 (B) is the region of interest when the likelihood threshold is set to 0.5.

[Processing in learning model using VGG model]
FIG. 13 is a flowchart of a learning process for constructing a learning model by a VGG model composed of a convolutional neural network (CNN). In the learning process of the learning model by the VGG model, first, a correct image is generated using the multiple damage image data and the corresponding damage image inspection range that have been read (ST231'). As shown in FIGS. 14 (A-1) and (A-2) to 14 (D-1) and (D-2), the correct image is a binary image in which the area including the damage image inspection range stored in the data storage unit is set to 1 and the other areas are set to 0. If the damage image inspection range is not rectangular, the correct image is a binary image in which the rectangular area including the range is set to 1 and the other areas are set to 0. The VGG model is learned by using a data set of correct rectangular images corresponding to multiple damage images as training data for learning (ST232').

In object detection based on CNN, network processing is performed up to the fully connected layer, and high-dimensional vectors corresponding to the pixels that make up the image obtained as output are reconstructed as a likelihood map. In learning, steps ST31 to ST34 in FIG. 15 described later are performed, and convolution processing and pooling processing are sequentially applied in the VGG model to all input images of training data (damage images in this embodiment), and the output vector of the fully connected layer in the final layer is used as a likelihood map of the damaged part, and the network parameters are learned based on the error with the correct image.

Figure 15 is a flowchart of estimating a region of interest using a learning model (VGG model). In this flowchart, in step ST31, an image from which a region of interest is to be extracted is read. Then, in step ST32, the input image is normalized. That is, the size of the image is changed to 256 x 256 [px], and the pixel values are standardized so that the average is 0 and the variance is 1. Next, in step ST33, CNN calculation is performed using the VGG model. In the CNN calculation, convolution processing and maximum value pooling processing are sequentially applied to the normalized image using the network structure of the VGG model shown in Figure 16, and the output of the fully connected layer in the final layer is output as a one-dimensional array. Next, in step ST34, the one-dimensionally arranged output of the fully connected layer is reconstructed into two dimensions. That is, the one-dimensional array output in the CNN calculation is reconstructed into an image-like two-dimensional array and output as a likelihood map. Next, in step ST35, a binarization process is performed. In this binarization process, a binarization process is applied in which the value of each element of the likelihood map is set to 1 if it exceeds a threshold value, and 0 if it does not. Next, in step ST36, morphological processing is performed. In morphological processing, the binarized likelihood map is subjected to erosion and expansion processing to remove noise and fill holes. Finally, in step ST37, a region of interest is extracted. In this step, a rectangle that includes an area where pixels showing 1 are connected by blob analysis of the likelihood map to which morphological processing has been applied is extracted as the region of interest. Figure 17 shows images of the process of determining the region of interest using the VGG model.

Figure 18 shows a comparison of the SSD model and the VGG model as learning models. The SSD model has low scale dependency and high accuracy, but has a slow processing speed. On the other hand, the VGG model has a simple network configuration, so it has a fast processing speed, but its estimation accuracy is inferior to that of the SSD model. For example, for an inexperienced engineer, the SSD model is suitable because the accuracy of the region of interest estimation is important, even if it takes time, but for an experienced engineer, the results of the speed-oriented VGG model can be corrected later in the region of interest correction unit described below.

According to the first embodiment, a learning model is created using training data including inspection ranges for multiple damaged areas previously determined by an expert in inspection or repair work, and the inspection range that should be inspected by an inspector for one or more target damaged areas is determined as an area of interest, so that an appropriate inspection range can be automatically presented to the inspector.

In addition, according to this embodiment, the learning model used in the region of interest determination unit 7 outputs the probability (likelihood) of the presence or absence of damage, and the region containing the portion where the probability of the presence or absence of damage exceeds a predetermined value is determined as the region of interest, so that the region of interest can be determined with high accuracy.

[Second embodiment]
FIG. 19 is a block diagram showing the configuration of the second embodiment of the present invention. In FIG. 19, the same components as those shown in FIG. 1 showing the configuration of the first embodiment are denoted by the same reference numerals as those in FIG. 1, and the description thereof will be omitted. In this embodiment, the region of interest correction unit 13 is further provided, which stores data obtained by manually correcting the region of interest determined by the region of interest determination unit 7 by an inspector in the data storage unit 5 as data of the inspection range. In this embodiment, even if the region of interest correction unit 13 does not correct the region of interest, the data created by the display data creation unit 9 is stored in the data storage unit 5. FIGS. 20(A) to (C) are diagrams used to explain the operation of the region of interest correction unit 13. When the region of interest determination unit 7 estimates the regions of interest b4 and b5 as shown in FIG. 20(A), in this embodiment, if the inspector feels that the region of interest b5 should be expanded a little more based on his actual inspection results, the region of interest b5 can be corrected to a region of interest b5' by, for example, moving the cursor as shown in FIG. 20(B). This correction enables the size and position of the region of interest to be changed.

In this way, it becomes possible to reflect the inspection results of newly inspected damaged areas as learning data to be used in future inspections. As a result, even if the accuracy of the area of interest presented by this system is poor, the accuracy can be improved by the intervention of an engineer.

[Third embodiment]
FIG. 21 is a block diagram showing the configuration of the third embodiment of the present invention. In FIG. 21, the same components as those shown in FIG. 1 showing the configuration of the first embodiment are denoted by the same reference numerals as those in FIG. 1, and the description thereof will be omitted. In this embodiment, the image feature amount calculation unit 15 obtains the image feature amount of an image including one or more target damaged parts and the image feature amount of an image including multiple damaged parts from image data including one or more target damaged parts and image data including multiple damaged parts, and further includes a similar image selection unit 17 that selects multiple images including multiple damaged parts similar to the image of one or more target damaged parts from the calculation result of the image feature amount calculation unit 15. The display data creation unit 9 creates display data for displaying a display image in which a region of interest is displayed on an image including a target damaged part on the display screen of the image display device 11, and also creates display data for selectably displaying an image including multiple damaged parts selected by the similar image selection unit 17 on the display screen of the image display device 11. FIG. 22 is a flowchart showing the steps (steps ST41 to ST43) of the display data creation process in the display data creation unit 9.

FIG. 23 is a diagram showing an example of the display on the display screen of the image display device 11. In the display example of FIG. 23, the detected region of interest is displayed by a rectangular frame on the image including the target damaged part in the "photographed image" column. In the "similar image search result" column, images of multiple damaged parts with high similarity are displayed as a display list in order of similarity, and the "attached information" column further describes the damage rank and the name of the damaged location. In this embodiment, four images of multiple similar damaged parts are displayed in a selectable manner, and by selecting an image, image repair history information of the damaged part is displayed on the display screen. Note that the images of multiple damaged parts do not necessarily need to be displayed in order of similarity. FIGS. 24(A) and 24(B) are examples of image repair history information of damaged parts, respectively. In FIGS. 24(A) and 24(B), it is shown that the damage has progressed from the state of "damage image 1" to the state of "damage image 2", and the "repair image" shows an image after the state of "damage image 2" has been repaired. By looking at this information, the inspector can obtain materials for determining the necessity of repair and the repair method. To achieve this, the data storage unit 5 stores repair history information for a plurality of damaged portions in association with image data including the plurality of damaged portions obtained through past inspections.

The data storage unit 5 also stores damage class data that indicates the degree of damage of a plurality of damaged parts by damage class corresponding to image data of the plurality of damaged parts obtained by past inspection. The display data creation unit 9 creates display data that displays on the display screen the repair history information and damage class corresponding to an image of a damaged part selected from images including the plurality of damaged parts displayed on the display screen. In addition, the proportion of each class is displayed as a "rank proportion" in a pie chart in the "Rank Proportion of Top 20 Similar Images" column in the display screen of Fig. 23. In this way, the inspector can select an image including a past damaged part that is close to the target damaged part using the damage class as a guide.

As in this embodiment, when images of multiple damaged parts similar to the image of the target damaged part are displayed in a selectable manner based on image features, it is possible to find similar past data that can serve as a reference for repairing the target damaged part, including the inspector's experience, from the data of multiple past damaged parts selected based on the image features.

[Explanation of Image Feature Calculation Unit]
The image feature amount calculation unit 15 in FIG. 21 calculates the combination of the shape feature amount and the color feature amount as the image feature amount.

(Image Pre-processing)
To reduce the effects of brightness fluctuations, the following preprocessing is performed on the shape features and color features.

Before calculating the shape features, the image is converted into a gradient image and the relative values between adjacent pixels are used to eliminate the effects of changes in the absolute amount of pixel values due to fluctuations in brightness. The gradient image is created by first applying noise removal using a Gaussian filter and then applying a Laplacian operator to the image. Shape features are extracted from this gradient image, resulting in shape features that are robust to fluctuations in brightness.

Before calculating color features, a color space conversion is applied to the image, separating it into brightness and color components, discarding the brightness component, and extracting features from only the color components, resulting in color features that are robust to variations in brightness. Color space conversion changes an image composed of three RGB channels into three channels expressed by the L*a*b* color system. In this case, the L* component represents brightness, and the a*b* component represents color.

As for angle correspondence, since the direction of the damaged area is not uniquely determined, it is necessary to use features that are robust to changes in angle. As for shape features, post-processing that absorbs the rotation of the subject is applied after feature extraction. As for color features, since the frequency of color appearance is calculated from the entire image, there is no change in feature values due to the direction of rotation, so no special processing is applied.

Regarding size compatibility, it is expected that the size of the damaged part captured in the image will differ depending on the distance of the photographer from the damaged part when the damage image is captured, and the absolute size of the damage. In order to treat each damage of different sizes under similar conditions, feature normalization is applied after feature extraction. The shape and color features applied in this embodiment are calculated as statistics, so they can be normalized by dividing by the total number of pixels in the image.

[Details of image feature amount calculation unit]
FIG. 25 shows an example of an algorithm when the image feature amount calculation unit 15 is realized by software. In the example of FIG. 25, image data including a target damaged part and image data including a plurality of damaged parts are cut out as RGB image data (step ST51). First, shape feature amount extraction is performed in steps ST52A to ST55A. In step ST52A, the RGB image is converted into a grayscale image. Then, in step ST53A, noise is removed from the image converted into the grayscale image by a Gaussian filter. Then, in step ST54A, edges (contours) are extracted by a Laplacian filter. Finally, in step ST55A, HLAC feature amount extraction is performed. The HLAC feature amount is a well-known higher-order local autocorrelation feature (HLAC). In the HLAC feature amount extraction, 35 types of HLAC mask patterns are applied to the gradient image, and 35-dimensional shape feature amount is calculated. After calculating the feature amount, the mask patterns that give the same shape when rotated by 45° for each shape are integrated to form rotation and inversion invariant features.

Next, the image feature calculation unit 15 extracts color features in steps ST52B to ST54B. In step ST52B, the RGB color space of the RGB image data is converted to the L*a*b* color system. Next, in step ST53B, only the a*b* component is separated from the L*a*b* components from the data converted to the L*a*b* color system. Then, in step ST54B, the separated a*b* component is quantized using a quantization model. The quantized color image obtained by quantization is classified into two types, color area I and color boundary B, and the occurrence frequency of each quantized color for the classified color area I and color boundary B is extracted as a color feature (quantized BIC feature).

The quantized BIC feature is calculated according to the flow shown in Figures 26(A) and (B). The steps in the figure are as follows. In step ST60, a color quantization model is created in advance from a group of images in the image database of the data storage unit 5.

[Creating a Color Quantization Model]
The color quantization model is created according to the procedure shown in FIG. 26(A). FIG. 27 is a diagram showing the concept of color information extraction, and FIG. 28 is a diagram showing the concept of representative color determination. First, in step ST60A, a group of multiple images is read from the image database. Next, in step 60B, as shown in FIG. 27, conversion from RGB color space to L*a*b* color space is performed for all input images. Next, in step ST60C, as shown in FIG. 27, extraction of color information is performed. That is, only color information of a*b* is extracted from the group of images expressed in L*a*b*. Next, in step 60D, a color quantization model is calculated, and in step ST60E, the color quantization model is output. As shown in FIG. 28, a group of color points is obtained from multiple damage images. That is, n points (for example, 10,000 points) are randomly selected from the group of images with color information of a*b*. A "point" is a two-dimensional vector composed of a*b* and represents a color. From this, a color quantization model is created. That is, for n randomly selected points, a representative color is determined by a clustering method such as k-means. The value of k is quantized to 32, for example. Fig. 28 shows an example of a representative color (k=3).

[Calculation of quantized BIC features]
In step ST61, the color components (a*b* components) image obtained by separating the color information in step ST53B are quantized based on the color quantization model described above. Next, in step ST63, the color information of all pixels in the color quantized image is classified into I (within the color area) and B (color boundary), as shown in Fig. 29. In this example, a color quantization model of three representative colors is used.

The classification criteria are as follows:

- If the four neighboring areas above, below, left and right are the same color, it is I.

If even one of the four neighborhoods contains a different color, classify it as B.

The rightmost figure in Figure 29 is an example of the results of classification using the above classification criteria.

Then, in step ST64, the quantized BIC feature is calculated. That is, as shown in FIG. 30, the occurrence frequency of the quantized colors for each of the classifications I and B is calculated, and in step ST65, it is output as a quantized BIC feature vector.

If the quantized BIC feature vector obtained in this way is used as a color feature, then as long as the image was taken with a certain degree of brightness that can be recognized, it is possible to extract a feature that focuses on color differences based on the data, regardless of changes in the brightness of the image.

The BIC feature is also introduced in the following literature.
“R. Stehling, et al. “A compact and efficient image retrieval approach based on border/interior pixel classification”, Proceedings of the 11th international conference on information and knowledge management”.

As shown in FIG. 25, the image feature calculation unit 15 concatenates the shape feature and the color feature in step ST56, normalizes the concatenated feature in step ST57, and outputs it as an image feature in step ST58. By using the normalized image feature, the image feature can be used regardless of the size of the image.

[Explanation of Similar Image Selection Unit and Display Data Creation Unit]
The similar image selection unit 17 performs a feature-based similar image search. For example, the Euclidean distance between the image feature of the search target image and the image feature in the database is compared, and the top n images with the shortest distance are presented as search candidates. The similar image search results are then read, and a similar image list, repair information list, and damage rank list are generated for the n search candidates.

The display data creation unit 9 also displays, on the image display device, information associated with the search result images. The display data creation unit 9 may display the top search result rank ratios, which display the ranks of the search result images as a percentage, or may display the top search result rank tally, which displays the search results for each rank. The display data creation unit 9 may also present information such as the repair history of the database, DB ancillary information, and repair charts (treatment, prognosis, etc.), and may display detailed information on damage and repair information for each search result.

According to the present invention, a learning model is created using training data including inspection ranges for multiple damaged areas previously determined by an expert in inspection or repair work, and the inspection range that should be inspected by an inspector for one or more target damaged areas is determined as an area of interest, so that an appropriate inspection range can be automatically presented to the inspector.

REFERENCE SIGNS LIST 1 Structure inspection and repair support system 3 Image data acquisition unit 5 Data storage unit 7 Region of interest determination unit 9 Display data creation unit 11 Image display device 13 Region of interest correction unit 15 Image feature amount calculation unit

Claims (7)

an image data acquisition unit that acquires image data including one or more target damaged portions obtained by photographing a damaged portion and the vicinity of the damaged portion in a concrete structure to be inspected with an imaging device;
a data storage unit that stores image data of a plurality of damaged parts obtained by performing past inspections and damage data of the plurality of damaged parts including at least data of an inspection range corresponding to the image data of the plurality of damaged parts;
a region of interest determination unit that receives the image data including the one or more target damaged parts as an input, and determines an inspection range that an inspector should inspect for the one or more target damaged parts as a region of interest by using a learning model created using image data including at least the multiple damaged parts obtained by performing an inspection in the past and data including the multiple damaged parts determined by an expert in inspection or repair work in response to the image data including the multiple damaged parts as teacher data;
a display data generating unit that generates display data for displaying a display image, in which the region of interest is displayed on an image including the one or more target damaged portions, on a display screen of an image display device;
a region of interest correction unit that stores data obtained by manually correcting the region of interest determined by the region of interest determination unit in the data storage unit as data of the inspection range;
an image feature amount calculation unit that calculates image feature amounts of an image including the one or more target damaged portions and an image feature amount of an image including the multiple damaged portions from image data including the one or more target damaged portions and image data including the multiple damaged portions;
A similar image selection unit that selects a plurality of images including the plurality of damaged portions similar to the image including the one or more target damaged portions from the calculation result of the image feature amount calculation unit,
The region of interest determination unit is provided with the learning model constructed to estimate, as an output, a probability of the presence or absence of damage in the damaged portion and around the damaged portion, using image data including the multiple damaged portions obtained by performing an inspection in the past and data including an inspection range of the multiple damaged portions determined by an expert in the inspection or repair work in response to the image data including the multiple damaged portions as teacher data, and determines, as the region of interest, a region including a portion where the probability of the presence or absence of damage in the target damaged portion and the surrounding region of the target damaged portion output from the learning model using image data including the target damaged portion as input exceeds a predetermined value;
the display data creation unit creates display data for selectably displaying the images including the plurality of damaged portions selected by the similar image selection unit on the display screen of the image display device;
The data storage unit stores repair history information of the plurality of damaged parts corresponding to image data including the plurality of damaged parts obtained by performing inspections in the past,
The display data creation unit creates display data for displaying on the display screen repair history information corresponding to an image including the damaged portion selected from the multiple images including the damaged portions displayed on the display screen. The structure inspection and repair support system according to claim 1, wherein the display data creation unit creates, as the display of the area of interest, a display showing the outline shape of the area to be inspected by the inspector or a display showing the rectangular shape surrounding the outline. The structure inspection and repair support system according to claim 1, wherein the image feature calculated by the image feature calculation unit is a combination of shape feature and color feature. the image data of the target damaged portion and the image data of the plurality of damaged portions are RGB image data;
The image feature amount calculation unit
converting the RGB color space of the RGB image data into an L*a*b* color system;
From the data converted into the L*a*b* color system, only the a*b* component is separated from the L*a*b* components;
The separated a*b* components are quantized using a quantization model;
The color information of the pixels in the quantized color image obtained by quantization is classified into two types: color area I and color boundary B.
4. The structure inspection and repair support system according to claim 3, wherein the quantized occurrence frequency of each color within the classified color region I and the color boundary B is extracted as the color feature. The image feature amount calculation unit
converting the RGB image data into grayscale image data;
Remove noise from the converted grayscale image data;
Extract the contours from the noise-removed grayscale image data,
5. The structure inspection and repair support system according to claim 4, wherein HLAC features are extracted as said shape features from the grayscale image data from which contours have been extracted. The structure inspection and repair support system according to claim 5, characterized in that the image feature calculation unit concatenates the shape feature and the color feature, normalizes the concatenated feature, and outputs it as the image feature. The data storage unit stores damage class data representing the degree of damage of the plurality of damaged parts by a damage class corresponding to image data of the plurality of damaged parts obtained by performing a past inspection,
The structure inspection and repair support system according to claim 1, characterized in that the display data creation unit creates display data that displays the damage class on the display screen together with images of the multiple damaged portions.