JP7410442B2 - Deterioration detection device, deterioration detection method, and program - Google Patents

Description

The present disclosure relates to a deterioration detection device, a deterioration detection method, and a program.

There are many methods for detecting specific image regions using image processing. Among these methods, segmentation methods based on deep learning have recently been considered effective due to their high detection accuracy and ease of construction (Non-Patent Document 1). The idea is to use this segmentation method to build an image processing system that detects structural deterioration, such as exposed reinforcement on concrete walls and corrosion of metal fittings installed in the frame, from images taken of communication manholes. It will be done.

O. Ronneberger, P. Fischer, T. Brox, “U-net: Convolutional networks for biomedical image segmentation”, Lecture Notes in Computer Science, Vol. 9351, pp. 234-241, 2015.

However, since communication manholes are installed underground, many dirts such as mud are reflected in the captured images, and these dirts are mixed with deteriorated parts such as exposed streaks and corrosion of metal parts and the colors in the images. are similar. Therefore, if conventional techniques are simply applied, these stains and the like will be mistakenly detected as deterioration of the structure. Further, the conventional technology detects all areas that are considered to be deteriorated from the photographed images, and even detects minute deterioration that has no effect on the durability of the structure. Therefore, if the conventional technology is simply applied as is, there will be a step after detection in which a person reconfirms whether the detected degraded area is likely to affect the durability of the structure.

An object of the present disclosure is to provide a deterioration detection device, a deterioration detection method, and a program that accurately detect deterioration that may affect a structure.

The deterioration detection device according to one embodiment is a judgment device that is machine-learned using a teacher image, which is an image obtained by photographing, and a teacher label that specifies a pixel in the teacher image that indicates a deteriorated portion of the photographed object as training data. , obtain a photographed image obtained by photographing a structure, convert the photographed image into a predetermined color space, and use the judger to determine the structure in the converted photographed image. A region occupied by a deteriorated part of the object is predicted, and a region including a predetermined number of pixels or more among the predicted regions is an area occupied by the deteriorated part that can affect the structure in the photographed image. The controller includes a control unit that determines a degraded area as a degraded area, and the control unit generates an ROC curve using a test image different from the teacher image and a test label that specifies pixels in the test image that indicate a degraded area of the photographing target. is obtained, based on the ROC curve, determines a threshold value of at least the number of pixels of a region to be determined as the degraded region among the predicted regions, and A region including pixels whose number of pixels is equal to or greater than the threshold value is determined as the degraded region .

In a deterioration detection method according to an embodiment, a control unit of a deterioration detection device selects a teacher image, which is an image obtained by photographing, and a teacher label that specifies a pixel indicating a degraded portion of a photographing target in the teacher image. A machine-learning judge is acquired as data, a photographed image obtained by photographing a structure is acquired, the photographed image is converted into a predetermined color space, and the judgment unit is used to perform the conversion. The area occupied by the deteriorated portion of the structure in the photographed image is predicted, and the area containing pixels of a predetermined number or more is selected from among the predicted areas as areas that will affect the structure in the photographed image. In the deterioration detection method of determining a degraded area that is an area occupied by a degraded portion that may cause A label is used to obtain an ROC curve, and based on the ROC curve, a threshold value of the number of pixels of at least a region to be determined as the degraded region among the predicted regions is determined, and the predicted region is Among them, an area including pixels whose number is equal to or greater than the threshold value as the predetermined number of pixels is determined as the degraded area .

A program according to an embodiment causes a computer to function as the deterioration detection device.

According to an embodiment of the present disclosure, it is possible to provide a deterioration detection device, a deterioration detection method, and a program that accurately detect deterioration that may affect a structure.

FIG. 1 is a diagram showing a hardware configuration of a deterioration detection device according to an embodiment of the present disclosure. FIG. 1 is a diagram showing a functional configuration of a deterioration detection device according to an embodiment of the present disclosure. FIG. 3 is a diagram showing an example of a teacher image. It is a figure which shows an example of a teacher label. FIG. 6 is a diagram illustrating an example of a change in detection rate depending on the type of color space. FIG. 6 is a diagram illustrating an example of a change in detection rate depending on the type of color space. FIG. 3 is a diagram illustrating a configuration example of a learning section included in the deterioration detection device of FIG. 2. FIG. FIG. 3 is a diagram illustrating an example of a confusion matrix showing the relationship between true values and determination results by a determiner. It is a figure showing the relationship between ROC curve and AUC. FIG. 6 is a diagram illustrating a process for determining a threshold value h based on an ROC curve. FIG. 3 is a diagram illustrating a configuration example of a filtering section included in the deterioration detection device of FIG. 2. FIG. FIG. 4 is a diagram illustrating four-neighbor connection. FIG. 3 is a diagram illustrating 8-neighbor connection. FIG. 3 is a diagram showing an example of data after binarization processing. FIG. 3 is a diagram showing an example of data after filtering processing. FIG. 3 is a diagram showing an example of data after filtering processing. FIG. 3 is a diagram illustrating an example of an image showing a deteriorated portion. FIG. 1 is a diagram showing a functional configuration of a deterioration detection device according to an embodiment of the present disclosure. It is a figure which shows an example of the matrix data determined by the determiner storage part, and the matrix data after a binarization process. It is a figure which shows an example of the ROC curve created from the photographed image of the corrosion of a metal object. It is a figure explaining the process of calculating|requiring the threshold value t and the threshold value k based on an ROC curve. It is a flowchart which shows an example of operation of a deterioration detection device concerning one embodiment of this indication.

Hereinafter, one embodiment of the present disclosure will be described with reference to the drawings. In each drawing, the same or corresponding parts are given the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

<Embodiment 1>
A deterioration detection device 10 that detects deterioration from images will be described. The deterioration detection device 10 relates to a device and system for detecting deterioration of communication manholes from photographed images. The deterioration to be detected includes exposed streaks occurring in the frame of the communication manhole in the photographed image and corrosion of metal fittings installed inside the communication manhole. When detecting deterioration using a segmentation method, the deterioration detection device 10 improves detection accuracy by removing dirt and the like from the detection results, and also removes minute deterioration that does not affect the structure. Thereby, the deterioration detection device 10 accurately detects a pixel region corresponding to deterioration from the photographed image. FIG. 1 is a diagram showing a hardware configuration of a deterioration detection device 10 according to an embodiment of the present disclosure.

The deterioration detection device 10 is one or a plurality of server devices that can communicate with each other. The deterioration detection device 10 is not limited to these, and may be any electronic device such as a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), or an electronic notepad. As shown in FIG. 1, the deterioration detection device 10 includes a control section 11, a storage section 12, a communication section 13, an input section 14, an output section 15, and a bus 16.

Control unit 11 includes one or more processors. In one embodiment, a "processor" is a general purpose processor or a dedicated processor specialized for a particular process, but is not limited thereto. The processor may be, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or an ASIC (Application Specific Integrated Circuit). The control unit 11 is communicably connected to each component forming the deterioration detection device 10 via the bus 16, and controls the operation of the deterioration detection device 10 as a whole.

The storage unit 12 includes any storage module including an HDD, SSD, EEPROM, ROM, and RAM. The storage unit 12 may function as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 12 stores arbitrary information used for the operation of the deterioration detection device 10. For example, the storage unit 12 may store system programs, application programs, various information received by the communication unit 13, and the like. The storage unit 12 is not limited to that built in the deterioration detection device 10, but may be an external database or an external storage module connected via a digital input/output port such as a USB. HDD is an abbreviation for Hard Disk Drive. SSD is an abbreviation for Solid State Drive. EEPROM is an abbreviation for Electrically Erasable Programmable Read-Only Memory. ROM is an abbreviation for Read-Only Memory. RAM is an abbreviation for Random Access Memory. USB is an abbreviation for Universal Serial Bus.

The communication unit 13 includes any communication module that can be communicatively connected to other devices using any communication technology. The communication unit 13 may further include a communication control module for controlling communication with other devices, and a storage module that stores communication data such as identification information required for communication with other devices.

The input unit 14 includes one or more input interfaces that accept a user's input operation and obtain input information based on the user's operation. For example, the input unit 14 may be, but is not limited to, a physical key, a capacitive key, a pointing device, a touch screen provided integrally with the display of the output unit 15, or a microphone that accepts audio input.

The output unit 15 includes one or more output interfaces that output information to and notify the user. For example, the output unit 15 may be a display that outputs information as an image, a speaker that outputs information as audio, or the like, but is not limited to these. Note that at least one of the input section 14 and the output section 15 described above may be configured integrally with the deterioration detection device 10, or may be provided separately.

The functions of the deterioration detection device 10 are realized by executing the program according to this embodiment with a processor included in the control unit 11. That is, the functions of the deterioration detection device 10 are realized by software. The program causes the computer to execute the processing of the steps included in the operation of the deterioration detection device 10, thereby causing the computer to realize the functions corresponding to the processing of the steps. That is, the program is a program for causing a computer to function as the deterioration detection device 10 according to the present embodiment. Program instructions may be program code, code segments, etc. to perform necessary tasks.

The program may be recorded on a computer readable recording medium. Using such a recording medium, it is possible to install a program on a computer. Here, the recording medium on which the program is recorded may be a non-transitory recording medium. Non-transitory recording media may be CD (Compact Disk)-ROM (Read-Only Memory), DVD (Digital Versatile Disc)-ROM, BD (Blu-ray (registered trademark) Disc)-ROM, etc. good. Alternatively, the program may be distributed by storing the program in the storage of a server and transferring the program from the server to another computer via a network. The program may be provided as a program product.

For example, a computer temporarily stores a program recorded on a portable recording medium or a program transferred from a server in a main storage device. Then, the computer uses a processor to read a program stored in the main memory, and causes the processor to execute processing according to the read program. A computer may directly read a program from a portable recording medium and execute processing according to the program. The computer may sequentially execute processing according to the received program each time the program is transferred to the computer from the server. Such processing may be executed by a so-called ASP type service that does not transfer the program from the server to the computer, but realizes the function only by issuing execution instructions and obtaining results. "ASP" is an abbreviation for Application Service Provider. The program includes information similar to a program that is used for processing by an electronic computer. For example, data that is not a direct command to a computer but has the property of regulating computer processing falls under "something similar to a program."

A part or all of the functions of the deterioration detection device 10 may be realized by a dedicated circuit included in the control unit 11. That is, some or all of the functions of the deterioration detection device 10 may be realized by hardware. Further, the deterioration detection device 10 may be realized by a single information processing device, or may be realized by cooperation of a plurality of information processing devices.

FIG. 2 is a diagram showing a functional configuration of the deterioration detection device 10 according to Embodiment 1 of the present disclosure. The deterioration detection device 10 includes a determiner construction section 20, a determination section 30, and a filtering section 40. The determiner construction unit 20 includes a teacher image input unit 21, a color space conversion unit 22, a teacher label input unit 23, and a learning unit 24. The determination unit 30 includes an image input unit 31 , a color space conversion unit 32 , and a determiner storage unit 33 .

A teacher image input unit 21 in the determiner construction unit 20 inputs a photographed image that becomes a learning image (teacher image) for constructing a determiner. A photographed image refers to digital data of a still image photographed by a digital camera or the like. In the teacher image input unit 21, the photographed image is input as a bitmap image, and data in a multidimensional array is stored. For example, if the still image is a gray image, it is stored as one layer of matrix data, and if the still image is a color image, it is stored as multiple layers of matrix data. The resolution of the bitmap image is arbitrary.

The teacher label input unit 23 in the determiner construction unit 20 has a function of inputting matrix data of teacher labels. FIG. 3A is a diagram showing an example of a teacher image. FIG. 3B is a diagram showing an example of a teacher label. In the examples shown in FIGS. 3A and 3B, in the photographed image that serves as the teacher image, the teacher assigns "1" (white) to the pixel of corrosion of metal objects that corresponds to deterioration, and "0" (black) to the other pixels. Creating label matrix data. The teacher label input unit 23 inputs the matrix data of this label. Although FIGS. 3A and 3B show an example in which the deterioration is corrosion of metal objects, the type of deterioration is not limited to corrosion of metal objects. Another type of deterioration is defined as deterioration, for example, when the surface concrete is pushed out and peeled off due to corroded expansion of the reinforcing bars inside the concrete, exposing the corroded reinforcing bars (referred to as exposed reinforcement). can be entered. The number of data input to the teacher image input section 21 and the teacher label input section 23 is arbitrary, but both data must necessarily form a pair. The photographed image used as the teacher image contains known deterioration. The teacher label corresponds to "1" for pixels corresponding to deterioration in the photographed images that form a pair, and "0" for the other pixels.

The color space conversion unit 22 in the determiner construction unit 20 has a function of converting the color space of the bitmap image input to the teacher image input unit 21. As a color space, a color image taken with a general digital camera image is stored in the image input unit 31 as matrix data in a numerical array of an RGB (Red, Green, Blue) color space. The color space conversion unit 22 converts the matrix data of the numerical array in the RGB color space to matrix data in other color spaces such as HSV (Hue, Saturation, Value) or L ^* a ^* b ^* , for example. be able to. By converting the color space, the determination accuracy of the determiner can be improved. While the RGB color space prioritizes the convenience of output equipment over human perception, the L ^* a ^* b ^* color space is a color space designed to suit human sensibilities. Corrosion of hardware during deterioration includes many minute rusts, rust-like stains, rust juices, etc. in the captured image, making it difficult to clearly identify the corroded pixel area. Therefore, when creating teacher labels, it is important to rely heavily on human vision. Even in the case of exposed reinforcing bars, the area of corroded reinforcing bars includes rust stains on the surrounding concrete or unevenness due to rust, so the pixel area cannot be clearly identified, and the creation of teacher labels relies on human vision. Therefore, the L ^* a ^* b ^* color space, which better expresses human vision, is more effective for the color space of corrosion and exposed streaks on metal objects.

FIGS. 4A and 4B are diagrams illustrating an example of a change in detection rate depending on the type of color space. Figures 4A and 4B show the results of verifying the deterioration detection rate by creating determiners in three types of color spaces: RGB color space, HSV color space, and L ^* a ^* b ^* color space. There is. FIG. 4A shows the results of verifying the detection rate using 320 images of exposed streaks, and FIG. 4B shows the results of verifying the detection rate using 400 images of corrosion of metal objects. FIGS. 4A and 4B show the results of creating a determination device using a segmentation method for exposed streaks and corrosion of metal objects, respectively, and evaluating the detection rate using the created determination device. As shown in FIGS. 4A and 4B, in detecting corroded areas and exposed streak areas of hardware, the highest detection rate was achieved when the matrix data of the photographed image was converted into the L ^* a ^* b ^* color space. Here, the detection rate is the coincidence rate between the degraded area determined by the determiner and the pixel area of the degraded area given as the true value by a human. Although the L ^* a ^* b ^* color space was used for the degradation in this case, when learning and constructing a determiner, a color space that improves the determination accuracy the most can be arbitrarily set.

In performing color space conversion, the color space conversion unit 22 in the determiner construction unit 20 performs processing to normalize stored matrix data. Normalization here means, when the value of each element of the matrix data is not within the range of 0 or more and 1 or less, converting the numerical value so that each element falls within the range of 0 or more and 1 or less in each layer of the matrix. For example, in the case of RGB color space, it is stored as three-layer matrix data, and the numerical values of elements in each layer are 0 to 255. Therefore, the color space conversion unit 22 normalizes the numerical value of each element by dividing it by 255. The color space conversion unit 22 determines processing based on the numerical value range of the elements input to each layer so that the numerical value of the element in each layer is 0 or more and 1 or less.

The learning unit 24 in the determiner construction unit 20 creates a determiner for detecting deterioration from a teacher image and a teacher label. The learning unit 24 inputs the matrix data output from the color space conversion unit 22 and the teacher label input unit 23 to construct a determiner. The learning unit 24 outputs the constructed determiner to the determining unit 30, and also outputs a threshold h for distinguishing between deterioration and non-deterioration, which will be described later, to the filtering unit 40.

FIG. 5 is a diagram showing a configuration example of the learning section 24 included in the deterioration detection device 10 of FIG. 2. The learning unit 24 includes a loss function determining unit 241 and a learning progressing unit 242.

The loss function determining unit 241 specifies a loss function used for evaluating the prediction accuracy of the neural network in the learning progress unit 242. The loss functions that can be specified include cross-entropy error, squared error, etc., but AUC (Area Under the Curve) maximization and F1 score are effective for detecting deterioration including corrosion and exposed streaks in hardware.

FIG. 6 is a diagram illustrating an example of a confusion matrix showing the relationship between the true value and the determination result by the determiner. A confusion matrix is a matrix that summarizes the class classification results output from a binary classification problem. A binary classification problem is a problem in which a certain proposition (for example, whether the pixel is degraded) is determined to be true (Positive) or false (False). There are four patterns between the results (prediction) and actual results produced by the judge for binary classification: TP (True Positive), TN (True Negative), FP (False Positive), and FN (False Negative). . TP refers to a case where the prediction by the determiner is positive (for example, the pixel is degraded) and the prediction correctly indicates the actual situation (True), that is, the actual situation is also positive. TN refers to a case where the prediction by the determiner is Negative (for example, the pixel is not degraded) and the prediction correctly indicates the reality (True), that is, the reality is also Negative. In FP, when the prediction by the determiner is Positive (e.g., the pixel is degraded), the prediction does not correctly indicate the reality (False), that is, it is actually Negative (e.g., the pixel is not degraded). This refers to the case where FN indicates that when the prediction by the determiner is Negative (e.g., the pixel is not degraded), the prediction does not correctly indicate the reality (False), that is, it is actually Positive (e.g., the pixel is degraded). This refers to the case where

The control unit 11 performs a labeling process on the photographed image as a teacher label that reflects the actual situation, labeling pixels with deterioration such as corrosion of metal objects and exposed streaks as "1", and labeling other pixels with no deterioration as "0". For the processed image, degraded pixels are defined as Positive, and non-degraded pixels are defined as Negative. In this case, the relationship between the true value (Actual) and the determination result (Predicted) by the determiner can be shown using a mixing matrix as shown in FIG. As a result of the determination by the determiner, a numerical value between 0 and 1 is input to each pixel of the matrix data output by the determiner. By binarizing into "0" or "1" by threshold processing using a value between 0 and 1 as a threshold, the judgment result by the judger is Positive, in which a pixel that becomes "1" is judged to be degraded; , pixels that are "0" can be classified as negative pixels, which are determined to be non-degraded pixels.

FIG. 7 is a diagram showing the relationship between the ROC curve and AUC. A curve showing the performance of a judgment result by a judgment device using a mixing matrix is called an ROC curve (Receiver Operating Characteristic curve). The ROC curve is a curve that shows the relationship between the FP rate and the TP rate when the threshold value for determining deterioration or non-deterioration is changed. The FP rate (False Positive Rate) is the ratio of data that is erroneously determined to be Positive among data that should actually be determined to be Negative, and the smaller this value is, the higher the performance of the determiner is. The FP rate is expressed as the number of FP samples/(the number of FP samples+the number of TN samples). The TP rate (True Positive Rate) is the ratio of data that is correctly determined to be positive among data that should actually be determined to be positive, and the larger this value is, the higher the performance of the determiner is. The TP rate is expressed as the number of TP samples/(the number of TP samples+the number of FN samples).

When an ROC curve is drawn with the horizontal axis as the FP rate and the vertical axis as the TP rate, the value obtained by integrating the ROC curve from 0 to 1 with respect to the FP rate axis is called AUC. As described above, the smaller the FP rate and the larger the TP rate, the higher the performance of the determiner, so the magnitude of AUC is one of the indicators of the performance of the determiner. Setting the loss function so that the area of AUC is maximized is called AUC maximization. Deterioration phenomena such as metal corrosion and exposed streaks occur in a relatively small area of pixel areas in each photographed image. When a degraded area is defined as positive and a non-degraded area is defined as negative, the phenomenon in which the amount of data for positive and negative becomes unbalanced is called a data imbalance problem. To deal with this data imbalance problem, we use a loss function to maximize AUC to suppress the FP rate while improving the TP rate so that a smaller proportion of positive pixels are detected compared to negative pixels. It is valid to set it to .

The F1 score is a mathematical formula shown in equation (1).
F1=(2Recall×Precision)/(Recall+Precision) (1)
However, Recall (recall rate) and Precision (precision rate) are shown by equations (2) and (3).
Recall=TP/(TP+FN) (2)
Precision=TP/(TP+FP) (3)
The F1 score is also effective for data imbalance problems, and setting this F1 score as a loss function is useful for solving problems such as corrosion of metal parts of structures or deterioration such as exposed streaks in the entire image, as in this case. This is effective when detecting small areas.

The learning progress unit 242 in the learning unit 24 performs learning using deep learning of the neural network model using the matrix data input to the color space conversion unit 22 and the matrix data input to the teacher label input unit 23. implement. The learning progressing unit 242 advances learning so that the index indicated by the loss function determined by the loss function determining unit 241 improves. For example, if the determined loss function is AUC maximization, the area of the AUC region increases as learning progresses. A large area of AUC corresponds to good performance as a detector. When using the F1 score as a loss function, the value of the F1 score also improves as learning progresses. In the learning progression unit 242, once learning has been completed beyond a certain level, the loss function hardly changes even if learning is performed further. For example, when AUC is used as a loss function, the area of AUC hardly changes even if it learns further. This corresponds to the fact that the area of AUC is already sufficiently large. Even when the F1 score is used as a loss function, the F1 score hardly changes after a certain level of learning is completed. Therefore, when the loss function hardly changes, the learning progressing unit 242 advances the learning until, for example, the rate of change of the loss function becomes less than a predetermined threshold. In this way, the learning progressing unit 242 performs sufficient learning until the value of the loss function determined by the loss function determining unit 241 does not substantially change even if the learning is further advanced, and then uses the determiner of the trained model. To construct. The deep learning model here corresponds to deep learning segmentation methods such as U-net and SegNet. This segmentation method makes it possible to detect areas of deterioration such as corrosion and exposed streaks on metal objects from the image. If corrosion or deterioration such as exposed streaks occurs on hardware of a certain size or more, in order to carry out repair or reinforcement, it not only detects the presence or absence of corrosion and deterioration such as exposed streaks, but also detects areas of deterioration. It is important to. Furthermore, the learning progression unit 242 performs learning using the matrix data output by the color space conversion unit 22 and the matrix data output by the teacher label input unit 23, and constructs a determiner.

Further, the learning progressing unit 242 calculates an ROC curve from the matrix data (predicted value of deterioration) processed by the determiner that constructed the matrix data output by the color space conversion unit 22 and the matrix data output by the teacher label input unit 23. generate. Even when learning is performed using the F1 score as a loss function, an ROC curve is generated based on the learning results. The learning progressing unit 242 determines a threshold value h that minimizes the Euclidean distance, as shown in FIG. FIG. 8 is a diagram illustrating the process of determining the threshold h based on the ROC curve. As shown in FIG. 8, the learning progressing unit 242 determines a threshold value h that minimizes the Euclidean distance between the coordinates (0, 1) and the ROC curve.

The image input unit 31 of the determination unit 30 inputs captured images. The image input unit 31 stores captured images as bitmap image matrix data, similar to the teacher image input unit 21. Here, the input image is a different image from the image input to the teacher image input section 21.

The color space conversion unit 32 of the determination unit 30 has a function of converting the color space of the bitmap image input by the image input unit 31 of the determination unit 30 using the same process as the color space conversion unit 22 of the determination unit construction unit 20. have What is important here is that the color space conversion unit 22 of the determiner construction unit 20 performs conversion to a similar color space specified. By using the same color space arrangement as the color space used when creating the determiner in the learning unit 24 of the determiner construction unit 20, the determination accuracy is improved.

The determiner storage unit 33 in the determiner 30 stores the determiner generated by the determiner construction unit 20. The determiner storage unit 33 performs arithmetic processing on the matrix data input from the color space converter 32 using a determiner. The data output as a result of the arithmetic processing is one-layer matrix data indicating the result of estimating whether each pixel of the captured image input to the image input unit 31 is in a degraded area. The result of estimating whether or not the pixel is in a degraded area outputted by the determiner storage unit 33 indicates a value of 0 or more and 1 or less for each pixel.

FIG. 9 is a diagram showing a configuration example of the filtering section 40 included in the deterioration detection device 10 of FIG. 2. As shown in FIG. The filtering section 40 includes a binarization processing section 41, a connectivity recognition section 42, and a removal section 43.

The binarization processing unit 41 has a function of performing binarization processing of matrix data output from the judge storage unit 33 in the judgment unit 30. The threshold value h output by the learning progressing section 242 in the learning section 24 is used as the threshold value when performing the binarization process. That is, the binarization processing unit 41 sets the value of a pixel whose value indicating the estimation result indicated by the matrix data is equal to or greater than the threshold h to a value of 1, and sets the value of a pixel whose value is less than the threshold h to a value of 0.

The connectivity recognition unit 42 in the filtering unit 40 counts the number of connected elements of elements in which “1” is input in the matrix data after the binarization process. The "number of connected elements" is defined as the number of elements included in a region formed by connecting elements having the same value. A region formed by connecting elements having the same value is called a "connected element." "Connection" is defined by whether the eight elements surrounding the attention element 71 have the same numerical value as the attention element, as shown in FIG. 10A. Focusing on the elements on the top, bottom, left and right of the target element, if any of the pixels on the top, bottom, left and right sides are the same as the target pixel, "4-neighbor connection" is used, and when focusing on the 8 elements surrounding the target element, " 8-neighbor connection. Here, the element for which "1" has been input after the binarization process is the element of interest. Starting from the element of interest, count how many elements are connected. The setting of 4-neighborhood or 8-neighborhood can be made arbitrarily depending on the user's settings and the like. When the counting is completed, the connectivity recognition unit 42 will have numerical data regarding the number of connected elements for the elements for which "1" has been input.

FIG. 10B shows an example of 8-neighbor connection. For the element of interest 71, only the upper right element is connected. The element on the right is connected to the element on the upper right of the element of interest (element 71 on the lower left has already been counted). There are no elements connected to the element on the right other than the element on the left (which has already been counted). Therefore, the number of connected elements in the example of FIG. 10B is three.

The removal unit 43 sets a predetermined threshold k that is a predetermined integer of 1 or more, and replaces the value of the element with "0" for an element whose number of connected elements is less than the threshold k. I do. In other words, the removal unit 43 determines that a pixel area in which the number of connected elements counted by the connectivity recognition unit 42 is less than the threshold k is noise, and includes that area in the output result of the deterioration detection device 10 as an area indicating deterioration. To avoid this, the pixels in the area are replaced with "0".

FIGS. 11A and 11B show data processed with k=200. FIG. 11A is a diagram showing an example of data after binarization processing. FIG. 11B is a diagram illustrating an example of data after filtering processing. In FIG. 11B, some of the pixels displayed as the value "1" (white) in FIG. 11A are replaced with the value "0" (black). Here, 8-neighbor connection is effective for corrosion of hardware and deterioration of exposed streaks. This is because corrosion and exposed streaks spread in an unspecified direction in an image, so it is better to consider the entire vicinity of the pixel of interest as a connection to better reflect the actual situation. 4-neighbor connection has a lower probability of connection than 8-neighbor connection. As a result, there are cases where deterioration such as corrosion and exposed streak areas that should be evaluated as being connected are separated, and corrosion or exposed streak areas that should not be replaced with a value of "0" (black) occur. A case may occur in which the degraded area is replaced with "0" (black). Four-neighbor connection is highly effective for problems where the direction of connection can be predicted, such as with geometric figures. The removal unit 43 outputs only degraded areas that have a large effect on structures, such as corrosion and exposed streaks on metal objects, which have a pixel area larger than a certain size, thereby reducing the effort required for humans to check the output results. be able to.

The result output unit 50 compares the results processed by the filtering unit 40 with a digital camera image and outputs the results. For example, the result output unit 50 can display the results processed by the filtering unit 40 on a monitor or display. FIG. 12A is a diagram illustrating an example of data after filtering processing. FIG. 12B is a diagram illustrating an example of an image showing degraded portions. In FIG. 12B, the portions determined to be deteriorated are distinguished by highlighting 73. In the example of FIG. 12B, the highlighted display 73 distinguishes degraded portions by hatching, but the highlighted display 73 is not limited to this. For example, the highlighted display 73 may be colored, bordered, or a combination thereof. Thereby, the user can easily recognize the portion determined to be degraded in the captured image using the highlighted display 73 as a clue.

As described above, the control unit 11 of the deterioration detection device 10 performs machine learning judgment using the teacher image, which is an image obtained by photographing, and the teacher label that specifies the pixel indicating the deteriorated part in the teacher image as the teacher data. Get the equipment. The control unit 11 acquires a photographed image obtained by photographing a structure, converts the photographed image into a predetermined color space, and uses a determiner to determine whether the structure is visible in the converted photographed image. Predict the area occupied by the degraded part. Further, the control unit 11 determines, among the predicted regions, a region including a predetermined number of pixels or more as a degraded region of the photographed image. Therefore, the deterioration detection device 10 can accurately detect deterioration that may affect the structure.

<Embodiment 2>
FIG. 13 is a diagram showing the functional configuration of the deterioration detection device 10 according to Embodiment 2 of the present disclosure. In the first embodiment, the filtering unit 40 performs a process of replacing the value of an element whose number of connected elements is "1" and whose value is less than a predetermined threshold value k with "0". In this embodiment, a configuration will be described in which a threshold value for the number of connected elements, which is a reference when replacing the value of an element with a value of "1" with "0" in the filtering unit 40, is determined by machine learning.

The deterioration detection device 10 according to the present embodiment has a structure in which a filtering construction section 60 is added to the deterioration detection device 10 according to the first embodiment. Detailed description of the same functional configuration as that of the deterioration detection device 10 according to the first embodiment will be omitted, and the description will focus on the configuration unique to the deterioration detection device 10 according to the present embodiment. The filtering construction section 60 includes a test image input section 61 , a color space conversion section 62 , a test label input section 63 , a judge storage section 64 , a binarization processing section 65 , a connectivity recognition section 66 , and a connection number determination section 67 . Be prepared.

Similar to the teacher image input unit 21, the test image input unit 61 in the filtering construction unit 60 stores captured images as bitmap image matrix data. Here, the image to be input is different from the image input to the teacher image input section 21. However, the image input to the test image input section 61 may be different from the image input section 31 in the determination section 30, or may include a similar image.

Similarly to the teacher label input unit 23, the test label input unit 63 in the filtering construction unit 60 assigns “1” to pixels in the photographed image that have deteriorated corresponding to corrosion or exposed streaks of metal in the image, and “1” to other pixels. 0" is input. Here, the number of data input to the test image input section 61 and the test label input section 63 is arbitrary. However, these data must be paired.

The color space conversion unit 62 in the filtering construction unit 60 has a function of converting the color space of the bitmap image input by the teacher image input unit 21, similarly to the color space conversion unit 22 in the determiner construction unit 20. What is important here is that when creating a determiner to be stored in the determiner storage unit 64 in the filtering construction unit 60, the color space is converted into a color space similar to the color space specified by the color space conversion unit 22 of the determination unit construction unit 20. It is to do. By using the same color space arrangement as the determiner, the determination accuracy is improved.

The determiner storage unit 64 in the filtering constructor 60 has a function of storing the determiner created in the learning unit 24 of the determiner constructor 20. Further, the determiner storage unit 64 has a function of performing arithmetic processing on the matrix data in the color space conversion unit 62 in the filtering construction unit 60 using a determiner. The data output as a result of the arithmetic processing is one-layer matrix data, and a numerical value between 0 and 1 is input into each matrix element. Here, the numerical value between 0 and 1 is called "confidence". When constructing a determiner in the learning unit 24 of the determiner constructing unit 20, if pixels with deterioration corresponding to corrosion or exposed streaks of metal in the image are trained as "1" and other pixels are trained as "0". The determiner determines that the closer each element of the matrix data is to 1, the more degraded, and the closer each element is to 0, the less degraded. Further, the determiner storage section 64 uses the determination result and the data inputted at the test label input section 63 to create an ROC curve for the captured image inputted at the test image input section 61.

The binarization processing unit 65 in the filtering construction unit 60 sets a threshold t and binarizes each element of the matrix data determined by the determiner storage unit 64 in the filtering construction unit 60 to “0” or “1”. do. A plurality of threshold values t for binarization are set between 0 and 1. The threshold value t is any positive number between 0 and 1. FIG. 14 shows an example where the threshold value t=0.7. FIG. 14 is a diagram showing an example of matrix data determined by the determiner storage unit 64 and matrix data after binarization processing.

The connectivity recognition unit 66 in the filtering construction unit 60 counts the number of connected elements of elements for which “1” is input with respect to the matrix data that has been binarized by the binarization processing unit 65 in the filtering construction unit 60. do. A threshold value k is set, and processing is performed to replace elements with "0" for those whose number of connected elements is less than the threshold value k. The connectivity recognition unit 66 increases the value of the threshold value k from 0 and outputs data processed using a plurality of threshold values k. However, k is an integer value. The maximum value of k becomes the number of elements of matrix data. Furthermore, the connectivity recognition unit 66 obtains the result when k is changed in each of the matrix data binarized by a plurality of t's by the determiner storage unit 64 in the filtering construction unit 60. If the number of set thresholds t is i and the number of set thresholds k is j, the total number of data is "i x j x (number of captured images input to the test image input section 61)". Become.

The connection number determining unit 67 in the filtering construction unit 60 calculates the matrix data of the arithmetic total number of data “i×j×(number of captured images input to the test image input unit)” in the connectivity recognition unit 66. An ROC curve is created using the determination result (Predicted) and the matrix data input to the test label input section 63 as the true value (Actual). FIG. 15 is a diagram showing an example of an ROC curve created from photographed images of corrosion of metal objects. The ROC curve in FIG. 15 is an ROC curve when the threshold value t is changed and set to k=0, 10, 20, and 50. There are places where TPR increases and FPR decreases when k is changed compared to the ROC curve with k=0. This connectivity recognition unit 66 has a function of creating ROC curves when k is changed variously.

Thereafter, as shown in FIG. 16, the number of connections determination unit 67 finds a combination of the threshold value t and the value k that minimizes the Euclidean distance when k=a (a is a positive integer). The function of the number of connections determining section 67 can improve TPR and reduce FPR. FIG. 16 is a diagram illustrating a process for determining the threshold value t and the threshold value k based on the ROC curve. As shown in FIG. 16, the number of connections determining unit 67 determines the combination of the threshold t and the threshold k that minimizes the Euclidean distance between the coordinates (0, 1) and the ROC curve. Regarding corrosion and exposed streaks on hardware, there may be cases where areas other than deteriorated areas are determined to be deteriorated due to rust juices, dirt, etc. performance can be further improved.

The threshold value k determined by the number of connections determining section 67 can be input to the removing section 43 in the filtering section 40 . Further, the threshold value t determined by the number of connections determining unit 67 is input as the threshold value of the determiner stored in the determiner storage unit 33 of the determining unit 30, and the threshold value of the determiner is updated. Based on the threshold value k, the removal unit 43 performs a process of replacing the elements of the matrix data processed by the connectivity recognition unit 42 in the filtering unit 40 with “0” for those whose number of connected elements is less than the threshold value k. conduct. This function of the filtering unit 40 allows detection of degraded regions by a segmentation method using a determiner. By using the threshold value k determined by the number of connections determining section 67, it is possible to improve the detection accuracy of degraded portions in the captured image input to the image input section 31 of the determining section 30.

<Operation example>
The operation of the deterioration detection device 10 according to the above embodiment will be explained with reference to FIG. 17. FIG. 17 is a flowchart illustrating an example of the operation of the deterioration detection device 10 according to an embodiment of the present disclosure. The operation of the deterioration detection device 10 described with reference to FIG. 17 corresponds to the deterioration detection method according to this embodiment. The operations of each step in FIG. 17 are executed under the control of the control section 11. A program for causing a computer to execute the deterioration detection method according to this embodiment includes each step shown in FIG. 17.

In step S1, the control unit 11 performs machine learning using a teacher image, which is an image obtained by photographing, and a teacher label specifying a pixel indicating a degraded portion in the teacher image as teacher data, and generates a determiner. The detailed operation of this step is the same as the operation of the determiner construction unit 20.

In step S2, the control unit 11 determines the number of connected elements that a region to be determined as a deteriorated portion should have. That is, the control unit 11 obtains the ROC curve using a test image different from the teacher image and a test label that specifies pixels indicating degraded portions in the test image. Based on the ROC curve, the control unit 11 uses a determiner to determine a threshold value for at least the number of pixels that an area to be determined as a deteriorated area has, among the areas predicted to be occupied by the deteriorated part. The detailed operation of this step is the same as the operation of the filtering construction unit 60. By this step, it is possible to determine a degraded portion more appropriately than when the number of connected elements that a region to be determined as a degraded portion should have is set to a predetermined fixed value. Note that this step is optional and may be omitted.

In step S3, the control unit 11 obtains a photographed image obtained by photographing the structure. The detailed operation of this step is the same as the operation of the image input section 31.

In step S4, the control unit 11 converts the captured image acquired in step S3 into a predetermined color space. Specifically, the control unit 11 converts the photographed image into a predetermined color space, such as an HSV color space or an L ^* a ^* b ^* color space, which better reflects human vision than an RGB color space. color space. This step enables highly accurate deterioration detection that more closely reflects human vision. The detailed operation of this step is the same as the operation of the color space converter 32.

In step S5, the control unit 11 uses the determiner acquired in step S1 to predict the area occupied by the degraded portion of the structure in the captured image converted in step S4. The detailed operation of this step is the same as the operation of the determiner storage section 33.

In step S6, the control unit 11 determines, as a degraded region of the photographed image, an area that includes a predetermined number of pixels or more among the regions predicted to be occupied by the degraded portion in step S5. When the process of step S2 is performed, a region including a predetermined number of pixels equal to or more than the threshold determined in step S2 among the predicted regions is determined as a degraded region of the photographed image. Through this step, it is possible to remove minute deterioration that has no effect on the structure and appropriately determine the deteriorated portion. The detailed operation of this step is the same as the operation of the filtering section 40.

In step S7, the control unit 11 displays the captured image on a display means (result output unit 50) such as a monitor or a display, distinguishing between areas included in the degraded area and areas not included in the degraded area. This step allows the user to easily recognize the portion of the captured image that is determined to be degraded. Then, the control unit 11 ends the process.

The present disclosure is not limited to the embodiments described above. For example, multiple blocks depicted in the block diagram may be combined, or one block may be divided. Instead of being performed chronologically according to the description, the steps described in the flowchart may be performed in parallel or in a different order depending on the processing power of the device performing each step or as necessary. . Other changes are possible without departing from the spirit of the present disclosure.

10 Deterioration detection device 11 Control unit 12 Storage unit 13 Communication unit 14 Input unit 15 Output unit 20 Determiner construction unit 21 Teacher image input unit 22 Color space conversion unit 23 Teacher label input unit 24 Learning unit 241 Loss function determination unit 242 Learning progress Section 30 Judgment section 31 Image input section 32 Color space conversion section 33 Judgment storage section 40 Filtering section 41 Binarization processing section 42 Connectivity recognition section 43 Removal section 50 Result output section 60 Filtering construction section 61 Test image input section 62 Color Space conversion section 63 Test label input section 64 Judgment device storage section 65 Binarization processing section 66 Connectivity recognition section 67 Connection number determination section

Claims (6)

Obtaining a machine learning determiner using a teacher image, which is an image obtained by photography, and a teacher label that specifies a pixel in the teacher image that indicates a deteriorated portion of the photographed object as training data,
Obtain images obtained by photographing structures,
converting the photographed image into a predetermined color space;
predicting an area occupied by a deteriorated portion of the structure in the converted captured image using the determiner;
A control unit that determines, of the predicted area , an area including a predetermined number of pixels or more as a degraded area that is an area occupied by a degraded portion that may affect the structure in the photographed image. ,
The control unit includes:
Obtaining an ROC curve using a test image different from the teacher image and a test label that specifies a pixel in the test image that indicates a degraded portion of the photographic target,
Based on the ROC curve, determine a threshold value for the number of pixels that an area to be determined as the deteriorated area has at least among the predicted areas;
Determining, as the degraded region, a region that includes pixels of the predetermined number of pixels equal to or greater than the threshold value among the predicted regions ;
Deterioration detection device. The deterioration detection device according to claim 1, wherein the control unit generates the determiner by performing machine learning using the teacher image and the teacher label for the teacher image as teacher data. The deterioration detection device according to claim 1 or 2 , wherein the control unit converts the photographed image into an HSV color space or an L*a*b* color space as the predetermined color space. 4. The control unit displays the photographed image on a display means by distinguishing between an area included in the degraded area and an area not included in the degraded area. Deterioration detection device. The control section of the deterioration detection device
Obtaining a machine learning determiner using a teacher image, which is an image obtained by photography, and a teacher label that specifies a pixel in the teacher image that indicates a degraded portion of the photographed object as teacher data,
Obtain images obtained by photographing structures,
converting the photographed image into a predetermined color space;
predicting an area occupied by a deteriorated portion of the structure in the converted captured image using the determiner;
In the deterioration detection method , an area including a predetermined number of pixels or more among the predicted areas is determined as a deteriorated area that is an area occupied by a deteriorated part that may affect the structure in the photographed image. ,
The control unit includes:
Obtaining an ROC curve using a test image different from the teacher image and a test label that specifies a pixel in the test image that indicates a degraded portion of the photographic target,
Based on the ROC curve, determine a threshold value of the number of pixels that an area to be determined as the deteriorated area has at least among the predicted areas;
Determining, as the degraded region, a region that includes pixels of the predetermined number of pixels equal to or more than the threshold value among the predicted regions;
Deterioration detection method . A program that causes a computer to function as the deterioration detection device according to any one of claims 1 to 4 .

Images