WO2022074746A1

WO2022074746A1 - Deterioration detection apparatus, deterioration detection method, and program

Info

Publication number: WO2022074746A1
Application number: PCT/JP2020/037908
Authority: WO
Inventors: 大輔内堀; 一旭渡邉; 勇臣濱野; 雅史中川; 淳荒武
Original assignee: 日本電信電話株式会社
Priority date: 2020-10-06
Filing date: 2020-10-06
Publication date: 2022-04-14
Also published as: JP7410442B2; JPWO2022074746A1

Abstract

A deterioration detection apparatus (10) comprises a control unit (11) for acquiring a determination device that has machine learned, as teaching data, a teaching image, which is an image obtained through imaging, and a teaching label that identifies pixels indicating a deteriorated portion in the teaching image, acquiring a captured image obtained by imaging a structure (31), converting the captured image to a predetermined color space (32), using the determination device to predict a region occupied by the deteriorated portion of the structure in the captured image that has been converted (33), and determining, as a deteriorated region of the captured image, a region, in the predicted region, that includes pixels of a quantity equal to or greater than a predetermined number of pixels (40).

Description

Deterioration detection device, deterioration detection method, and program

This disclosure relates to a deterioration detection device, a deterioration detection method, and a program.

There are many methods to detect a specific image area using image processing. Among these methods, recently, the segmentation method in deep learning is considered to be effective because of its high detection accuracy and ease of construction (Non-Patent Document 1). Using this segmentation method, it is conceivable to construct an image processing system that detects deterioration of structures such as dew streaks generated on concrete walls and corrosion of hardware installed on the skeleton from images taken of communication manholes. Be done.

However, since the communication manhole is installed in the ground, a lot of dirt such as mud is reflected in the captured image, and these dirt are deteriorated parts such as dew streaks and corrosion of hardware and the color in the image. Is similar. Therefore, if the conventional technique is applied as it is, these stains and the like are erroneously detected as deterioration of the structure. Further, the conventional technique detects a region that seems to be deteriorated without omission from the captured image, and detects even a minute deterioration that has no effect in terms of the durability of the structure. Therefore, if the conventional technique is simply applied as it is, a step of reconfirming whether the detected deteriorated region can affect the durability of the structure after detection is required.

An object of the present disclosure is to provide a deterioration detection device, a deterioration detection method, and a program for accurately detecting deterioration that may affect a structure.

The deterioration detection device according to one embodiment acquires a machine-learned determination device using a teacher image, which is an image obtained by photographing, and a teacher label that identifies a pixel indicating a deteriorated portion in the teacher image as teacher data. , The photographed image obtained by photographing the structure is acquired, the photographed image is converted into a predetermined color space, and the structure is deteriorated in the converted photographed image by using the determination device. It is provided with a control unit that predicts the area occupied by the portion and determines as the deteriorated area of the captured image the area including the pixels of the predetermined number of pixels or more among the predicted areas.

In the deterioration detection method according to one embodiment, the control unit of the deterioration detection device uses a teacher image, which is an image obtained by photographing, and a teacher label for specifying a pixel indicating a deteriorated portion in the teacher image as teacher data. The learned determination device is acquired, the captured image obtained by photographing the structure is acquired, the captured image is converted into a predetermined color space, and the converted determination device is used. A region occupied by the deteriorated portion of the structure in the captured image is predicted, and among the predicted regions, a region including a predetermined number of pixels or more is determined as a degraded region of the captured image.

The program according to the embodiment causes the computer to function as the deterioration detection device.

According to one embodiment of the present disclosure, it is possible to provide a deterioration detection device, a deterioration detection method, and a program for accurately detecting deterioration that may affect a structure.

It is a figure which shows the hardware configuration of the deterioration detection apparatus which concerns on one Embodiment of this disclosure. It is a figure which shows the functional structure of the deterioration detection apparatus which concerns on one Embodiment of this disclosure. It is a figure which shows an example of a teacher image. It is a figure which shows an example of a teacher label. It is a figure which shows an example of the change of the detection rate according to the type of a color space. It is a figure which shows an example of the change of the detection rate according to the type of a color space. It is a figure which shows the structural example of the learning part provided in the deterioration detection apparatus of FIG. It is a figure which shows an example of the confusion matrix which shows the relationship between a true value and a judgment result by a judgment device. It is a figure which shows the relationship between the ROC curve and AUC. It is a figure explaining the process of obtaining the threshold value h based on the ROC curve. It is a figure which shows the structural example of the filtering part provided in the deterioration detection apparatus of FIG. It is a figure explaining 4 neighborhood connection. It is a figure explaining 8 neighborhood connection. It is a figure which shows an example of the data after the binarization process. It is a figure which shows an example of the data after a filtering process. It is a figure which shows an example of the data after a filtering process. It is a figure which shows an example of the image which showed the deteriorated part. It is a figure which shows the functional structure of the deterioration detection apparatus which concerns on one Embodiment of this disclosure. It is a figure which shows an example of the matrix data determined by the determination device storage part, and the matrix data after binarization processing. It is a figure which shows an example of the ROC curve created by the photograph image of the corrosion of a hardware. It is a figure explaining the process of obtaining the threshold value t and the threshold value k based on the ROC curve. It is a flowchart which shows an example of the operation of the deterioration detection apparatus which concerns on one Embodiment of this disclosure.

Hereinafter, one embodiment of the present disclosure will be described with reference to the drawings. In each drawing, the same or corresponding parts are designated by the same reference numerals. In the description of the present embodiment, the description will be omitted or simplified as appropriate for the same or corresponding parts.

<Embodiment 1>
A deterioration detection device 10 for detecting deterioration from an image will be described. The deterioration detection device 10 relates to a device and a system for detecting deterioration of a communication manhole from a captured image. Deterioration to be detected is dew streaks generated in the skeleton of the communication manhole in the captured image and corrosion of hardware installed inside the communication manhole. When the deterioration detection device 10 detects deterioration by using the segmentation method, the deterioration detection device 10 improves the detection accuracy by removing dirt and the like from the detection result, and deletes minute deterioration that does not affect the structure. As a result, the deterioration detection device 10 accurately detects the pixel region corresponding to the deterioration from the captured image. FIG. 1 is a diagram showing a hardware configuration of a deterioration detection device 10 according to an embodiment of the present disclosure.

The deterioration detection device 10 is one or a plurality of server devices capable of communicating with each other. The deterioration detection device 10 is not limited to these, and may be any electronic device such as a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), and an electronic notepad. As shown in FIG. 1, the deterioration detection device 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, an output unit 15, and a bus 16.

The control unit 11 includes one or more processors. In one embodiment, the "processor" is a general-purpose processor or a dedicated processor specialized for a specific process, but is not limited thereto. The processor may be, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or the like. The control unit 11 is communicably connected to each component constituting the deterioration detection device 10 via the bus 16 and controls the operation of the entire deterioration detection device 10.

The storage unit 12 includes an arbitrary storage module including an HDD, SSD, EEPROM, ROM, and RAM. The storage unit 12 may function as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 12 stores arbitrary information used for the operation of the deterioration detection device 10. For example, the storage unit 12 may store various information received by the system program, the application program, and the communication unit 13. The storage unit 12 is not limited to the one built in the deterioration detection device 10, and may be an external database or an external storage module connected by a digital input / output port such as USB. HDD is an abbreviation for Hard Disk Drive. SSD is an abbreviation for Solid State Drive. EEPROM is an abbreviation for Electrically Erasable Programmable Read-Only Memory. ROM is an abbreviation for Read-Only Memory. RAM is an abbreviation for Random Access Memory. USB is an abbreviation for Universal Serial Bus.

The communication unit 13 includes an arbitrary communication module capable of communicating with another device by any communication technology. The communication unit 13 may further include a communication control module for controlling communication with other devices, and a storage module for storing communication data such as identification information required for communication with other devices.

The input unit 14 includes one or more input interfaces that accept user input operations and acquire input information based on the user's operations. For example, the input unit 14 is, but is not limited to, a physical key, a capacitance key, a pointing device, a touch screen provided integrally with the display of the output unit 15, a microphone that accepts voice input, and the like.

The output unit 15 includes one or more output interfaces that output information to the user and notify the user. For example, the output unit 15 is, but is not limited to, a display that outputs information as an image, a speaker that outputs information as voice, and the like. At least one of the above-mentioned input unit 14 and output unit 15 may be integrally configured with the deterioration detection device 10 or may be provided as a separate body.

The function of the deterioration detection device 10 is realized by executing the program according to the present embodiment on the processor included in the control unit 11. That is, the function of the deterioration detection device 10 is realized by software. The program causes the computer to execute the processing of the steps included in the operation of the deterioration detection device 10, so that the computer realizes the function corresponding to the processing of the steps. That is, the program is a program for making the computer function as the deterioration detection device 10 according to the present embodiment. The program instruction may be a program code, a code segment, or the like for executing a necessary task.

The program may be recorded on a computer-readable recording medium. Using such a recording medium, it is possible to install the program on the computer. Here, the recording medium on which the program is recorded may be a non-transient (non-temporary) recording medium. Even if the non-transient recording medium is a CD (Compact Disk) -ROM (Read-Only Memory), DVD (Digital Versatile Disc) -ROM, BD (Blu-ray (registered trademark) Disc) -ROM, etc. good. Further, the program may be distributed by storing the program in the storage of the server and transferring the program from the server to another computer via the network. The program may be provided as a program product.

The computer temporarily stores the program recorded on the portable recording medium or the program transferred from the server in the main storage device. Then, the computer reads the program stored in the main storage device by the processor, and executes the processing according to the read program by the processor. The computer may read the program directly from the portable recording medium and perform processing according to the program. The computer may sequentially execute processing according to the received program each time the program is transferred from the server to the computer. Such processing may be executed by a so-called ASP type service that realizes a function only by an execution instruction and result acquisition without transferring a program from a server to a computer. "ASP" is an abbreviation for Application Service Provider. The program includes information used for processing by a computer and equivalent to the program. For example, data that is not a direct command to the computer but has the property of defining the processing of the computer corresponds to "a program-like data".

A part or all the functions of the deterioration detection device 10 may be realized by a dedicated circuit included in the control unit 11. That is, some or all the functions of the deterioration detection device 10 may be realized by hardware. Further, the deterioration detection device 10 may be realized by a single information processing device or may be realized by the cooperation of a plurality of information processing devices.

FIG. 2 is a diagram showing a functional configuration of the deterioration detection device 10 according to the first embodiment of the present disclosure. The deterioration detection device 10 includes a determination device construction unit 20, a determination unit 30, and a filtering unit 40. The determination device construction unit 20 includes a teacher image input unit 21, a color space conversion unit 22, a teacher label input unit 23, and a learning unit 24. The determination unit 30 includes an image input unit 31, a color space conversion unit 32, and a determination device storage unit 33.

The teacher image input unit 21 in the determination device construction unit 20 inputs a captured image that is a learning image (teacher image) for constructing the determination device. The captured image refers to digital data of a still image captured by a digital camera or the like. In the teacher image input unit 21, the captured image is input as a bitmap image, and the data of the multidimensional array is stored. For example, if the still image is a gray image, it is stored as one-layer matrix data, and if it is a color image, it is stored as a plurality of layers of matrix data. The resolution of the bitmap image is arbitrary.

The teacher label input unit 23 in the determination device construction unit 20 has a function of inputting matrix data of the teacher label. FIG. 3A is a diagram showing an example of a teacher image. FIG. 3B is a diagram showing an example of a teacher label. In the examples of FIGS. 3A and 3B, in the captured image as the teacher image, the teacher in which the pixel of the corrosion of the hardware corresponding to the deterioration is associated with "1" (white) and the other pixels are associated with "0" (black). Creating matrix data for labels. The teacher label input unit 23 inputs the matrix data of this label. 3A and 3B show an example in which deterioration is corrosion of hardware, but the type of deterioration is not limited to corrosion of hardware. As another type of deterioration, for example, the state where the surface concrete is extruded and peeled off due to the corrosive expansion of the reinforcing bar inside the concrete and the corroded reinforcing bar is exposed (called dew bar) is also defined as deterioration. Can be entered. The number of data to be input to the teacher image input unit 21 and the teacher label input unit 23 is arbitrary, but both data must be paired. The captured image used as the teacher image contains known degradation. The teacher label corresponds to "1" for the pixel corresponding to the deterioration in the paired captured image and "0" for the other pixels.

The color space conversion unit 22 in the determination device construction unit 20 has a function of converting the color space of the bitmap image input to the teacher image input unit 21. As a color space, a color image taken by a general digital camera image is stored as matrix data in an image input unit 31 in a numerical arrangement of an RGB (Red, Green, Blue) color space. The color space conversion unit 22 converts the matrix data of such a numerical arrangement of the RGB color space into the matrix data of another color space such as HSV (Hue, Saturation, Value) or L ^* a ^* b ^* . be able to. By converting the color space, the determination accuracy of the determination device can be improved. In the RGB color space, the convenience of the output device is prioritized over the human perception, but the L ^* a ^* b ^* color space is a color space devised according to the human sense. In the deterioration, the corrosion of the hardware contains a large amount of minute rust, rust-like stains, rust juice, etc. in the process of progress, so that it is difficult to clearly identify the corroded pixel region. Therefore, it is important to rely heavily on human vision when creating teacher labels. Even in the dew bar, the area of the corroded reinforcing bar contains rust juice or unevenness due to rust on the surrounding concrete, so that the pixel area cannot be clearly specified, and the teacher label depends on human vision. Therefore, the L ^* a ^* b ^* color space, which more expresses human vision, is more effective for the corrosion of hardware and the color space of dew streaks.

4A and 4B are diagrams showing an example of a change in the detection rate depending on the type of color space. FIGS. 4A and 4B show the results of verifying the deterioration detection rate by creating a judgment device in three types of color spaces: RGB color space, HSV color space, and L ^* a ^* b ^* color space. There is. FIG. 4A shows the result of verifying the detection rate with 320 images of dew streaks, and FIG. 4B shows the result of verifying the detection rate with 400 images of corrosion of hardware. FIGS. 4A and 4B show the results of creating a determination device by a segmentation method and evaluating the detection rate using the created determination device for each of the dew streaks and the corrosion of hardware. As shown in FIGS. 4A and 4B, in the detection of the corroded region and the exposed streak region of the hardware, the detection rate was the highest when the matrix data of the captured image was converted into the L ^* a ^* b ^* color space. Here, the detection rate is the matching rate between the deteriorated region determined by the determination device and the pixel region of the deteriorated region given as a true value by a human. In the deterioration of this case, the L ^* a ^* b ^* color space was used, but when learning and constructing the judgment device, the color space with the highest judgment accuracy can be arbitrarily set.

In performing the color space conversion, the color space conversion unit 22 in the determination device construction unit 20 performs a process of normalizing the stored matrix data. The normalization here means that when the value of each element of the matrix data does not fit within 0 or more and 1 or less, the numerical value is converted so that each element fits within 0 or more and 1 or less in each layer of the matrix. For example, in the case of the RGB color space, it is stored as matrix data of three layers, but the numerical value of the element in each layer is 0 or more and 255 or less. Therefore, the color space conversion unit 22 normalizes by the process of dividing the numerical value of each element by 255. The color space conversion unit 22 determines the processing within the numerical range of the elements input to each layer so that the numerical value of the element of each layer is 0 or more and 1 or less.

The learning unit 24 in the determination device construction unit 20 creates a determination device for detecting deterioration from the teacher image and the teacher label. The learning unit 24 constructs a determination device by inputting matrix data output by each of the color space conversion unit 22 and the teacher label input unit 23. The learning unit 24 outputs the constructed determination device to the determination unit 30, and also outputs the threshold value h for distinguishing between deterioration and non-deterioration, which will be described later, to the filtering unit 40.

FIG. 5 is a diagram showing a configuration example of the learning unit 24 included in the deterioration detection device 10 of FIG. The learning unit 24 includes a loss function determination unit 241 and a learning progress unit 242.

The loss function determination unit 241 specifies a loss function used for evaluating the prediction accuracy of the neural network in the learning progress unit 242. The loss function that can be specified can arbitrarily set the cross entropy error, square error, etc., but AUC (Area Under the Curve) maximization and F1 score are effective for detecting deterioration including corrosion of hardware and dew streaks.

FIG. 6 is a diagram showing an example of a confusion matrix showing the relationship between the true value and the judgment result by the judgment device. The confusion matrix is a matrix that summarizes the results of class classification output in the binary classification problem. The binary classification problem is a problem of determining true (Positive) or false (False) for a certain proposition (for example, does the pixel correspond to deterioration). There are four patterns of TP (True Positive), TN (True Negative), FP (False Positive), and FN (False Negative) in the result (prediction) and the actual result given by the judgment device for binary classification. .. TP means that when the prediction by the determination device is Positive (for example, the pixel corresponds to deterioration), the prediction correctly indicates the actual (True), that is, the case where the prediction is also Positive. TN means that when the prediction by the determination device is Negative (for example, the pixel does not correspond to deterioration), the prediction correctly indicates the actual (True), that is, the case where the prediction is also Negative. The FP does not indicate the actual fact correctly (False) when the prediction by the determiner is Positive (for example, the pixel corresponds to deterioration), that is, it is actually Negative (for example, the pixel does not correspond to deterioration). Refers to the case of. FN does not indicate the actual fact correctly (False), that is, it is actually Positive (eg, the pixel corresponds to deterioration) when the prediction by the determiner is Negative (for example, the pixel does not correspond to deterioration). Refers to the case where.

The control unit 11 labels the captured image with "1" for pixels with deterioration such as corrosion of hardware and dew streaks and "0" for other non-deteriorated pixels as a teacher label that reflects the actual situation. For the applied image, the deteriorated pixel is Positive and the non-deteriorated pixel is Negative. In this case, the relationship between the true value (actual: Actual) and the determination result (predicted) by the determination device can be shown by the mixed matrix as shown in FIG. As a judgment result by the judgment device, a numerical value of 0 or more and 1 or less is input to each pixel of the matrix data output by the judgment device. By binarizing to "0" or "1" by the threshold processing with the value of 0 or more and 1 or less as the threshold value, the judgment result by the judgment device is Positive, which judges that the pixel which becomes "1" is deteriorated, and , "0" can be classified as Negative, which is determined to be a non-degraded pixel.

FIG. 7 is a diagram showing the relationship between the ROC curve and the AUC. The curve showing the performance of the judgment result by the judgment device using the mixed matrix is called a ROC curve (Receiver Operating Characteristic curve, receiver operating characteristic curve). The ROC curve is a curve showing the relationship between the FP rate and the TP rate when the threshold value for determining deterioration or non-deterioration is changed. The FP rate (False Positive Rate) is the ratio of the data that is erroneously determined to be Positive among the data that should be determined to be Negative, and the smaller this value, the higher the performance of the determination device. The FP rate is expressed as the number of FP samples / (the number of FP samples + the number of TN samples). The TP rate (True Positive Rate) is the ratio of the data that can be correctly determined to be Positive among the data that should be actually determined to be Positive, and the larger this value is, the higher the performance of the determination device is. The TP rate is expressed as the number of TP samples / (the number of TP samples + the number of FN samples).

When an ROC curve is drawn with the FP rate on the horizontal axis and the TP rate on the vertical axis, the value obtained by integrating the ROC curve from 0 to 1 with respect to the FP rate axis is called AUC. As described above, the smaller the FP rate and the larger the TP rate, the higher the performance of the determination device. Therefore, the size of the AUC is one of the indexes indicating the performance of the determination device. Setting the loss function so that the area of the AUC is maximized is called AUC maximization. Deterioration phenomena such as corrosion of hardware and dew streaks are relatively small as a pixel region in each captured image. If the deteriorated area is Positive and the non-deteriorated area is Negative, the phenomenon that the amount of Positive and Negative data becomes unbalanced is called the data imbalance problem. For this data imbalance problem, the loss function is AUC maximization that suppresses the FP rate while improving the TP rate so that positive pixels with a smaller ratio than the Negative pixels are detected. It is effective to set to.

The F1 score is a mathematical formula shown in the formula (1).
F1 = (2Recall × Precision) / (Recall + Precision) (1)
However, Recall (recall rate) and Precision (precision rate) are expressed by equations (2) and (3).
Recall = TP / (TP + FN) (2)
Precision = TP / (TP + FP) (3)
The F1 score is also effective for the data imbalance problem, and setting this F1 score as a loss function is used throughout the image, such as corrosion of hardware of structures or deterioration of dew streaks, as in this case. It is effective for detecting a small area.

The learning progress unit 242 in the learning unit 24 uses the matrix data input to the color space conversion unit 22 and the matrix data input to the teacher label input unit 23 to perform learning using deep learning of the neural network model. implement. The learning progress unit 242 proceeds with learning so that the index indicated by the loss function determined by the loss function determination unit 241 is improved. For example, when the determined loss function is AUC maximization, the area of the AUC region increases as the learning progresses. A large area of AUC corresponds to good performance as a detector. Even when the F1 score is used as the loss function, the value of the F1 score improves as the learning progresses. In the learning progress unit 242, when the learning is completed for a certain period or more, the loss function hardly changes even if the learning is further performed. For example, when AUC is used as a loss function, the area of AUC hardly changes even if further learning is performed. This corresponds to the area of the AUC already large enough. Even when the F1 score is used as the loss function, the F1 score hardly changes after learning for a certain period or more. Therefore, when the loss function hardly changes, the learning progress unit 242 proceeds with learning until, for example, the rate of change of the loss function becomes less than a predetermined threshold value. In this way, the learning progress unit 242 learns sufficiently until the value of the loss function determined by the loss function determination unit 241 does not substantially change even if the learning is further advanced, and determines the trained model. To construct. The model of deep learning here corresponds to a segmentation method such as U-net and SegNet of deep learning. By this segmentation method, it becomes possible to detect the corrosion of hardware and the deteriorated region such as dew streaks from the image. When corrosion of hardware of a certain size or more and deterioration of dew streaks, etc. occur, in order to repair or reinforce, not only the presence or absence of corrosion and dew streaks, but also the area of deterioration is detected. It is important to. Further, the learning progress unit 242 performs learning using the matrix data output by the color space conversion unit 22 and the matrix data output by the teacher label input unit 23, and constructs a determination device.

Further, the learning progress unit 242 obtains an ROC curve from the matrix data (predicted value of deterioration) processed by the determination device that constructs the matrix data output by the color space conversion unit 22 and the matrix data output by the teacher label input unit 23. Generate. Even when training is performed using the F1 score as a loss function, a ROC curve is generated based on the training result. As shown in FIG. 8, the learning progress unit 242 sets a threshold value h that minimizes the Euclidean distance. FIG. 8 is a diagram illustrating a process of determining the threshold value h based on the ROC curve. As shown in FIG. 8, the learning progress unit 242 determines the threshold value h at which the Euclidean distance between the coordinates (0,1) and the ROC curve is minimized.

The image input unit 31 of the determination unit 30 inputs a captured image. The image input unit 31 stores the captured image as matrix data of the bitmap image in the same manner as the teacher image input unit 21. Here, the image to be input is an image different from the image input to the teacher image input unit 21.

The color space conversion unit 32 of the determination unit 30 has a function of converting the color space of the bitmap image input by the image input unit 31 of the determination unit 30 by the same processing as the color space conversion unit 22 in the determination unit construction unit 20. Have. Here, what is important is to perform conversion to the same color space specified by the color space conversion unit 22 of the determination device construction unit 20. The determination accuracy is improved by using the same color space arrangement as the color space used when creating the determination device in the learning unit 24 of the determination device construction unit 20.

The determination device storage unit 33 in the determination unit 30 stores the determination device generated by the determination device construction unit 20. The determination device storage unit 33 performs arithmetic processing on the matrix data input from the color space conversion unit 32 using the determination device. The data output as a result of the arithmetic processing is one-layer matrix data showing the result of estimating whether or not each pixel of the captured image input to the image input unit 31 is a deteriorated region. The result of estimating whether or not it is the deteriorated region output by the determination device storage unit 33 shows a value of 0 or more and 1 or less for each pixel.

FIG. 9 is a diagram showing a configuration example of the filtering unit 40 included in the deterioration detection device 10 of FIG. The filtering unit 40 includes a binarization processing unit 41, a connectivity recognition unit 42, and a removal unit 43.

The binarization processing unit 41 has a function of performing binarization processing of the matrix data output from the determination device storage unit 33 in the determination unit 30. As the threshold value when performing the binarization process, the threshold value h output by the learning progress unit 242 in the learning unit 24 is used. That is, the binarization processing unit 41 sets the value of the pixel whose value indicating the estimation result indicated by the matrix data is equal to or greater than the threshold value h to 1, and sets the value of the pixel whose value is less than the threshold value h to 0.

The connectivity recognition unit 42 in the filtering unit 40 counts the number of connected elements in which "1" is input to the matrix data after the binarization process. "Number of connected elements" is defined as the number of elements included in a region formed by connecting elements having the same value. A region formed by connecting elements having the same value is called a "connecting element". “Concatenation” is defined by whether or not the eight elements surrounding the element of interest 71 have the same numerical value as the element of interest, as shown in FIG. 10A. Focus on the elements of interest, up, down, left, and right. If any of the pixels on the top, bottom, left, or right is the same as the pixel of interest, "4 neighborhood connection". It is defined as "8 neighborhood connection". Here, the element in which "1" is input after the binarization process is set as the element of interest. Starting from the element of interest, the number of connected elements is counted. The setting of the vicinity of 4 or the vicinity of 8 can be arbitrarily performed according to the user's setting or the like. When the count is completed, the connectivity recognition unit 42 will have numerical data regarding the number of connected elements for which "1" is input.

FIG. 10B shows an example in the case of 8 neighborhood connection. Only the upper right element is connected to the attention element 71. The element on the right is connected to the element on the upper right of the element of interest (the element 71 on the lower left has already been counted). For the element on the right, there is no concatenated element other than the element on the left (which has already been counted). Therefore, the number of connected elements in the example of FIG. 10B is 3.

The removal unit 43 sets a predetermined threshold value k, which is a predetermined integer of 1 or more, and replaces the value of the element with “0” for an element whose number of connected elements is less than the threshold value k. I do. That is, the removing unit 43 determines that the pixel region in which the number of connected elements counted by the connectivity recognition unit 42 is less than the threshold value k is noise, and the region is included in the output result of the deterioration detection device 10 as a region showing deterioration. The process of replacing the pixel in the region with "0" is performed so as not to prevent the pixel.

FIGS. 11A and 11B show data processed with k = 200. FIG. 11A is a diagram showing an example of data after binarization processing. FIG. 11B is a diagram showing an example of data after filtering processing. In FIG. 11B, a part of the pixel displayed as the value “1” (white) in FIG. 11A is replaced with the value “0” (black). Here, in the case of corrosion of hardware and deterioration of dew streaks, connection in the vicinity of 8 is effective. This is because corrosion and dew streaks spread in unspecified directions in the image, so it is better to consider all the periphery of the pixel of interest as a connection to better reflect the actual situation. The probability of connecting in the 4-neighborhood connection is lower than that in the 8-neighborhood connection. Therefore, there may be cases where corrosion that should be evaluated as being actually connected and deterioration of the dew streaks are divided, and corrosion or dew streaks that should not be replaced with the value "0" (black). In some cases, the deteriorated area is replaced with "0" (black). The four-neighborhood connection is highly effective for problems such as geometric figures where the connection direction can be predicted. By outputting only the deteriorated region having a large pixel region of a certain size or more and having a large influence on the structure such as corrosion and dew streaks, the removing unit 43 reduces the labor of human confirmation of the output result. be able to.

The result output unit 50 outputs the result processed by the filtering unit 40 in comparison with a digital camera image or the like. For example, the result output unit 50 can display the result processed by the filtering unit 40 by the monitor or the display. FIG. 12A is a diagram showing an example of data after filtering processing. FIG. 12B is a diagram showing an example of an image in which a deteriorated portion is shown. In FIG. 12B, the portion determined to be deteriorated is distinguished by the highlighting 73. In the example of FIG. 12B, the highlighting 73 distinguishes the deteriorated portion by hatching, but the highlighting 73 is not limited to this. For example, the highlighting 73 may be colored, bordered, or a combination thereof. As a result, the user can easily recognize the portion determined to be deteriorated in the captured image by using the highlighting 73 as a clue.

As described above, the control unit 11 of the deterioration detection device 10 determines that the teacher image, which is an image obtained by photographing, and the teacher label that identifies the pixel indicating the deteriorated portion in the teacher image are machine-learned as teacher data. Get a vessel. The control unit 11 acquires a photographed image obtained by photographing the structure, converts the photographed image into a predetermined color space, and then uses a determination device to capture the structure in the converted photographed image. Predict the area occupied by the deteriorated part. Further, the control unit 11 determines, among the predicted regions, a region including a predetermined number of pixels or more as a deteriorated region of the captured image. Therefore, according to the deterioration detection device 10, deterioration that may affect the structure can be detected with high accuracy.

<Embodiment 2>
FIG. 13 is a diagram showing a functional configuration of the deterioration detection device 10 according to the second embodiment of the present disclosure. In the first embodiment, the filtering unit 40 performs a process of replacing the value of the element whose value is less than the predetermined threshold value k of "1" with "0". In the present embodiment, a configuration will be described in which the threshold value of the number of connected elements as a reference when replacing the value of the element whose value is “1” with “0” in the filtering unit 40 is determined by machine learning.

The deterioration detection device 10 according to the present embodiment has a structure in which the filtering construction unit 60 is added to the deterioration detection device 10 according to the first embodiment. The same functional configuration as that of the deterioration detection device 10 according to the first embodiment will be omitted in detail, and the configuration peculiar to the deterioration detection device 10 according to the present embodiment will be mainly described. The filtering construction unit 60 includes a test image input unit 61, a color space conversion unit 62, a test label input unit 63, a determination device storage unit 64, a binarization processing unit 65, a connectivity recognition unit 66, and a connection number determination unit 67. Be prepared.

The test image input unit 61 in the filtering construction unit 60 stores the captured image as matrix data of the bitmap image in the same manner as the teacher image input unit 21. Here, the image to be input is different from the image input to the teacher image input unit 21. However, the image input to the test image input unit 61 may be different from the image input unit 31 in the determination unit 30, or may include a similar image.

Similar to the teacher label input unit 23, the test label input unit 63 in the filtering construction unit 60 has "1" for the deteriorated pixel corresponding to corrosion or dew streaks of the metal in the image, and "1" for the other pixels. Input the matrix data corresponding to "0". Here, the number of data to be input to the test image input unit 61 and the test label input unit 63 is arbitrary. However, these data must be paired.

The color space conversion unit 62 in the filtering construction unit 60 has a function of converting the color space of the bitmap image input by the teacher image input unit 21 in the same manner as the color space conversion unit 22 in the determination device construction unit 20. Here, what is important is that when the determination device to be stored in the determination device storage unit 64 in the filtering construction unit 60 is created, the color space is converted to the same color space as the color space specified by the color space conversion unit 22 of the determination device construction unit 20. Is to do. By using the same color space arrangement as the judgment device, the judgment accuracy is improved.

The determination device storage unit 64 in the filtering construction unit 60 has a function of storing the determination device created in the learning unit 24 of the determination device construction unit 20. Further, the determination device storage unit 64 has a function of performing arithmetic processing on the matrix data in the color space conversion unit 62 in the filtering construction unit 60 using the determination device. The data output as a result of the arithmetic processing is one-layer matrix data, and a numerical value of 0 or more and 1 or less is input to each matrix element. Here, the numerical value from 0 to 1 is called "confidence". When the learning unit 24 of the determining device construction unit 20 learns the deteriorated pixels corresponding to the corrosion or dew streaks of the metal in the image as "1" and the other pixels as "0". The determination device determines that each element of the matrix data is deteriorated as it is closer to 1, and is not deteriorated as it is closer to 0. Further, the determination device storage unit 64 creates an ROC curve for the captured image input by the test image input unit 61 by using the determination result and the data input by the test label input unit 63.

The binarization processing unit 65 in the filtering construction unit 60 sets the threshold value t and binarizes each element of the matrix data determined by the determination device storage unit 64 in the filtering construction unit 60 to "0" or "1". do. A plurality of threshold values t for binarization are set between 0 and 1. The threshold value t is an arbitrary positive number of 0 or more and 1 or less. An example in the case of the threshold value t = 0.7 is shown in FIG. FIG. 14 is a diagram showing an example of the matrix data determined by the determination device storage unit 64 and the matrix data after the binarization process.

The connectivity recognition unit 66 in the filtering construction unit 60 counts the number of connected elements in which "1" is input to the matrix data after the binarization processing by the binarization processing unit 65 in the filtering construction unit 60. do. A threshold value k is set, and a process of replacing an element with "0" is performed for an element having a number of connected elements less than the threshold value k. The connectivity recognition unit 66 increases the value of the threshold value k from 0 and outputs the data processed by the plurality of threshold values k. However, k is an integer value. The maximum value of k is the number of elements of the matrix data. Further, the connectivity recognition unit 66 obtains the result when k is changed in each of the matrix data binarized by the plurality of t by the determination device storage unit 64 in the filtering construction unit 60. Assuming that the number of set threshold values t is i and the number of set threshold values k is j, the total number of data is "i × j × (number of captured images input to the test image input unit 61)". Become.

The connection number determination unit 67 in the filtering construction unit 60 inputs matrix data of the total number of calculated data “i × j × (number of captured images input to the test image input unit)” in the connectivity recognition unit 66. The ROC curve is created with the matrix data input to the test label input unit 63 as the determination result (Predicted) as the true value (Actual). FIG. 15 is a diagram showing an example of a ROC curve created by a photographed image of corrosion of hardware. The ROC curve of FIG. 15 is an ROC curve when k = 0, 10, 20, 50 while changing the threshold value t. There are places where TPR increases and FPR decreases when k is changed rather than the ROC curve where k = 0. The connectivity recognition unit 66 has a function of creating a ROC curve when k is changed in various ways.

After that, as shown in FIG. 16, the connection number determination unit 67 obtains a combination of the threshold value t and the value k so that the Euclidean distance is minimized when k = a (a is a positive integer). By the function of the connection number determination unit 67, improvement of TPR and reduction of FPR can be realized. FIG. 16 is a diagram illustrating a process of obtaining a threshold value t and a threshold value k based on the ROC curve. As shown in FIG. 16, the connection number determination unit 67 determines the combination of the threshold value t and the threshold value k that minimizes the Euclidean distance between the coordinates (0,1) and the ROC curve. Regarding corrosion and dew streaks of hardware, there may be cases where areas other than the deteriorated area are judged to be deteriorated due to rust juice, dirt, etc., but by judging from the size of the detected area that the area is not deteriorated, the judgment device Performance can be further improved.

The threshold value k determined by the connection number determination unit 67 can be input to the removal unit 43 of the filtering unit 40. Further, the threshold value t determined by the connection number determination unit 67 is input as the threshold value of the determination device stored in the determination device storage unit 33 of the determination unit 30, and the threshold value of the determination device is updated. Based on the threshold value k, the removal unit 43 performs a process of replacing the matrix data processed by the connectivity recognition unit 42 in the filtering unit 40 with "0" for the matrix data whose number of connected elements is less than the threshold value k. conduct. By the function of this filtering unit 40, it is possible to detect a deteriorated region by a segmentation method using a determination device. By using the threshold value k determined by the connection number determination unit 67, it is possible to improve the detection accuracy of the deteriorated portion in the captured image input to the image input unit 31 of the determination unit 30.

<Operation example>
The operation of the deterioration detection device 10 according to the above embodiment will be described with reference to FIG. FIG. 17 is a flowchart showing an example of the operation of the deterioration detection device 10 according to the embodiment of the present disclosure. The operation of the deterioration detection device 10 described with reference to FIG. 17 corresponds to the deterioration detection method according to the present embodiment. The operation of each step in FIG. 17 is executed based on the control of the control unit 11. The program for causing the computer to execute the deterioration detection method according to the present embodiment includes each step shown in FIG.

In step S1, the control unit 11 performs machine learning using the teacher image, which is an image obtained by shooting, and the teacher label that identifies the pixel indicating the deteriorated portion in the teacher image as teacher data, and generates a determination device. The detailed operation content of this step is the same as the operation of the determination device construction unit 20.

In step S2, the control unit 11 determines the number of connecting elements that the region to be determined as the deteriorated portion should have. That is, the control unit 11 acquires the ROC curve by using a test image different from the teacher image and a test label that identifies a pixel indicating a deteriorated portion in the test image. Based on the ROC curve, the control unit 11 uses a determination device to determine a threshold value of at least the number of pixels that the region to be determined as the deteriorated region has at least among the regions predicted to be occupied by the deteriorated portion. The detailed operation content of this step is the same as the operation of the filtering construction unit 60. By this step, the deteriorated portion can be determined more appropriately than when the number of connecting elements to be possessed by the region to be determined as the deteriorated portion is set to a predetermined fixed value. Note that this step is optional and may be omitted.

In step S3, the control unit 11 acquires a photographed image obtained by photographing the structure. The detailed operation content of this step is the same as the operation of the image input unit 31.

In step S4, the control unit 11 converts the captured image acquired in step S3 into a predetermined color space. Specifically, the control unit 11 reflects the captured image as a predetermined color space, for example, the HSV color space or the L ^* a ^* b ^* color space, which reflects human vision more than the RGB color space. Convert to the color space. By this step, it is possible to perform accurate deterioration detection that more reflects human vision. The detailed operation content of this step is the same as the operation of the color space conversion unit 32.

In step S5, the control unit 11 predicts the area occupied by the deteriorated portion of the structure in the captured image converted in step S4 by using the determination device acquired in step S1. The detailed operation content of this step is the same as the operation of the determination device storage unit 33.

In step S6, the control unit 11 determines, among the regions predicted to be occupied by the deteriorated portion in step S5, a region including a pixel equal to or larger than a predetermined number of pixels as a degraded region of the captured image. When the processing of step S2 is performed, the region including the pixels equal to or larger than the threshold value determined in step S2 is determined as the deteriorated region of the captured image as the predetermined number of pixels in the predicted region. By this step, it is possible to remove minute deterioration that does not affect the structure and appropriately determine the deteriorated portion. The detailed operation content of this step is the same as the operation of the filtering unit 40.

In step S7, the control unit 11 distinguishes between the area included in the deteriorated area and the area not included in the deteriorated area and displays the captured image on a display means (result output unit 50) such as a monitor or a display. By this step, the user can easily recognize the portion determined to be deteriorated in the captured image. Then, the control unit 11 ends the process.

The present disclosure is not limited to the above-described embodiment. For example, the plurality of blocks shown in the block diagram may be integrated, or one block may be divided. The plurality of steps described in the flowchart may be executed in parallel or in a different order depending on the processing power of the device that executes each step, or as necessary, instead of executing the steps in chronological order according to the description. .. Other changes are possible without departing from the spirit of this disclosure.

10 Deterioration detection device 11 Control unit 12 Storage unit 13 Communication unit 14 Input unit 15 Output unit 20 Judgment unit construction unit 21 Teacher image input unit 22 Color space conversion unit 23 Teacher label input unit 24 Learning unit 241 Loss function determination unit 242 Learning progress Part 30 Judgment unit 31 Image input unit 32 Color space conversion unit 33 Judgment unit storage unit 40 Filtering unit 41 Binification processing unit 42 Connectivity recognition unit 43 Removal unit 50 Result output unit 60 Filtering construction unit 61 Test image input unit 62 colors Spatial conversion unit 63 Test label input unit 64 Judgment unit storage unit 65 Binarization processing unit 66 Connectivity recognition unit 67 Connection number determination unit

Claims

A machine-learned judgment device is acquired using the teacher image, which is an image obtained by shooting, and the teacher label, which identifies the pixel indicating the deteriorated portion in the teacher image, as teacher data.
Acquire the captured image obtained by photographing the structure,
The captured image is converted into a predetermined color space, and the photographed image is converted into a predetermined color space.
Using the determination device, the region occupied by the deteriorated portion of the structure in the converted captured image is predicted.
A deterioration detection device including a control unit for determining a region including a predetermined number of pixels or more of the predicted regions as a deterioration region of the captured image.
The deterioration detection device according to claim 1, wherein the control unit performs machine learning using the teacher image and the teacher label for the teacher image as teacher data to generate the determination device.
The control unit
An ROC curve is obtained by using a test image different from the teacher image and a test label that identifies a pixel indicating a deteriorated portion in the test image.
Based on the ROC curve, the determination device is used to determine the threshold value of the number of pixels that the region to be determined as the deteriorated region has at least among the regions predicted to be occupied by the deteriorated portion.
The deterioration detection device according to claim 1 or 2, wherein a region including pixels equal to or larger than the threshold value is determined as a deterioration region of the captured image as a predetermined number of pixels in the predicted region.
The deterioration detection according to any one of claims 1 to 3, wherein the control unit converts the captured image into an HSV color space or an L * a * b * color space as the predetermined color space. Device.
The one according to any one of claims 1 to 4, wherein the control unit distinguishes between a region included in the deteriorated region and a region not included in the deteriorated region and displays the captured image on a display means. Deterioration detector.
The control unit of the deterioration detection device
A machine-learned judgment device is acquired using the teacher image, which is an image obtained by shooting, and the teacher label, which identifies the pixel indicating the deteriorated portion in the teacher image, as teacher data.
Acquire the captured image obtained by photographing the structure,
The captured image is converted into a predetermined color space, and the photographed image is converted into a predetermined color space.
Using the determination device, the region occupied by the deteriorated portion of the structure in the converted captured image is predicted.
A deterioration detection method for determining a region including a predetermined number of pixels or more among the predicted regions as a deterioration region of the captured image.
A program that causes a computer to function as the deterioration detection device according to any one of claims 1 to 5.