JP7485356B2 - Crack detection method, crack detection device and program - Google Patents

Description

The present invention relates to a crack detection method, a crack detection device, and a program, and in particular to a crack detection method that recognizes cracks captured in detection image data.

For example, Patent Document 1 describes the detection of cracks on a concrete surface using pixels whose wavelet coefficients are greater than a threshold value.

JP 2013-238449 A

However, as described in Patent Document 1, sufficient accuracy cannot be expected by simply determining whether or not there is a crack in a binary manner for each pixel.

Therefore, the present invention aims to propose a crack detection method that is suitable for improving accuracy by focusing on the resolution of image data captured to detect cracks.

The first aspect of the present invention is a crack detection method for recognizing cracks captured in detection image data, comprising a learning step in which a learning means included in an information processing device performs a learning process using teacher data including teaching image data in which cracks with a width narrower than the resolution of the detection image data are captured, and a recognition step in which a recognition processing unit included in a crack detection device uses the learning process to recognize cracks with a width narrower than the resolution of the detection image data from the detection image data obtained by capturing the cracks.

A second aspect of the present invention is a crack detection method according to the first aspect, in which the teacher data includes the teacher image data and annotation image data that specifies the crack position and crack width in the teacher image data, the crack width includes a first width narrower than the resolution of the detected image data and a second width narrower than the first width, in the learning step, the learning means performs a learning process to input the teacher image data to a model and output the annotation image data, and in the recognition step, the recognition processing unit uses the model to distinguish between crack portions of the first width and the second width from the detected image data and detect the position and width of the crack portion.

A third aspect of the present invention is a crack detection method according to the second aspect, in which the teacher image data is obtained by a teacher image processing engine processing data output from a teacher imaging element, and the detection image data is obtained by a detection image processing engine processing data output from a detection imaging element, and in the recognition step, a recognition adjustment unit provided in the crack detection device adjusts the difference in appearance in the image data of the crack portion of the first width and/or the second width due to the difference between the teacher imaging element and the detection imaging element and/or the difference between the processing by the teacher image processing engine and the processing by the detection image processing engine, and the recognition processing unit uses the model to distinguish between the crack portion of the first width and the crack portion of the second width from the detection image data and detects the position and width of the crack portion.

The fourth aspect of the present invention is a crack detection device that recognizes cracks captured in detection image data, and includes a recognition processing unit that includes a recognition step that uses a learning process using teacher data including teaching image data in which cracks with a width narrower than the resolution of the detection image data are captured, to recognize cracks whose width is narrower than the resolution of the detection image data from the detection image data obtained by capturing the cracks.

The fifth aspect of the present invention is a program for implementing the recognition step in the crack detection method of any one of the first to third aspects in a computer.

The present invention may be understood as a computer-readable recording medium that records the program of the fifth aspect of the present invention. The present invention may also be understood as a learning means that improves detection accuracy by utilizing a process for detecting cracks that are narrower than the resolution of the detection image data in a process for detecting cracks that are wider than the resolution of the detection image data.

According to the present invention, by using a learning process to detect cracks narrower than the resolution of the image data, it is possible to achieve a crack detection process with high accuracy that exceeds the resolution.

1 is a block diagram showing an example of the configuration of a crack detection system according to an embodiment of the present invention. A flow chart showing an example of the operation of the crack detection system 1 of Figure 1. 1 shows an example of (a) teacher image data, (b) annotation data, (c) detected image data, and (d) detection result data actually used by the inventors. 1 shows an example of detection image data used to determine whether a crack exists. The results of the detection process are shown. An example of a method for improving the detection accuracy of teacher image data will be described below. 11 is a diagram for explaining an example of an evaluation index used in the learning process in the learning processing unit 25. FIG. FIG. 13 is a block diagram for explaining an example of processing when the teacher photographing device 3 and the detection photographing device 7 are different.

Below, an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to this embodiment.

In a photographing device (such as a camera), the image sensor is an electronic component that converts the light entering through the lens into an electrical signal. RAW image data is unprocessed sensor data obtained from the image sensor. An image processing engine converts the RAW image data into an image file. The photographing device converts the RAW image data into an image file format such as JPEG and outputs it. Some photographing devices are also capable of outputting RAW image data.

Resolution is the number of pixels per unit area in image data output from a photographing device. Conventionally, in crack detection, as in Patent Document 1 and the like, a determination has been made as to whether or not a crack exists, using pixels as a unit. A crack is a binary state of whether it exists or not, and it is quite natural to make a binary determination using pixels as a unit. In this case, it is expected that the accuracy of crack detection will improve by increasing the resolution.

However, when the size of the imaging element remains the same or becomes smaller, the image quality may not improve even if the resolution is increased. With imaging elements, the entire pixel cannot usually be used as a light-receiving area. As the resolution increases, the light-receiving area becomes smaller, and image quality may decrease. Therefore, if the resolution is increased, the light from the crack may not be received by the light-receiving area, and it may not be possible to detect the crack. In order to detect cracks with high accuracy, it is necessary to consider the balance between resolution and light-receiving area. As imaging devices become smaller, it may be desirable to improve the accuracy of crack detection by using a device with a low resolution to ensure a light-receiving area and maintain image quality.

The use of RAW image data is considered desirable from the standpoint of detection accuracy. However, in order to use advanced detection processing, it may be desirable to reduce the data size to make it easier to send and receive data. In that case, it may be desirable to reduce the resolution using the image processing engine and output image data with a small file size to make it easier to use advanced detection processing.

To improve the accuracy of crack detection, it is necessary to detect the light from the crack with an imaging element, and then to properly analyze how the light from the crack detected by the imaging element appears in the image file using an image processing engine. Using Figure 1 and other figures, we explain how a learning process can be used to simultaneously evaluate the crack position and crack width in image data, making it possible to detect cracks with high accuracy even if the width is smaller than the resolution.

Also, conventionally, the imaging device used in a system was fixed. However, imaging device technology is evolving day by day, and even with the same hardware, for example, the processing of the image processing engine may differ due to software updates. If the imaging device is fixed, technological advances in imaging devices cannot be utilized to improve detection accuracy. Using Figure 8 and other figures, the detection process when the imaging device is changed will be explained.

Figure 1 is a block diagram showing an example of the configuration of a crack detection system according to an embodiment of the present invention.

The crack detection system 1 includes a teacher imaging device 3, a learning processing device 5, a detection imaging device 7, and a crack detection device 9.

The teacher photography device 3 includes a teacher imaging element 11, a teacher image processing engine 13, and a teacher photography storage unit 15. The teacher photography storage unit 15 stores teacher photography data 19.

The learning processing device 5 includes a teacher data storage unit 21, an annotation processing unit 23, a learning processing unit 25, and a model storage unit 27. The teacher data storage unit 21 stores a plurality of teacher data 31. Each teacher data 31 includes teacher image data 33 and annotation data 35. The annotation data 35 includes annotation position data 37 and annotation width data 39. The model storage unit 27 stores a model 43.

The detection and photography device 7 includes a detection imaging element 51, a detection image processing engine 53, and a detection and photography storage unit 55. The detection and photography storage unit 55 stores detection and photography data 59.

The crack detection device 9 includes a detection data storage unit 61, a recognition processing unit 65, and a detection result data storage unit 67. The detection data storage unit 61 stores detection image data 71. The detection result data storage unit 67 stores detection result data 73.

Figure 2 is a flow diagram showing an example of the operation of the crack detection system 1 of Figure 1. The teacher photographing device 3 photographs cracks occurring in buildings and structures, etc., and obtains teacher photographing data 19 (step ST1). Specifically, in the teacher photographing device 3, the teacher imaging element 11 converts the cracks and the surrounding light entering through the lens into an electrical signal to obtain RAW image data. The teacher image processing engine 13 converts the RAW image data into teacher photographing data 19 with a predetermined resolution and outputs it. The teacher photographing memory unit 15 stores the teacher photographing data 19. There are many teacher photographing data 19, but in this example, the resolution is the same, and hereinafter this resolution is referred to as "teacher resolution". The teacher resolution may be the same as the RAW image data, or it may be smaller.

In the learning processing device 5, the learning processing unit 25 obtains teacher shooting data 19 from the teacher shooting device 3. The learning processing device 5 can obtain the teacher shooting data 19 from the teacher shooting device 3, for example, by using a communication means such as the Internet, or by reconnecting the teacher shooting memory unit 15 as a USB memory or the like. The teacher data memory unit 21 stores the obtained teacher shooting data 19 as teacher image data 33. The teacher image data 33 may store the teacher shooting data 19 as is, or may store one or more crack locations extracted from the teacher shooting data 19.

The annotation processing unit 23 combines annotation data 35 corresponding to each teacher image data 33 (step ST2). The annotation data 35 is information that specifies the position and width of the crack in the teacher image data 33. The annotation position data 37 specifies the position of the crack. The annotation width data 39 specifies the width of the crack. The crack width includes at least a first width smaller than the teacher resolution and a second width smaller than the first width. There are many teacher data 31 corresponding to the teacher image data 33. The annotation processing unit 23 may combine annotation data 35 whose position and width are specified by the administrator of the learning processing device 5, may combine annotation data 35 whose position and width are automatically detected, or may combine both.

In the learning processing device 5, the learning processing unit 25 generates a model 43 that uses a learning process that uses a large amount of teacher data 31, with teacher image data 33 as input and annotation data 35 as output (step ST3). The model can be a commonly known learning model. The model storage unit 27 stores the generated model 43.

The detection and photography device 7 photographs the wall of a building or the like that is the target of the crack detection process, and obtains detection photography data 59 (step ST4). Specifically, in the detection and photography device 7, the detection imaging element 51 converts the light of the processing target area that enters through the lens into an electrical signal to obtain RAW image data. The detection image processing engine 53 converts the RAW image data into detection photography data 59 with a predetermined resolution and outputs it. The detection photography storage unit 55 stores the detection photography data 59. In this example, the resolution of the detection photography data 59 is the same as the teacher resolution.

In the crack detection device 9, the recognition processing unit 65 obtains detection shooting data 59 from the detection shooting device 7. The detection data storage unit 61 stores the obtained detection shooting data 59 as detection image data 71. The detection image data 71 may store the detection shooting data 59 as is, or may store the detection shooting data 59 excluding parts that are not subject to crack detection, such as the sky.

The recognition processing unit 65 uses the model 43 stored in the learning processing device 5 to input the detected image data 71 and output the detection result data 73 (step ST5). The detection result data storage unit 67 stores the detection result data 73. The detection result data 73 includes information that specifies the position and width of the crack in the detected image data 71. The width can be detected by distinguishing between a first width and a second width that are smaller than the teacher resolution.

Figure 3 shows an example of (a) teacher image data, (b) annotation data, (c) detection image data, and (d) detection result data that the inventors actually used. The teacher resolution was fixed at 0.3 mm/pixel. As shown in Figure 3(a), the teacher image data was created by extracting the crack locations photographed. As shown in Figure 3(b), the annotation data specifies the crack location and its width. Figures 3(a) and (b) were combined to create the teacher data. A learning process was performed with Figure 3(a) as input and Figure 3(b) as output. A large amount of teacher data was created and used in the same way. Figure 3(c) shows the crack detection process, and Figure 3(d) shows the detection result. By learning the crack location and width together, it is possible to detect the crack location and width simultaneously.

Figure 4 shows an example of detected image data used to determine cracks. In Figure 4, the crack widths are (a) 0.1 mm or less, (b) 0.2 mm, (c) 0.5 mm, (d) 0.8 mm, (e) 1.5 mm, and (f) 2.0 mm. As in Figures 4(c) to (f), cracks that are much larger than a pixel can be detected on a pixel-by-pixel basis. However, for cracks that are narrower than a pixel, as in Figures 4(a) and (b), a binary judgment on a pixel-by-pixel basis is not appropriate. Therefore, accuracy can be improved by using a learning process to enable the model to distinguish between these and output them.

The present invention detects crack positions and crack widths. For cracks smaller than a pixel, the pixel contains cracked and non-cracked areas. For cracks much larger than a pixel, there are pixels that detect the light of all cracked areas (hereinafter referred to as the "center") and pixels located around it (pixels located on the boundary between the cracked and non-cracked areas. Hereinafter referred to as the "periphery"). For cracks smaller than a pixel and cracks larger than a pixel, there are pixels that contain cracked and non-cracked areas. For such pixels, for example, color information (such as the distribution of RGB values and the difference (gradient) in color intensity between each pixel) is different between cracks smaller than a pixel and cracks larger than a pixel. In the model used by the inventor, the detected image data is passed through a number of filters (convolutional layers) to gradually detect whether it is a crack or not and the crack width. In the model, by analyzing color information for cracks smaller than a pixel and cracks larger than a pixel, it is possible to comprehensively detect cracks smaller than a pixel and cracks larger than a pixel, thereby improving detection accuracy.

For example, in detected image data with a resolution of 0.3 mm/pixel, a crack with a width of 0.5 mm appears in three pixels in the width direction, and light from a crack with a width of 0.3 mm is detected in the central pixel, and light with a width of 0.1 mm is detected in the pixels at both ends (peripheral). In this case, if the crack is detected in units of pixels as in the past, for example by setting the threshold at half, and judging whether more than half of the cracks are detected, the crack can be detected in the central pixel and the crack position can be identified. On the other hand, if the crack width is detected as not being present in the pixels at both ends, it will be judged to be a crack with a width of 0.3 mm. With the present invention, for example, by analyzing the pixels in the center and peripheral parts using a learning process for comprehensively detecting cracks smaller than a pixel and cracks larger than a pixel, and analyzing the presence and width of the crack, it is possible to detect the crack width including the peripheral part, and improve the detection accuracy of crack widths larger than the resolution.

In this way, by applying the learning process of the present invention to a unit that combines central and peripheral pixels, it is possible to improve the detection accuracy of cracks whose crack width is larger than a pixel. For example, when analyzing central and peripheral pixels together, it is possible to use detection image data as well as resolution-changed detection image data that has been converted to a different resolution. Similarly, even in pixel units, detection accuracy can be improved by comprehensively using learning processes for pixels with crack widths smaller than a pixel and learning processes for peripheral pixels.

Figure 5 shows the results of the detection process. The resolution was 0.3 mm/pixel, and cracks with widths of 0.2 mm, 0.1 mm, and less than 0.05 mm were detected.

Figure 6 shows an example of a technique for improving the detection accuracy of teacher image data. (a) By (b) rotating, (c) increasing the brightness, (d) decreasing the brightness, (e) grayscaling, (f) changing the color tone (RGB), or (g) changing the color tone (HSV) of the original image, the teacher data can be increased without increasing the annotation data, thereby improving the detection accuracy. Similar image processing can be performed on the detection image data to improve the detection accuracy.

Figure 7 is a diagram for explaining an example of an evaluation index used in the learning process in the learning processing unit 25. In Figure 7, the pixels in the black part and the lower right diagonal line part indicate the crack location, and indicate the annotation position data in the teacher data. The pixels in the lower right diagonal line part are cracks that have been correctly detected. The pixels in the lower left diagonal line part indicate pixels that are not cracks but have been detected. In the lower left diagonal line part, in addition to the presence or absence of a crack, the width of the crack can be detected, but in this example, the evaluation index focuses on whether or not the crack was detected so as not to overlook it.

The evaluation index can be the recall rate, which places importance on not overlooking cracks. The recall rate is the number of correctly detected crack pixels (the number of pixels in the lower right diagonal line) divided by the number of crack pixels to be detected (the number of pixels in the black part and the lower right diagonal line). In the example of Figure 7, the number of correctly detected crack pixels is 23, and the number of crack pixels to be detected is 35, so the recall rate is about 0.657. In addition, other indexes such as precision (the ratio of correct answers (the number of pixels in the lower right diagonal line) among the detection results (the number of pixels in the lower right diagonal line and the upper right diagonal line)) and IoU (intersection over union, the ratio of correct answers (the number of pixels in the lower right diagonal line) among the pixels to be detected and the detection results (the number of pixels in the black part, the lower right diagonal line and the upper right diagonal line)) may also be used.

Figure 8 is a block diagram for explaining an example of processing when the teacher photography device 3 and the detection photography device 7 are different. The same symbols as in Figure 1 perform similar processing. In the crack detection system 101 in Figure 8, in addition to the crack detection system 1 in Figure 1, teacher adjustment data 17 is stored in the teacher photography memory unit 15 of the teacher photography device 3, teacher adjustment data 29 is stored in the teacher data memory unit 21 of the learning processing device 5, model adjustment data 41 is stored in the model memory unit 27, detection adjustment data 57 is stored in the detection photography memory unit 55 of the detection photography device 7, detection adjustment data 69 is stored in the detection data memory unit 61 of the crack detection device 9, and the crack detection device 9 is equipped with a recognition adjustment unit 63.

In this example, the teacher imaging device 3 and the detection imaging device 7 are products with the same model number, and the teacher imaging element 11 and the detection imaging element 51 have the same hardware configuration, but the software versions are different, so the processing of the teacher image processing engine 13 and the detection image processing engine 53 is different.

When the image processing engine processes, the influence of the crack's color (usually black) becomes stronger in pixels where the crack makes up a large proportion of the pixel, and the influence of the crack's color becomes weaker in pixels where the crack makes up a small proportion of the pixel. In this way, a monotonically changing relationship is observed between the proportion of the crack and the influence of the crack's color. Therefore, even if the processing by the image processing engine differs, only the degree of influence of the crack's color changes, and the proportion of the crack and the influence of the crack's color do not become reversed. Therefore, adjustments are usually sufficient to change the degree of influence of the crack in the photograph, and major changes are not necessary.

The teacher photographing memory unit 15 stores teacher adjustment data 17. This is, for example, image data obtained by photographing a reference figure or the like and performing image processing by the teacher image processing engine 13. The learning processing device 5 obtains the teacher adjustment data 17 from the teacher photographing device 3 and stores it as teacher adjustment data 29. When performing the processing of step ST3, the learning processing unit 25 uses the teacher adjustment data 29 to generate model adjustment data 41 that specifies the teacher image processing engine 13 on which the model 43 to be generated is based.

The detection and photography storage unit 55 stores the detection and adjustment data 57. This is, for example, image data obtained by photographing a reference figure used when creating the teacher adjustment data 17 and performing image processing by the detection image processing engine 53. The crack detection device 9 obtains the detection and adjustment data 57 from the detection and photography device 7 and stores it as detection and adjustment data 69. Before the recognition processing unit 65 performs the processing of step ST5, the recognition adjustment unit 63 uses the model adjustment data 41 and the detection adjustment data 69 to adjust the model 43 so as to correct the difference in the effect of the crack color due to the difference in image processing between the teacher image processing engine 13 and the detection image processing engine 53. This may, for example, be to adjust the threshold value that distinguishes the first width and the second width in the model 43, or to change the color tone of the detection image data 71. The recognition processing unit 65 performs crack detection after the adjustment by the recognition adjustment unit 63 (step ST5).

Note that by using teacher adjustment data, it is possible to allow for multiple teacher image data with different resolutions. Similarly, by using detection adjustment data, it is possible to allow for different resolutions. It is also possible to accommodate cases where the teacher imaging element and the detection imaging element are different.

1,101 Crack detection system, 3 Teacher photography device, 5 Learning processing device, 7 Detection photography device, 9 Crack detection device, 11 Teacher imaging element, 13 Teacher image processing engine, 15 Teacher photography storage unit, 17 Teacher adjustment data, 19 Teacher photography data, 21 Teacher data storage unit, 23 Annotation processing unit, 25 Learning processing unit, 27 Model storage unit, 29 Teacher adjustment data, 31 Teacher data, 33 Teacher image data, 35 Annotation data, 37 Annotation position data, 39 Annotation width data, 41 Model adjustment data, 43 Model, 51 Detection imaging element, 53 Detection image processing engine, 55 Detection photography storage unit, 57 Detection adjustment data, 59 Detection photography data, 61 Detection adjustment data, 63 Recognition adjustment unit, 65 Recognition processing unit, 67 Detection result data storage unit, 71 Detection image data, 73 Detection result data

Claims (4)

A crack detection method for recognizing a crack portion captured in detection image data, comprising:
A learning step in which a learning means included in the information processing device performs a learning process using teacher data including teacher image data in which a crack having a width narrower than the resolution of the detection image data is captured;
The recognition processing unit of the crack detection device includes a recognition step of recognizing a crack portion having a width smaller than the resolution of the detection image data from the detection image data obtained by photographing the crack portion using the learning process ,
The teacher data includes the teacher image data and annotation image data that specifies a crack position and a crack width in the teacher image data,
The crack width includes a first width narrower than a resolution of the detected image data and a second width narrower than the first width,
In the learning step, the learning means performs a learning process on a model so as to input the teacher image data and output the annotation image data;
A crack detection method in which, in the recognition step, the recognition processing unit uses the model to distinguish between crack portions of the first width and the second width from the detected image data and detect the position and width of the crack portion . The teacher image data is obtained by a teacher image processing engine processing data output from a teacher imaging element,
The detected image data is obtained by processing data output from a detected image sensor by a detected image processing engine,
A crack detection method as described in claim 1, wherein, in the recognition step, a recognition adjustment unit provided in the crack detection device adjusts the difference in how cracks of the first width and/or the second width appear in the image data due to the difference between the teacher imaging element and the detection imaging element and/or the difference between the processing by the teacher image processing engine and the processing by the detection image processing engine, and the recognition processing unit uses the model to distinguish between cracks of the first width and the second width from the detection image data and detect the position and width of the crack. A crack detection device that recognizes a crack portion captured in detection image data,
A recognition processing unit includes a recognition step of recognizing a crack portion having a width smaller than the resolution of the detection image data from the detection image data obtained by photographing the crack portion, using a learning process using teacher data including teacher image data in which a crack having a width narrower than the resolution of the detection image data is photographed ,
The teacher data includes the teacher image data and annotation image data that specifies a crack position and a crack width in the teacher image data,
The crack width includes a first width narrower than a resolution of the detected image data and a second width narrower than the first width,
a learning means provided in the information processing device performs a learning process on a model so as to input the teacher image data and output the annotation image data;
The recognition processing unit uses the model to distinguish between cracks of the first width and the second width from the detected image data and detects the position and width of the crack . A program for implementing, in a computer, the recognition step in the crack detection method according to claim 1 or 2 .

Images