JP2019200512A - Structure deterioration detection system - Google Patents

Abstract

To provide a technique capable of shortening a time including a calculation time required for learning and diagnosis in a computer and a user's work time and reducing a user's work load.SOLUTION: A computer system 1 (application 10) of a structure deterioration detection system executes first processing using the deep learning of a first image of a structure 5 as an input to output a second image representing a diagnosis result of deterioration and second processing for visualizing information including a second image to display the visualized information on a GUI screen 21. A CNN consisting of a model 31 of the deep learning includes a widening convolution layer. The first process at a training time inputs a first image patch of a predetermined first input size to the model 31 to obtain a first diagnosis result image of a first output size. The first process at a diagnosis time inputs a second image patch of a second input size to the model 31 from a target image of a variable size to obtain a second diagnosis result image of a second output size.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technology such as an information processing system for detecting a state such as deterioration of a structure. In particular, the present invention relates to a technique for learning and diagnosing a state such as deterioration using machine learning.

  Structures such as various buildings and infrastructure equipment (including houses, buildings, roads, railways, bridges, tunnels, electrical equipment, water supply equipment, communication equipment, etc.) are cracked, Deterioration or damage such as rust / corrosion, peeling, adhesion of foreign matter, etc. (may be collectively referred to as “deterioration”) occurs. Therefore, countermeasure work such as inspection and repair is necessary for maintenance. However, there are social issues such as a lack of personnel and high costs for the work. On the other hand, a system (which may be described as a structure deterioration detection system) for diagnosing and detecting a state such as deterioration of a structure using a computer has been developed and is expected to be effective.

  In the related art example related to the structure deterioration detection, the surface of the target structure is imaged using a camera, and the person (operator) visually diagnoses and detects the deterioration of the captured image. Alternatively, an image is input to the computer, and a process of estimating and detecting a deteriorated part by image processing or the like is performed. Alternatively, in particular, a system has been developed in which features are learned from images (also called training) and diagnosed using machine learning on a computer.

  Japanese Patent No. 6294529 (Patent Document 1) is given as an example of the prior art relating to structure deterioration detection using machine learning. Japanese Patent Application Laid-Open No. 2005-228561 describes a “crack detection processing device” or the like that uses machine learning to detect a crack from a road surface image, and processes a block image obtained by dividing the road surface image.

  One type of machine learning is deep learning. In recent years, techniques for learning and diagnosing images using a convolutional neural network (CNN) or the like in deep learning have been developed. Non-Patent Document 1 describes that as an example of the CNN method, training is performed in units of image patches and diagnosis (inference) is performed in units of variable-size input images.

Japanese Patent No. 6294529

P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, “OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks”, arXiv: 1312.6229 [cs], Dec. 2013. <URL: https://arxiv.org/pdf/1312.6229.pdf>

  In the structure deterioration detection system of the prior art example, the computer learns and diagnoses features such as deterioration from the input image using machine learning including teacher information input, and detects it. At that time, particularly when deep learning is used, although it depends on the computer performance, it takes a long time to calculate a CNN model (also called a network). Since a large number of images need to be processed and the model calculation needs to be performed many times, it takes a long calculation time. In addition, the system takes a long time as a whole, including the calculation time and the work time by the user (worker).

  In addition, the system requires a lot of work for the user. For example, the user looks at a diagnosis result image by machine learning of a computer on the screen and performs correct answering work for inputting whether or not the degradation estimation result is correct for each pixel. The accuracy of diagnosis can be improved by reflecting the correct answer information as teacher information in the model. However, it takes a lot of time for correct answering work by the user. There is also a need to reduce the work burden on the user.

  An object of the present invention is to reduce the time required for learning and diagnosis in a computer and the time required for the user and the work time of the user while ensuring the accuracy of the deterioration detection with respect to the structure deterioration detection system technology. It is to provide technology that can be reduced.

  A typical embodiment of the present invention is a structure deterioration detection system having the following configuration. A structure deterioration detection system according to an embodiment is a structure deterioration detection system configured on a computer system to detect deterioration including cracks on the surface of the structure, and the computer system includes a surface of the structure. The first process that outputs the second image including the information representing the diagnosis result of the deterioration using the deep learning, and the information including the first image and the second image A convolutional neural network constituting the deep learning model includes a widening convolutional layer for calculating a widening convolutional filter, and performing a second process of accepting an input operation by a user. In the first process, a first image patch having a predetermined first input size is input to the model based on the training image data during training, and a first diagnosis having a first output size is performed. During the training process for obtaining a result image and the diagnosis of the target image of the structure, a second image patch having a second input size is cut out from the target image that is not less than the first input size as a variable size. Diagnostic processing for inputting an image patch to the model and obtaining a second diagnostic result image of each of the second output sizes.

  According to the representative embodiment of the present invention, regarding the structure deterioration detection system technology, the time including the calculation time required for learning and diagnosis in the computer and the user's work time is ensured while ensuring the accuracy of deterioration detection. It is to provide a technique that can be shortened and reduce the work burden on the user.

It is a figure which shows the structure of the structure deterioration detection system of Embodiment 1 of this invention. In Embodiment 1, it is a figure which shows the structure of structure deterioration detection software. In Embodiment 1, it is a figure which shows an image patch, a DL-CNN model, etc. FIG. In Embodiment 1, it is a figure which shows the example of a wide convolution filter. In Embodiment 1, it is a figure which shows the widening convolution process. In Embodiment 1, it is a figure which shows a CNN model and calculation. In Embodiment 1, it is a figure which shows image size relationship etc. FIG. In Embodiment 1, it is a figure which shows the processing flow at the time of training. In Embodiment 1, it is a figure shown about a weak point image. In Embodiment 1, it is a figure which shows the example of a weak point image. FIG. 10 is a diagram showing MIL rotation processing in the first embodiment. In Embodiment 1, it is a figure which shows the processing flow of the 1st process at the time of diagnosis. In Embodiment 1, it is a figure which shows the processing flow of the 2nd process at the time of a visualization screen display. In Embodiment 1, it is a figure which shows the example of a display of a visualization screen. 6 is a diagram illustrating examples of various images in Embodiment 1. FIG. In Embodiment 1, it is a figure which shows examples, such as a binarized image. In Embodiment 1, it is a figure shown about model input size setting. It is a figure which shows the model etc. of DL-CNN in the structure deterioration detection system of a comparative example. It is a figure which shows a model and calculation in the structure deterioration detection system of a comparative example.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that components having the same function are denoted by the same reference symbols throughout the drawings for describing the embodiment, and the repetitive description thereof will be omitted.

[Issues]
Supplementary explanations of the prerequisite technologies and issues are given below.

  (1) In the case of a system for diagnosing deterioration from an image without using machine learning such as deep learning as a structure deterioration detection system of the prior art example, the flow of inspection work of a user such as an operator is as follows: It is. (1-1) A user photographs a structure such as an infrastructure facility to be inspected on site with a camera. For example, for a certain target structure, the number of images taken by the camera is 1000. (1-2) The user visually confirms and diagnoses whether or not there is a degraded portion such as a crack in each of the captured images. For example, 1000 camera-captured images become viewing target images, and 80 images are extracted from the 1000 viewing target images as images including a deteriorated portion. In the case of the prior art example described above, it takes a lot of time and effort for the user's diagnosis work and the like and takes a long time.

  (2) In the case of a system in which the computer learns and diagnoses deterioration from an image using machine learning as the structure deterioration detection system of the prior art example, the flow of the inspection work of the user is as follows. (2-1) Similarly, an image obtained by photographing the target structure with a camera, for example, 1000 images is prepared. Those image data are input to the computer. (2-2) The computer executes deterioration diagnosis by machine learning for each image of all images. Model calculation is applied to each image, and as the diagnosis result information, an image (diagnosis result image) including, for example, a degradation estimated portion, for example, 100 sheets is obtained. The model calculation needs to be repeated according to the number of images, which takes a long calculation time. (2-3) The user visually confirms the estimated deterioration location in each diagnosis result image on the screen, and finally determines whether the estimated deterioration location is actually deteriorated. Further, the user confirms whether or not the degradation estimation result in the image is a correct answer on the screen, inputs correct answer information, and reflects it in the model. In the case of the above prior art example, it takes a long time including the calculation time of the computer and the work time of the user.

(Embodiment 1)
The structure deterioration detection system according to the first embodiment of the present invention will be described with reference to FIGS. The structure deterioration detection system according to the first embodiment is a system configured on a computer system to detect deterioration including cracks on the surface of the structure. The computer system uses a first image (image group) obtained by imaging the surface of the structure as an input, and uses a deep learning (sometimes abbreviated as DL) to generate a second image including information representing a diagnosis result of deterioration. The output first process (deterioration diagnosis process) is performed. The computer system also performs a second process (visualization process) that visualizes information including the first image and the second image, displays the information on the screen, and receives an input operation by the user. The CNN constituting the deep learning model includes a widening convolution layer for calculating a widening convolution filter. The first process includes a training process for inputting a first image patch having a predetermined first input size to a model and obtaining a first diagnosis result image having a first output size based on the training image data during training. . In the first process, when diagnosing the target image of the target structure, a second image patch having the second input size is cut out from the target image having a variable size equal to or larger than the first input size, and each second image patch is extracted. A diagnostic process is included that is input to the model and obtains a second diagnostic result image of each of the second output sizes.

  The flow of the user's inspection work in the case of the structure deterioration detection system of the first embodiment is as follows. (3-1) Similarly, an image obtained by photographing the target structure with a camera, for example, 1000 sheets is prepared. Those image data are input to the computer. (3-2) This system executes deterioration diagnosis by deep learning for each image of all images. As the diagnosis result image, an image including a degradation estimated portion, for example, 100 images is obtained. In one model calculation, diagnostic result information of the second output size (a plurality of vertical and horizontal pixels) is obtained. In the prior art example in which the number of strides is set to 2 or more in a predetermined layer of the CNN, the output size is multiple times smaller than the input size, and the output becomes sparse. Therefore, it is necessary to perform a process for dense output by using a model several tens of times as in Non-Patent Document 1 or using a model for each pixel as in the prior art. On the other hand, in the system of the first embodiment, by setting the stride number to 1 in all layers of the CNN, a dense output can be created by using one model, and the calculation required for one image Time is suppressed for a short time. The model calculation of the deterioration diagnosis is performed as an actual diagnosis at the time of training or work for each image, and the model is learned (updated) each time.

  (3-3) This system displays a diagnostic result image on a screen on a computer application. There are, for example, 100 visual target images corresponding to the diagnosis result image. The user visually confirms each image on the screen, and makes a final determination as to whether or not the degradation estimated location is actually degraded. For example, 80 images are extracted as the images including the degraded portion of the final determination result of the user. In addition, the user corrects whether or not the deterioration estimation result is correct with respect to the diagnosis result image of the training image on the screen, and the correct image is reflected in the model. This increases the accuracy of diagnosis.

  As described above, in this system, the user's visual target image can be narrowed down by deterioration diagnosis using deep learning. In the above example, the number is reduced from 1000 to 100. Thereby, the man-hour, time, cost, etc. concerning a user's inspection work can be reduced. As described above, in the present system, deterioration diagnosis is not completely automated, but part of the deterioration diagnosis is automated using deep learning. In this system, the diagnosis result image is visualized on the screen on the application, and the work including the final determination and the correct answer in the visual confirmation of the user is supported. As a result, the time for degradation detection is shortened while ensuring accuracy.

[Structural deterioration detection system]
FIG. 1 shows an overall configuration including the structure deterioration detection system of the first embodiment. The whole of FIG. 1 includes a computer system 1, a structure 5, and a camera 4. The structure 5 is an object of deterioration diagnosis, and corresponds to various buildings and infrastructure facilities. The camera 4 captures the surface of the structure 5 based on a user (operator) operation, and obtains image data including an image 41 (still image or moving image). The image 41 may include a portion of deterioration 42. In the first embodiment, the deterioration 42 includes at least a crack.

  The structure deterioration detection system according to the first embodiment is mainly configured by a computer system 1. The computer system 1 is a system including an arbitrary computer. For example, a computer 2 that is a PC and a computer 3 that is a server are connected via a communication network 6. The computer system 1 may be composed of a plurality of computers. The computer 2 is, for example, a PC serving as a client terminal device used by a user (operator) who performs work related to detection of structure deterioration in an arbitrary organization such as an administrator of the structure 5 or a business operator who has been inspected. It is. There may be a plurality of people as users. The computer 3 is a server device provided on a system such as a cloud computing system or a data center by an operator, for example. The computer 2 and the computer 3 may include a GPU (Graphics Processing Unit).

  The computer 2 and the computer 3 are provided with structure deterioration detection software 10 (also referred to as an application). The computer 2 is installed with a client program 20 of the structure deterioration detection software 10. A server program 30 of the structure deterioration detection software 10 is installed in the computer 3. The client program 20 of the computer 2 implements a client function, and the server program 30 of the computer 3 implements a server function. The client program 20 and the server program 30 cooperate with each other via client-server communication via the communication network 6.

  The structure deterioration detection software 10 realizes a deterioration detection function and a visualization function based on program processing by a CPU or the like. The deterioration detection function performs processing for learning and diagnosing deterioration such as cracks in an image using a deep learning model (DL-CNN) 31 including CNN. The process of the deterioration detection function includes a correct answer process. In the deterioration detection function, the input size of the image patch at the time of training for the CNN model 31 is set to a constant first input size, and the input size of the image patch at the time of diagnosis of the target image is set to the second input based on the variable-size image. Let it be a size (greater than the first input size). The visualization function visualizes information including the target image and the degradation diagnosis result image, displays the information on the GUI screen 21, and performs a visualization process for accepting an input operation by the user.

  The client program 20 of the computer 2 provides a user with a GUI screen (visualization screen) 21 serving as a graphical user interface (GUI) based on communication with the service of the server program 30. The client program 20 of the computer 2 is in charge of user operation input and display processing of the GUI screen 21. When the computer 2 includes a touch panel display device, the GUI screen 21 may be a touch operable screen.

  The client program 20 of the computer 2 inputs image data of an image captured by the camera 4 with respect to the structure 5 and stores it in a storage device on the computer 2 side or a DB 32 on the computer 3 side. As the storage device and the DB 32, an image memory handled by a CPU or GPU, a non-volatile memory for storage, various storage devices, a DB server, and the like are applicable. The image data input to the computer 2 is accompanied by image information such as attribute information. The image information of the image data includes information such as the number of pixels of the camera 4, the angle of view (or the focal length and the sensor size capable of calculating the angle of view), etc., in addition to the ID (identification information) and the shooting date and time. The image information may include object distance information (distance between the camera 4 and the surface of the structure 5). Or the computer 2 may set image information, such as a camera pixel number, to image data based on a user's operation. The computer 2 may calculate the object distance or the like by image processing. An object distance or the like may be measured using a distance sensor or the like different from the camera 4 and input together with the image data.

  The server program 30 of the computer 3 constitutes a deep learning model (DL-CNN) 31 including CNN, and performs processing such as deterioration learning and diagnosis by calculation using the model 31. The server program 30 of the computer 3 is in charge of a calculation process using a deep learning model 31 that generally has a high calculation load. Further, the server program 30 transmits screen data for the GUI screen 21 to the computer 2. The server program 30 stores and manages various data and information including the model 31 in a DB (database) 32.

  Examples of data and information in the DB 32 include image data, structure data, and diagnostic data. The image data is data such as an image group captured by the camera 4, a training image group, and a diagnostic result image group. The structure data is management information related to the structure 5, three-dimensional object data, or the like. The diagnosis data is data that summarizes the result of the inspection work including the result of diagnosing the deterioration of the structure 5 by the structure deterioration detection system of the first embodiment and the user.

  The computer 2 that is a PC includes known elements such as a CPU, ROM, RAM, nonvolatile memory, input devices such as a mouse and a keyboard, output devices such as a liquid crystal display device, input / output interface devices, communication interface devices, and the like. Prepare. The computer 3 as a server includes known elements such as a CPU, a ROM, a RAM, a nonvolatile memory, an input device, an output device, an input / output interface device, and a communication interface device.

  The configuration is not limited to the configuration example of the computer system 1 in FIG. For example, a form in which the processing of the computer 2 and the processing of the computer 3 are integrated into one computer may be used. A mode in which main processing is performed by a portable information terminal device with a camera function possessed by the user may be employed. The technical means for acquiring the image group of the structure 5 is not limited to manual shooting with the camera 4, and various means can be applied.

[Structural deterioration detection software]
FIG. 2 shows a configuration related to the structure deterioration detection software (application) 10. The application 10 includes a data set 101, a network configuration 102 (corresponding to the model 31), and a GUI screen 103 (corresponding to the GUI screen 21). The application 10 includes a first processing function 11 (training / diagnosis function), a second processing function 12 (visualization function), a camera image input function 13, a MIL rotation function 14, a model input size setting function 15, an evaluation / narrowing function 16, and the like. Have

  The data set 101 includes an original image group 211 and a weak point image group 212 as image data. The original image group 211 is an image group obtained by capturing the surface of the object 5 with the camera 4. The original image group 211 includes an image including an actual crack and an image not including an actual crack. The weak point image group 212 is an image group for learning the weak points of the model 31 as described later. The image data of the data set 101 includes a training image, a correct answer image, a diagnosis target image, a diagnosis result image, and the like based on the image of the camera 4. The training image is an image for learning the model 31 before the diagnosis at the time of inspection work. The correct answer image is an image that the user corrects and inputs the diagnosis result image. The diagnosis target image is a target image to be actually diagnosed at the time of inspection work, and is a variable size that is not limited to a certain size. The diagnosis result image is an image output as a calculation result from the model 31.

  The camera image input function 13 inputs image data and image information from the camera 4 and manages them as a part of the data set 101. The application 10 creates and manages management information 213 including image information.

  The first processing function 11 (training / diagnosis function) manages the network configuration 102 (model 31) used for training and diagnosis. The network configuration 102 has a MIL rotation 221 and a widening convolution 222. The first image of the model 31 has an image patch having a predetermined first input size during training. Based on the model input size setting function 15, the first input size is set as a size that is changed according to the parameters of the model 31 (such as the filter size of each layer), and after the parameters of the model 31 are initially set. Is set as a predetermined fixed size accordingly. A diagnostic result image (deterioration diagnostic result information) is provided as the second image output from the model 31. One diagnostic result image has a predetermined first output size corresponding to the first input size of one image patch. Each pixel in the diagnosis result image has a probability value that represents the possibility of deterioration as a probability.

  The first processing function 11 includes a MIL rotation function 14. The MIL rotation function 14 performs processing of MIL rotation 221. MIL is Multiple Instance Learning, which represents the concept of learning by inputting a plurality of instances into a model. The MIL rotation 221 is processing that takes into consideration the characteristics of deterioration that is unique to the first embodiment, and includes processing for generating a plurality of images by rotating the original image so as to be able to cope with the direction of deterioration. The MIL rotation 221 can turn on / off the function according to the user's operation and setting, and is performed in the on state, and is omitted in the off state.

  Widening convolution 222 (Dilated Convolution) is an arithmetic processing using a widening convolution filter as one type of convolution processing. The filter of the widening convolution 222 is defined to have a stride number of 1 and a dilate number of 2 or more. The model 31 may include convolution processing (the number of filter dilates is 1) that is not widened in some layers.

  The widening convolution 222 includes a fully coupled convolutional layer, which is implemented as a widening convolutional layer. The fully connected convolution layer is a layer for classifying and extracting feature information (feature amount) extracted through the widening convolution layer.

  The GUI screen 103 displays the target image 231, the training image, and the diagnosis result image 232, and displays the small region removal image 233, the straight line removal image 234, the correct answer image (third image) 235, and the like. The diagnosis result image 232 includes a multi-tone image and a binarized image. When the user changes the threshold value on the GUI screen 103, a binarized image corresponding to the threshold value is displayed. Further, when the user operates on the GUI screen 103, the small area removal image 233 and the straight line removal image 234 are displayed. Further, the correct answer image 235 is displayed when the user operates the GUI screen 103 and inputs correct answer. The user inputs, as correct answer information, information indicating whether or not the degradation estimation result by the model 31 is correct for each pixel of the diagnosis result image. The correct answer image including the correct answer information is reflected in the model 31 as teacher information.

  The evaluation / narrowing function 16 evaluates the diagnosis result image, and narrows down the diagnosis result image including the deterioration portion estimated to be highly likely to be deteriorated by the DL from the plurality of diagnosis result images. At that time, the evaluation / narrowing function 16 may binarize the probability value for each pixel using the threshold value of the deterioration estimation probability, and extract a portion that is equal to or higher than the threshold value as a deterioration portion. The evaluation / squeezing function 16 narrows down the image by determining that the degradation area is large or the degradation degree is large, for example, when the number of pixels in the degradation area (greater than or equal to the threshold) in the image is greater than or equal to a predetermined threshold. But you can. By narrowing down, for example, 100 images are extracted from 1000 diagnostic result images. The user can make a final determination by visually confirming the narrowed down image. The user can arbitrarily specify and check other images that have not been narrowed down.

[Deep learning]
The following is a supplementary explanation of known deep learning and CNN. Basically, the CNN performs matrix product and activation function operations on the input. However, in the case of image input, [number of dimensions of image input] = [number of pixels] × 3 (3 corresponds to R, G, B color pixels), and the number of input nodes of the network is very large (described later). FIG. 18). Due to the nature of the image, meaningful information is condensed into adjacent pixels. Therefore, in CNN, a matrix product between adjacent pixels is obtained. CNN uses small parameters (corresponding filters) such as 3 × 3, 5 × 5, etc. instead of obtaining a matrix product with parameters having the same size as the image input dimension (for example, the number of vertical pixels × the number of horizontal pixels). To reduce the total number of parameters in the model. Since a product with a size that does not match cannot be obtained, CNN uses a convolution process instead of a normal matrix product. In the CNN, higher-dimensional features are extracted as feature maps by repeatedly performing convolution processing on the input using a convolution filter (using a plurality of layers). The number, size, layer depth, etc. of the filters are all called hyperparameters and need to be designed or set by the person.

  In conventional deep learning, in order to increase accuracy, it is effective to reflect teacher information in a model based on correct answering work by a user. Also in the structure deterioration detection system of the first embodiment, the accuracy of diagnosis is improved by reflecting the teacher information (correct answer image) based on the correct answer work on the model 31.

[Comparative example: DL-CNN model]
FIG. 18 shows an image patch, a DL-CNN model, and the like in the structural deterioration detection system of the comparative example with respect to the first embodiment. (A) shows the diagnostic object image 181. The diagnosis target image 181 is, for example, a square image in which the number of vertical pixels in the vertical direction (y) is m, the number of horizontal pixels in the horizontal direction (x) is m, and the total number of pixels is m × m = M. (B) shows an image patch 182 of a predetermined size to be input to the DL-CNN model 183. The image patch 182 is a square image having n vertical pixels, n horizontal pixels, and a total pixel number of n × n = N. The size of the diagnosis target image 181 in (A) is larger than the size of the image patch 182. The size of the diagnosis target image 181 varies and is often different from the input size of the model 183. The pixel is composed of R, G, and B color pixels. An image patch 182 is cut out from the diagnosis target image 181 for each pixel. That is, M image patches 182 are cut out from M pixels of the diagnosis target image 181. The central pixel of the image patch 182 is a pixel whose deterioration probability is calculated by the model 183.

  The model 183 in (C) includes an input layer, a plurality of hidden layers, a fully coupled layer, an output layer, and the like as a known network configuration. The hidden layer includes a convolution layer and a pooling layer. The pixel value of each pixel of the input image (image patch 182) is input to each of a plurality of nodes in the input layer. The number of input nodes is n × n × 3. Between each node in the input layer and each node in the hidden layer, a convolution operation process using a convolution filter is performed. The convolution filter 184 of the conventional example has a size of 3 × 3 in length and width, for example, and has no widening. The convolution filter 184 has a size smaller than that of the image patch 183, and is applied to the pixel group of the image patch 183 with a predetermined stride. The number of strides of the convolution filter 184 of the conventional example is 2 or more. The stride number corresponds to the amount of movement between the central pixels when the filtering process is repeated. A degradation estimation probability value related to the center pixel of the image patch 183 is output through all the coupling layers and the output layer of the model 183.

  As described above, in the case of the comparative example, in order to diagnose the entire diagnosis target image 181, the calculation of the model 183 is repeatedly performed in the same manner M times corresponding to M pixels and M image patches 183. There is a need. Therefore, it takes a long calculation time corresponding to the number M.

[DL-CNN model]
FIG. 3 shows an image patch, DL-CNN model 31 and the like in the structure deterioration detection system of the first embodiment. (A) shows the target image (first image) 301. The target image 301 is, for example, a rectangular image in which the number of vertical pixels in the vertical direction (y) is c1, the number of horizontal pixels in the horizontal direction (x) is c2, and the total number of pixels is c1 × c2. (B) shows an image patch 302 of a predetermined input size for input to the model 31. It is assumed that the image patch 302 has a square image in which the number of vertical pixels is n, the number of horizontal pixels is n, and the total number of pixels is n × n = N. The size of the target image 301 is larger than the size of the image patch 302. The size of the target image 301 is a variable size and is larger than the input size of the model 31. A plurality of image patches 302 are cut out from the target image 301 as necessary. Details of the clipping will be described later.

The model 31 of (C) includes an input layer, a plurality of hidden layers, a fully coupled convolution layer, an output layer, and the like as a network configuration. The hidden layer includes a widening convolution layer. A pixel value of each pixel corresponding to the input size (n × n) of the image patch 302 that is an input image is input to each of a plurality of nodes in the input layer. The number of input nodes is n × n × 3. Between each node of the input layer and each node of the hidden layer, a widening convolution calculation process using a widening convolution filter is performed. Of the layers of the CNN model 31, for example, the first few layers may be applied with a non-widened convolution filter. The widening convolution filter 304 has, for example, a size of 3 × 3 for the vertical and horizontal calculation target pixels (hatched pattern portion), a total size of 5 × 5, and a dilate number of 2. The dilate number corresponds to the widened number between calculation target pixels. The widening convolution filter 304 has a size smaller than that of the image patch 302 and is applied to the pixel group of the image patch 302 with a stride number = 1. Through the all combined convolution layer and output layer of the model 31, the deterioration estimation probability value for the pixel of the image patch 302 is output. Further, in the first embodiment, a diagnosis result image (image patch 303) composed of a plurality of vertical and horizontal pixels is output from the model 31, and each pixel has a deterioration estimation probability value.
* Renumbering of paragraph numbers In the first embodiment, the output of the model 31 is configured as an image patch 303. The image patch 303 has an output size of q × q. The output size is proportional to the input size. The output size is a size obtained by subtracting a fixed value (E) from the input size. The fixed value E is a rectangular size required for the deterioration estimation calculation for one pixel, and is also the minimum input size. The fixed value E depends on the parameters of the model 31 (such as the filter size of each layer). At the time of training, for example, an image patch having a fixed value E and a minimum input size (image patch 302 having a first input size) is used. Therefore, the first output size during training is q × q = 1 × 1. At the time of diagnosis, an image patch 302 having a second input size larger than the first input size at the time of training is used. Therefore, the second output size at the time of diagnosis is a size of q × q = 2 × 2 or more. For example, when the minimum input size (fixed value E) is 75 × 75, the 75 × 75 image patch 302 is used during training, and the output size of the image patch 303 is 1 × 1. At the time of diagnosis, when an image patch 302 having a larger size, for example, 100 × 100 is used, the output size of the image patch 303 is 26 × 26.

  In FIG. 3, the image patch 302 is cut out from the target image 301 at the time of diagnosis. However, this takes into consideration the efficiency and limitations of calculation (such as GPU memory) in the computer system 1 and is theoretically indispensable. Absent. The first input size at the time of training and the second input size at the time of diagnosis are independent, and the second input size can be selected without depending on the first input size. The degree of freedom of the second input size with respect to the first input size is large.

  As described above, in the case of Embodiment 1, in order to diagnose the entire target image 301, it is not necessary to repeat the calculation of the model 31 in the same manner M times corresponding to M pixels. It can be calculated with a smaller number of times than the comparative example. Therefore, the calculation time is shortened.

[Wide convolution filter]
FIG. 4 shows an example of a dilated convolution filter used in the first embodiment. The widening convolution filter 401 in (A) has the same configuration as the widening convolution filter 304 in FIG. 3, and the number of dilates = 2. The size of this filter is 3 × 3 with respect to the pixel to be calculated, and the overall size including the widening is 5 × 5. The pixel value (node value) of the next layer is obtained by a predetermined calculation from the pixel values (corresponding node values) of the central pixel and the surrounding eight pixels. The widening convolution filter 402 in (B) is a case where the number of dilates = 4. The size of the filter is 3 × 3 with respect to the calculation target pixel, and the size including the widening is 9 × 9 as a whole. Other widening convolution filters may be applied.

[Wide-width convolution processing]
FIG. 5 shows an example of the widening convolution operation processing in the first embodiment. An example using the widening convolution filter 402 of FIG. 4 is shown. Each pixel of the first image 500 is indicated by a square. For example, starting from the upper left pixel 501 of the first image 500, attention is paid to each pixel with the stride number = 1 sequentially in the x direction. Repeat the process. A filter 511 that is the widening convolution filter 402 is applied with the first pixel 501 (x1, y1) as the central pixel. Next, the filter 512 that is the widening convolution filter 402 is applied in the same manner with the adjacent second pixel 502 (x2, y1) as the central pixel. As shown in the figure, the widening convolution filter 402 is similarly applied up to the last M-th pixel. Note that when applying the filter, for pixels outside the area of the original first image 500, for example, an appropriate value may be used as padding. By repeating such a widening convolution operation, each pixel value (node value) of the next layer image is obtained.

[Comparative example: Model calculation]
FIG. 19 schematically shows the DL-CNN model 183 and the contents of calculation in the comparative example of FIG. Here, the model 183 is composed of k layers, and shows a first layer L1, a second layer L2, a (k−1) th layer Lk−1, and a kth layer Lk. The first layer L1 corresponds to the input of the image patch 183, and the size of the vertical and horizontal pixels is a1 × a1 = n × n. The depth size is 3 (corresponding to R, G, B color pixels). A region of the size (n × n × 3) is illustrated by a rectangular parallelepiped. a1 = n is 232, for example.

  As described above, the convolution process using the convolution filter is applied to each pixel of the image of the first layer L1. A convolution filter G1 is used in the convolution process CNV1 when obtaining the image of the second layer L2 from the image of the first layer L1. For example, as in FIG. 18, the convolution filter G1 is not widened, has a size of 3 × 3, uses, for example, 24 as the filter type number g2, and has a stride number of 2 or more. A broken line schematically shows how a pixel value of the next second layer L2 is obtained by applying the process of the convolution filter G1 to one pixel of the first layer L1.

  The image of the second layer L2 is reduced in size as a result of the convolution process CNV1. The size of this image is indicated by a2 × a2 × g2. For example, a2 is 116. Similarly, the convolution processing or the like is applied to the third and subsequent layers, the feature amount is extracted, and the size is reduced. For example, in the (k−1) th layer Lk−1, the image size is 3 × 3 × gk−1. The filter type number gk−1 is 45, for example. A fully combined convolution process FCCNV1 is applied to the image. At that time, the filter has a size of 1 × 1 and two types. As a result, in the k-th layer Lk, the image information has a size of 3 × 3 × 2, and each pixel has a deterioration estimation probability value. The image information is output.

[Model calculation]
FIG. 6 schematically shows the DL-CNN model 31 and the content of calculation in the first embodiment. Here, the model 31 is assumed to be composed of j layers, and shows a first layer L1, a second layer L2, a (j-1) th layer Lj-1, and a jth layer Lj. The first layer L1 corresponds to the input of the image patch 302, and the size of vertical and horizontal pixels is b1 × b1 = n × n. The depth size is 3. A region of the size (input size) (n × n × 3) of the input image patch 302 is illustrated by a rectangular solid. For example, b1 = n is 232, which is the same as a1 = n = 232 in the comparative example.

  As described above, the widening convolution process using the widening convolution filter is applied to each pixel of the image of the first layer L1. In the widening convolution process DCNV1 when obtaining the image of the second layer L2 from the image of the first layer L1, the widening convolution filter F1 is used. For example, the widening convolution filter F1 has a size of 3 × 3, uses 24 types, and the number of dilates = 4 and the number of strides = 1, similarly to the widening convolution filter 402 of FIG. The number of filter types f2 of the second layer L2 is 24.

  A state in which a pixel value of the next second layer L2 is obtained by applying the processing of the widening convolution filter F1 to one pixel of the first layer L1 is schematically shown. The size of the image of the second layer L2 is reduced as a result of the widening convolution process DCNV1. The size of this image is indicated by b2 × b2 × f2. For example, b2 is 224 and f2 is 24, for example. Similarly, the widening convolution process or the non-widening convolution process is applied to the third and subsequent layers, and feature quantities are extracted and the size is reduced. For example, in the (j−1) th layer Lj−1, the image size is bj−1 × bj−1 × fj−1. The number of filter types fj-1 in the (j-1) th layer Lj-1 is 45. A fully combined convolution process FCCNV1 is applied to the image. At that time, the filter has a size of 1 × 1 and two types. As a result, in the j-th layer Lj, an image having a size of bj × bj × 2 (diagnosis result image corresponding to the image patch 303) is obtained, and has a deterioration estimation probability value for each pixel. The image is output. The size bj is 142, for example. This size bj is larger than the size of the comparative example = 3.

  As described above, in the first embodiment, as a result of the calculation of the model 31 once, a diagnostic result image having an output size of bj × bj is obtained like an image of the jth layer. Therefore, the diagnosis result image can be obtained by performing the calculation less than M times for the diagnosis target image.

[Image size]
FIG. 7 shows the relationship between the image size and the like during training and diagnosis in the first embodiment. (A) shows the size of the image at the time of training, and the like. As in FIG. 6, the first image patch that is the first image to be input has b1 × b1 as the first input size, and is 91 × 91, for example. The first diagnosis result image that is the output second image has bj × bj as the first output size, for example, 1 × 1. The first image patch input to the model 31 is fixed to the first input size.

  (B) shows the size of the image at the time of diagnosis. The target image 701 (indicated by a solid rectangle) that is the first image has a variable input size that is equal to or larger than the first input size, and is c1 × c2 as the number of vertical × horizontal pixels. The target image 701 may be non-square. c1, c2 ≧ b1.

  The first processing function 11 cuts out a plurality of images from the target image 701 with the second input size (b1 × b1). The second input size (b1 × b1) at the time of this diagnosis may be different from the first input size (b1 × b1) at the time of training. The first processing function 11 may determine the second input size as the second input size, for example, in accordance with the size of the GPU memory so as to be as large as possible below the maximum value of the memory usage rate. . Thereby, it is possible to calculate in a short time by making the best use of the computer performance. The first processing function 11 classifies the c1 × c2 region of the target image 701 by the second output size (bj × bj) corresponding to the second input size (b1 × b1). This takes into account that the output size is smaller than the input size. A square region (especially, six regions (1) to (6)) of the second input size (b1 × b1) and the second output size (bj × bj) is shown. A region of the second output size is indicated by a broken line.

  The first processing function 11 cuts out each region of each second input size (b1 × b1) corresponding to each divided second output size (bj × bj). The regions of the second input size to be cut out partially overlap with each other. The first processing function 11 cuts out a plurality of images so as to cover all the pixels of the target image 701. In this example, the case where it can cover by cutting out 6 images is shown. Further, when cutting out, the remainder outside the target image 701 is set to an appropriate pixel value by, for example, padding.

  A plurality of cut images (cut images), for example, images 711 to 716 are included. Each cut-out image is set as a second image patch. The first processing function 11 inputs a plurality of second image patches to the model 31 and applies the calculation. In this example, the calculation is six times. As a result, a plurality of respective diagnosis result images (second diagnosis result images), for example, image 721 to image 726 are obtained. The first processing function 11 obtains one second diagnostic result image 702 by connecting the obtained second diagnostic result images.

[Processing during training]
FIG. 8 shows a processing flow during training in the application 10 (particularly, the first processing function 11) of the computer system 1. This training processing includes training image data creation processing. FIG. 8 includes steps S1 to S9. Hereinafter, it demonstrates in order of a step.

  (S1) In S1, the computer system 1 (particularly the camera image input function 13) inputs the original image group 211 of FIG. The original image group 211 includes a plurality of images obtained by photographing the structure 5, and particularly includes image data for training including deterioration such as actual cracks. For example, the computer 2 inputs the image data of the camera 4 and stores it in the DB 32. The computer 3 sequentially reads out images from the image group stored in the DB 32 to the image memory.

  (S2) In S2, the computer system 1 displays a diagnosis result image obtained by inputting and calculating the original image into the model 31 on the visualization screen (GUI screen 21), and the image based on the manual operation of the user. The correct answer work for is performed. For example, when a pixel estimated to be deteriorated among the pixels of the diagnosis result image is not actually deteriorated, or a pixel estimated to be non-degraded is actually deteriorated, a value indicating an incorrect answer is Entered. The computer system 1 stores the correct answer image (third image) to which the correct answer information is input as a part of the training image.

  (S3) In S3, the computer system 1 inputs the weak spot image group 212 of FIG. The weak point image group 212 is an image that causes erroneous detection as a result of the calculation of the model 31, and is an image including, for example, a straight line group of the wall surface of the structure 5, surrounding plants, and the like.

  (S4) In S4, the computer system 1 performs edge detection processing, etc., on the weak point image group 212 to perform automatic correct answer processing. As a result, the correct weak point image is converted into the training image. Save as part (negative sample image). The automatic correct answering process is a process of setting a value representing non-deterioration, for example, to pixels corresponding to a group of straight lines or plants in the image.

  (S5) In S5, the computer system 1 applies the necessary number of first image patches of the first input size (n × n) for training to the images of the training image data based on S2 and S4. Cut out (FIG. 7).

  (S6) In S6, the computer system 1 performs a data expansion process on the first image patch in S5 to generate a plurality of training images (variations) in order to increase the number of training images. The data expansion processing is processing according to the type and characteristics of deterioration such as cracks, and includes known processing such as addition of noise and inversion, and does not include shift processing (processing for moving the pixel area in parallel).

  (S7) In S7, the computer system 1 performs the MIL rotation process when the MIL rotation function 14 is turned on by user setting (FIG. 11). In the MIL rotation process, the original image is rotated so as to be equally divided within a range of 180 degrees, for example, using a predetermined rotation angle θ, and a plurality of rotated images are generated.

  (S8) In S8, the MIL rotation function 14 of the computer system 1 inputs the calculation into the model 31 for each of the plurality of first image patches generated as described above, that is, executes the diagnosis for training, A plurality of diagnosis result images are obtained. Then, the MIL rotation function 14 acquires an image obtained by integrating a plurality of diagnosis result images into one, and stores the result.

  (S9) In S9, the computer system 1 (especially the evaluation / narrowing function 16) performs an evaluation process on the diagnosis result image and makes a final determination by the user. The user looks at the diagnosis result image on the GUI screen 21 and confirms whether or not the degradation estimation result by the model 31 is correct, and inputs correct answer information. The computer system 1 stores the correct answer image.

  Moreover, the evaluation process of S9 includes evaluation and determination for a specific rotation direction according to the structure 5 (identified by the structure ID). The computer system 1 determines a specific rotation direction (corresponding rotation angle θ) based on the image (image g40 in FIG. 11) obtained as a result of the MIL rotation process in S8, and stores the information. As an example of the evaluation process, the user tags pixels corresponding to deterioration in each image for a plurality of images in advance and stores them as evaluation data. And calculating the quality of the model 31 with respect to the data for evaluation is mentioned.

[Weak image]
FIG. 9 shows weak image settings and correct images. In FIG. 9, an original image group 211, a weak point image group 212, a model 31, a diagnosis result image (second image), and the like are shown. The application 10 performs training by inputting an original image or a weak point image into the model 31.

  When an image is input to the model 31 and deterioration is diagnosed and detected, not a deteriorated portion such as a crack but another portion (non-deteriorated portion) is erroneously detected as a deteriorated state. Examples of erroneous detection include a group of straight lines originally provided as a design on the wall surface of the structure 5, surrounding plants, and the like. In the case of general machine learning, such erroneous detection can usually be dealt with by increasing the number of learning data and improving accuracy. However, in the system of the first embodiment, since it is desired to shorten the calculation time, a mechanism is provided for taking measures instead of simply increasing the number of learning data (number of images).

  The system according to Embodiment 1 has a function of reducing the number of false detections and enabling efficient learning even when the number of training image data is limited. In this system, a weak point of erroneous detection of the DL-CNN model 31 is extracted and set as a weak point image, and the weak point image is input to the model 31 to perform training so as to overcome the weak point. The data set 101 is expanded to use the weak point image group 212.

  The weak point image is an image including a straight line group or an image including a plant, as in the example of FIG. The application 10 extracts a part with many false detections based on the evaluation from the diagnosis result image input to the model 31 based on the original image, and sets it as a weak point image. In other words, the weak point image is a negative sample image representing a case that does not include deterioration. Alternatively, the application 10 can set an image arbitrarily designated by the user as a weak point image. The set weak point image group 212 is added as a weak point emphasis data set to the data set 101 of the regular original image group 211 and is expanded as a new data set 101.

  The weak point image may be automatically extracted from the diagnosis result image of the original image, or may be an image in which the user arbitrarily designates an image that has nothing to do with the structure 5. For example, among the plurality of diagnosis result images, a diagnosis result image 901 including an erroneously detected portion is extracted, an area of an erroneously detected portion (for example, a straight line group, a plant) in the diagnostic result image 901 is extracted, and the region is processed. A weak point image is created. Since it is not necessary for the weak point image to be an image manually created by the user or an image that has been correctly answered, labor for the weak point image is reduced.

  In FIG. 9, for a diagnosis result image 902 including a deterioration estimated place among a plurality of diagnosis result images of the training result, the user inputs whether the pixel of the deterioration estimated place is a correct answer (incorrect answer) on the screen. Perform correct answering work. As a result, a correct image with the correct answer information input is created and becomes a part of the data set 101.

  FIG. 10 shows an example of the weak spot image. (A) is an image including a straight line group, and (B) is an image including a plant. These images do not include deterioration such as cracks. When these images are input to the model 31, the straight line group or the plant part reacts with sensitivity, and is erroneously detected and erroneously detected as deterioration such as cracks (deterioration estimated part). Therefore, the model 31 is learned using such an image as a weak point image. In the model 31 after learning by inputting the weak point image, when the actual target image is diagnosed, false detection of deterioration of the straight line group of the wall surface and surrounding plants is reduced.

  As an example of how to create and set a weak point image, a known edge detection process is applied to the input image (for example, the image of FIG. 10A) by the computer system 1 as shown in S4 of FIG. Then, the straight line group region may be extracted, the straight line group region may be processed to have a predetermined size, and the result may be set as a weak point image.

[MIL rotation function]
Regarding the DL-CNN, in the case of diagnosis of a general object (for example, a person) in an image, since there is a direction of gravity (usually a downward direction in the image), the direction of the object in the image is limited to some extent. Judgment is possible. On the other hand, the main direction of the deteriorated portion in the image is basically unknown. For example, it is basically unknown which crack direction is caused by extending along the main direction in any angular direction within an angle range of 360 degrees in the image plane. Usually, a model (referred to as a first model) that can handle all angle directions regardless of the direction in the image plane is created so that deterioration can be detected at an arbitrary angle of 360 degrees.

  Here, if the main angular direction of degradation in each image is aligned with the same specific angular direction (for example, in-plane vertical direction), it is not necessary to correspond to all angular directions as in the first model, and the specific It is possible to cope with a model corresponding to the angle direction (referred to as a second model). That is, when the second model is used, the same level of performance can be obtained with a smaller number of parameters than the first model. However, if the user manually performs the work of aligning the deterioration angle direction in the image, it takes time and effort, and the main point is mistaken.

  For example, when the second model is a model corresponding to the deterioration occurring in the in-plane vertical direction, and the deterioration portion in the input image is mainly generated along the in-plane vertical direction, the deterioration is highly likely to occur. It can be detected (in other words, a high probability value is output as the deterioration estimation probability). The second model can detect the deterioration only with a low probability when the deterioration portion in the input image is generated in an angular direction deviated from the vertical direction in the plane (in other words, a low probability value is used as the deterioration estimation probability). Output).

  Therefore, the system according to the first embodiment has a function capable of handling both the first model and the second model, and in particular, has a MIL rotation function 14 for performing learning corresponding to the second model. When the MIL rotation function 14 is used, a plurality of images (for example, images g1 to g3) are generated by rotating the training target image at a predetermined rotation angle θ as in the example of FIG. The model 31 is inputted to make a diagnosis for trial. The evaluation / narrowing function 16 selects the angular direction of the image corresponding to the one with the highest probability value as the specific angular direction from the respective deterioration estimation probabilities of the diagnosis result images of a plurality of images. The system according to the first embodiment diagnoses the target image using the second model corresponding to the specific angular direction at the time of diagnosis. Thereby, degradation, such as a crack, can be detected with high accuracy while reducing the user's work.

[MIL rotation processing]
FIG. 11 shows an example of MIL rotation processing by the MIL rotation function 14 during training in the first embodiment. The image g0 is an input image example. The image g0 is an image in which the character “A” appears in a normal direction. Let θ be the rotation angle associated with the MIL rotation processing. In the application 10, the value of the rotation angle θ is set in advance. The MIL rotation function 14 performs a rotation process 801 for rotating the image g0 in a plurality of directions using the rotation angle θ. In this example, three types of rotation angles θ of 0 degrees, θa degrees, and θb degrees are shown, but the present invention is not limited to this. An image g0 rotated by 0 degrees, an image g2 rotated by θa degrees, an image g3 rotated by θb degrees, and the like are obtained. Since image g1 is 0 degrees, it is not rotated. The image g2 is a square that includes a region whose sides are inclined by rotation of θa degrees.

  The MIL rotation function 14 performs a diagnostic process (DNN 802) in which the rotated images (images g1 to g3) are input to the DL-CNN model 31 and calculation is applied. As a result of DNN 802, images corresponding to the respective rotation directions, for example, images g11, g12, and g13 are obtained. These images include a region where the sides after rotation are slanted. Since it is necessary to integrate the results of a plurality of images, reverse rotation or the like is required for each image. The reverse rotation angle is a minus angle of the rotation angle θ. The MIL rotation function 14 performs reverse rotation processing 803 regarding the rotation angle θ on each image. That is, the rotation process is performed with the rotation angles set to 0 degrees, -θa degrees, and -θb degrees. As a result, each image, for example, images g21, g22, and g23 are obtained. For example, the image g22 is a square that includes a region whose sides are slanted by a rotation of −θa degrees.

  The MIL rotation function 14 performs a clipping process 804 for cutting out an area corresponding to the size of the original image from each image (images g21 to g23) after the reverse rotation. As a result, for example, images g31, g32, and g33 are obtained as images corresponding to the feature map. For example, an image g32 cut out from the image g22 is included. The image g31 is represented by f (x, y, 0). The image g32 is represented by f (x, y, a). The image g33 is represented by f (x, y, b). Each image after clipping has a degradation estimation probability value for each pixel.

  The MIL rotation function 14 performs a max pooling process 805 on all images (images g31 to g33) at the rotation angle θ. The max pooling process 805 is a process for obtaining the maximum value corresponding to the higher probability of deterioration for each pixel at the corresponding position in the image. As a result of this processing, one image g40 is obtained as an output. The image g40 is represented by max (f (x, y, i)). As described above, when each image obtained by rotating the original image is input to the model 31, the rotated diagnosis result information is output. Therefore, in order to integrate a plurality of diagnosis result information, reverse rotation or clipping processing is performed. .

  Details of the rotation angle θ of the MIL rotation processing are as follows. The rotation angle θ can be basically any angle. The rotation angle θ is preferably an equally divided angle within the 360 ° range in the image plane, but is not particularly limited. Since the crack is a substantially linear pattern, it can be said that the direction of the crack is the same even when the image including the crack is rotated 180 degrees. Therefore, in the first embodiment, as an implementation example, the MIL rotation function 14 generates a plurality of images by rotating in an equal division within a range of 180 degrees for each predetermined rotation angle θ. According to the experiment, in the range of 180 degrees, the image is divided into two with a rotation angle θ = 90 degrees (two types of images of 0 degrees and 90 degrees), and the image is divided into four with a rotation angle θ = 45 degrees (0 degrees). , 45 degrees, 90 degrees, and 135 degrees) are equally divided at a predetermined rotation angle θ to obtain a result that balances accuracy and processing speed.

[Diagnosis processing]
FIG. 12 shows a processing flow at the time of diagnosis (actual diagnosis) of the target image during inspection work or the like. This diagnostic processing includes diagnostic processing performed in advance in the computer system 1. The user designates diagnostic processing on the screen and waits until the calculation is completed. FIG. 12 includes steps S21 to S27. Hereinafter, it demonstrates in order of a step.

  (S21) The computer system 1 inputs the target image of the structure 5 (identified by the structure ID) to be diagnosed. For example, the computer 3 performs a preparation process such as reading the target image from the DB 32 and developing it in the image memory.

  (S22) When the computer system 1 (especially the MIL rotation function 14) performs diagnosis using the model B learned about the specific rotation direction, the computer system 1 (the rotation angle θ) displays the diagnosis target image as the specific rotation direction (rotation angle θ). And take a square that encompasses the area after rotation. The computer system 1 uses the rotated image as an image for inputting to the model 31 (model B).

  (S23) The computer system 1 (especially the first processing function 11) uses, for example, a GPU memory and a processing size corresponding to the size of the GPU from the second input size (FIG. 7, variable input size, c1 × c2) of the target image. The target image is cut into two input sizes to obtain a plurality of images (second image patches).

  (S24) The computer system 1 inputs a plurality of second image patches to the DL-CNN model 31 in order and applies the calculation, that is, executes a diagnosis process. As a result, a plurality of second diagnosis result images are sequentially obtained.

  (S25) The computer system 1 repeats the process of S24 in the same manner while confirming whether the model calculation (diagnosis process) of all of the plurality of second image patches has been completed.

  (S26) The computer system 1 reversely rotates each of the plurality of second diagnosis result images in correspondence with the specific rotation direction described above, and selects the original size (second output size) from the images after the reverse rotation. ) Are cut out to form a plurality of second diagnosis result images (deterioration probability images).

  (S27) The computer system 1 obtains one image (second diagnosis result image) by rearranging and connecting a plurality of second diagnosis result images (deterioration probability images) in correspondence with the separation of S22. ,save.

[Visualization screen display processing]
FIG. 13 shows a processing flow of the visualization screen display by the second processing function 12 of the computer system 1. This process is a real-time process corresponding to a user operation on the visualization screen. A user performs an input operation on the GUI screen 21 of the computer 2 in FIG. 2, and a request or the like is sent to the computer 3. The computer 3 processes the request, generates screen data, and responds to the computer 2. And the computer 2 displays a visualization screen based on screen data. FIG. 13 includes steps S31 to S38. Hereinafter, it demonstrates in order of a step.

  (S31) The computer system 1 inputs a diagnostic result image designated based on a user operation. For example, the computer 3 develops the image data of the diagnostic result image read from the DB 32 in the image memory.

  (S32) The computer system 1 (particularly the second processing function 12) displays a diagnostic result image of the type specified based on the user's operation in a predetermined area on the visualization screen. Further, the computer system 1 displays components such as operation threshold values (deterioration probability threshold value, region size threshold value) and command buttons on the screen, and displays information such as the ID of the target structure 5 and the target image. indicate. A display example of the visualization screen is shown in FIG. The computer system 1 changes the threshold when the threshold GUI component in the screen is operated by the user.

  (S33) The computer system 1 generates a binarized image in which the probability value for each pixel in the diagnosis result image is binarized from multiple gradations according to the deterioration probability threshold (binarization threshold), and the diagnosis result An image.

  (S34) In addition, the computer system 1 extracts a small region in which the area (number of pixels) of an adjacent region representing deterioration in the binarized image of the diagnosis result image is equal to or smaller than the region size threshold according to the region size threshold. Then, a small area removal image from which the small area has been removed is generated and used as a diagnosis result image.

  (S35) In addition, the computer system 1 determines the binary value of the diagnosis result image in accordance with the designation of the straight line removal by the user operation (for example, on setting of the straight line removal button or predetermined key input corresponding to the setting of the straight line removal command). A linear region in the digitized image is extracted, and a straight line-removed image from which the linear region is removed is generated and used as a diagnosis result image.

  (S36) The computer system 1 displays a diagnostic result image reflecting the processing of S33 to S35 in a predetermined area in the screen. The user can check the presence / absence and location of deterioration while appropriately changing the threshold value by looking at the diagnosis result image on the visualization screen, and can make a final determination.

  (S37) Moreover, the computer system 1 accepts a correct answer operation by a user operation for the diagnosis result image. The user can designate and input whether or not the deterioration estimation result is correct for each pixel in the diagnosis result image. The computer system 1 stores the correct answer image (third image) including the input correct answer information as a part of the data set 101.

  (S38) The computer system 1 ends the display of the screen in response to the user's end operation on the screen, etc. If the operation is not the end operation, the processing is repeated from S31.

[Visualization screen (GUI screen)]
FIG. 14 shows a display example of a visualization screen (GUI screen 21) provided by the second processing function 12. The user visually confirms the image of the deterioration diagnosis result on the visualization screen, and performs the final determination of the deterioration detection and the correct answer work. The second processing function 12 displays the diagnosis result image (degradation probability image) of the output of the model 31 in a predetermined image area in the visualization screen, for example, the first image area 141 and the second image area 142.

  In the diagnosis result image data, the probability value (value of 0 to 1) of the degradation estimation result is provided as a gradation value (for example, a value of 0 to 255) for each pixel. In this image, the probability value for each pixel can be displayed as it is as a multi-tone color of the pixel. Further, in this image, the probability value of the pixel can be binarized using the binarization threshold value, that is, can be displayed as a binary value indicating whether or not the pixel is deteriorated. In this example, a diagnosis result multi-tone image (corresponding to FIG. 15B) is displayed in the image first area 141, and the diagnosis result is displayed on the original image in the image second area 142. An image (corresponding to (D) of FIG. 15) on which the binarized image is superimposed is displayed.

  In the image first area 141 and the image second area 142, other types of images as shown in FIG. 15 can be selected and displayed. In one visualization screen, one image may be displayed in one image area, or two or more images may be displayed in parallel in two or more image areas as in the example of FIG. . Two or more images may be displayed alternately in one image area. Also, user operations such as enlargement / reduction display and shift display can be performed on the image in the image area.

  In the visualization screen, an area for displaying information such as the structure 5 and the ID of the image and information such as a command button is also provided. In this area, for example, a structure ID, an image ID, and the like are displayed and can be selected by a user operation. In this area, operations for selecting and setting functions of the application 10 are possible. For example, the type of image to be displayed in the image area can be selected. In this region, for example, a straight line removal button 147 is provided. When the straight line removal button 147 is turned on, a straight line removal image 234 (S35 in FIG. 13) is displayed in the image area.

  In the visualization screen, a deterioration probability threshold slider 144 and a region size threshold slider 145 are provided as GUI components. With the deterioration probability threshold slider 144, the user can variably operate the deterioration probability threshold (binarization threshold) within a predetermined range. With the operation of the deterioration probability threshold slider 144, the deterioration probability threshold is changed, and the display content of the binarized image in the screen is updated in real time (S33 in FIG. 13).

  Note that the computer system 1 may generate an image corresponding to each binarization threshold in advance and hold it in the image memory, and display the corresponding image on the screen according to the user's operation. In addition, images corresponding to different binarization threshold values may be displayed in parallel on the screen.

  The area size threshold slider 145 allows the user to variably operate the area size threshold within a predetermined range. With the operation of the area size threshold slider 145, the area size threshold is changed, and the display content of the binarized image in the screen is updated in real time using the small area removal image 233 (S34 in FIG. 13).

[Image example]
FIG. 15 shows examples of various images. Each image in FIGS. 15A to 15D has a correspondence relationship. (A) shows an original image to be diagnosed, and is an example of an image 151 including crack degradation (within a broken line frame) on the wall surface of the structure 5. (B) shows a diagnosis result multi-tone image, and in this image 152, the deterioration estimation probability value is expressed in multi-tone for each pixel. For example, the low gradation is blue and the high gradation is red, which is an image like a heat map. (C) shows the binarized diagnostic result binarized image. In this image 153, pixel values are binarized based on the multi-gradation image 152 of (B) and the deterioration probability threshold. For example, when the original multi-gradation pixel value is less than the deterioration probability threshold, the value 0 is represented in black, and when the original multi-gradation pixel value is greater than or equal to the deterioration probability threshold, the value 1 is red. Expressed. The display color can be set by the user. (D) shows an image obtained by superimposing a degraded portion of the binarized image (C) on the image 151 (A). In this example, the degradation of cracks occurs in the image along the vertical direction (y). Further, in the correct answering work, the deterioration area may be made transparent and the outline may be displayed.

  FIG. 16 shows an example of deterioration or the like in the diagnostic result binarized image. The image 161 in (A) is an example in which a diagnostic result binarized image is superimposed and displayed on the original image. This image 161 includes not only a deteriorated portion that is estimated to be deteriorated, but also a misdetected portion such as a linear region 163, as shown in part by a broken line frame. In addition, the image 161 includes a small region (small region group) 164 of a degraded portion according to the region size threshold. (B) is an example of the image 162 after linear area removal and small area removal are performed on the image 161 of (A). In this image 162, the linear region and the small region are removed, so that it is easy to check the deterioration portion and make a final determination. For example, as shown by the alternate long and short dash line frame, a cracked deterioration portion 165 can be confirmed. It can be seen that the deteriorated portion 165 is roughly along a certain direction.

[Small area removal]
It is considered that deterioration such as a crack should be detected as a pixel region continuously connected to a certain extent. Therefore, by removing a region (small region) having a small area from the degraded portion of the diagnosis result image, the user can easily perform the final determination of the degradation detection. The computer system 1 (especially the second processing function 12) determines the area (size) of a deteriorated area in which a deteriorated (value 1) pixel is adjacent in a diagnosis result image, particularly a binarized image, based on the number of pixels. . When the area of the degraded area is equal to or smaller than the area size threshold, the area (small area) is removed without being a degraded part, and is displayed as a small area removed image 233 as in the example of FIG.

[Linear area removal]
In the diagnosis result image, when the region estimated as the deteriorated portion is close to a straight line shape, it is highly likely that the region is not deteriorated such as a crack. For example, there is a possibility of a linear groove or frame originally formed as a design on the wall surface. Therefore, by removing the linear region from the degraded portion in the diagnosis result image, the user can easily perform the final determination of the degradation detection. The computer system 1 (particularly the second processing function 12) applies a well-known straight line detection algorithm process, for example, a probabilistic Hough transform process, particularly to a binarized image of the diagnosis result image. Thereby, a linear region is extracted from the diagnosis result image. The computer system 1 removes the extracted linear region and displays it as a straight line removed image 234 as in the example of FIG.

[Model input size setting]
FIG. 17 shows the model input size setting by the model input size setting function 15 in the first embodiment. This setting process is a process for determining and setting a suitable first input size of the first image patch for the model 31 shown in FIG. (A) shows a plurality of images in the first image corresponding to the image of the camera 4. As a first image related to a certain structure 5 (for example, structure ID = STR1), for example, there are images P # 1, P # 2, P # 3, and the like. These multiple images may differ in size and resolution. As image information associated with an image, a size (SZ), a camera pixel number (PN), a camera angle of view (AN), an object distance (DS), and a resolution (DF) are shown. For example, the image information is attached as attribute information of the image data, or is created or set by the computer system 1. The application 10 performs setting processing using image information of image data acquired from the camera 4. For example, even in the case of degradation of a crack having the same size of 1 mm, in the case of each camera image having a different distance from the degradation (object), the number of pixels in the degradation location is different. Therefore, information on the object distance (DS) is also stored.

  (B) shows the correction process regarding the first image of (A). The size of the plurality of images (A) is corrected so that the resolution is constant. For example, a certain resolution is set as the resolution DFC. For example, in the image P # 3, the size SZ3 is corrected to the size SZ3C so that the resolution DF3 becomes the resolution DFC. The corrected images C # 1, C # 2, and C # 3 have the same resolution DFC. The camera image input function 15 includes a function for performing such correction processing.

  (C) shows the setting of the first input size of the model 31 of DL-CNN. For example, a square having a minimum size is selected from each size (SZ1C, SZ2C, SZ3C) of the plurality of corrected images in (B). The size of the square is determined as the first input size (b1 × b1) of the first image patch input to the model 31, and information is set. This model 31 is a model 31 suitable for diagnosis of the target structure 5 (structure ID = STR1). As described above, by using the model input size setting function 15, a suitable input size can be set, and more accurate diagnosis is possible. In addition, the user can arbitrarily set and specify a suitable input size and output size using model setting items (not shown) on the visualization screen.

[Effects]
As described above, according to the structure deterioration detection system of the first embodiment, it is possible to reduce the time including the calculation time required for learning and diagnosis in the computer and the work time of the user while ensuring the accuracy of deterioration detection. User workload can be reduced. Structure managers can efficiently perform inspection and repair work. Degradation diagnosis using deep learning is possible even in a system with limited computer performance. According to the present system, it is possible to cope with various sizes, resolutions, and the like of the image group of the structure, so that the training and diagnosis work including the image group acquisition work can be efficiently realized. In addition, since learning is performed in consideration of deterioration characteristics such as cracks, the accuracy of diagnosis can be ensured. Further, since a suitable image input size can be selected, the most efficient calculation can be performed in accordance with the performance (GPU or the like) of the computer system 1.

  Other embodiments are also possible as follows. An image based on the three-dimensional object model of the structure 5 is displayed in the visualization screen, and a binarized image or the like is pasted and displayed on the surface of the three-dimensional object of the image while associating the position and direction. May be. Moreover, it is good also as making it possible to confirm the progress degree of degradation, etc. on a screen by associating images for each diagnosis date and time in time series for each position on the structure 5.

  Although the present invention has been specifically described above based on the embodiments, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the invention.

  DESCRIPTION OF SYMBOLS 1 ... Computer system, 2 ... Computer, 3 ... Computer, 4 ... Camera, 5 ... Structure, 6 ... Communication network, 10 ... Structure deterioration detection software (application), 20 ... Client program, 21 ... GUI screen, 30 ... Server program, 31 ... DL-CNN (model), 32 ... DB, 41 ... image, 42 ... deterioration.

Claims (12)

A structure deterioration detection system configured on a computer system for detecting deterioration including cracks on the surface of the structure,
The computer system is
A first process in which a first image obtained by imaging the surface of the structure is input, and a second image including information representing a diagnosis result of the deterioration is output using deep learning;
A second process for visualizing information including the first image and the second image, displaying the information on a screen, and receiving an input operation by a user;
And
The convolutional neural network constituting the deep learning model includes a widening convolutional layer that calculates a widening convolution filter,
The first process includes
A training process for inputting a first image patch having a predetermined first input size to the model and obtaining a first diagnosis result image having a first output size based on the training image data during training;
At the time of diagnosis of the target image of the structure, a second image patch having a second input size is cut out from the target image having a variable size equal to or larger than the first input size, and each second image patch is input to the model. A diagnostic process for obtaining a second diagnostic result image of each of the second output sizes;
Structure deterioration detection system including In the structure deterioration detection system according to claim 1,
The widening convolution filter of the model has a stride number of 1, a dilate number of 2 or more,
The second output size has a plurality of vertical and horizontal pixels,
The second image has a deterioration estimation probability value for each pixel.
Structure deterioration detection system. In the structure deterioration detection system according to claim 1,
The second image has a deterioration estimation probability value for each pixel,
The second process is a process of creating a third image in which correct attachment information indicating whether or not the diagnosis result is correct is input for each pixel based on the user's operation on the second image on the screen. Including
The first process uses the third image for the training process.
Structure deterioration detection system. In the structure deterioration detection system according to claim 1,
The first process includes a data expansion process for creating the training image data,
The data expansion process includes an inversion process and a noise addition process that do not involve a shift process on an image.
Structure deterioration detection system. In the structure deterioration detection system according to claim 1,
The first process is a process of performing the training process using weak point emphasis data for learning a weak point of erroneous detection of the model as the training image data based on a diagnosis result by the model or a setting by a user. Including,
Structure deterioration detection system. In the structure deterioration detection system according to claim 5,
The weak point emphasis data includes a plant image or a line group image,
Structure deterioration detection system. In the structure deterioration detection system according to claim 1,
The first process includes a rotation process,
The rotation processing generates a plurality of images by rotating the first image patch, inputs each of the plurality of images to the model, and estimates deterioration for each pixel from each diagnosis result image Including a process of extracting a portion having the largest probability value and integrating it into one diagnostic result image,
Structure deterioration detection system. In the structure deterioration detection system according to claim 7,
The rotation process includes a process of generating the plurality of images by rotating the first image patch in an equal division within a 180 degree range at a predetermined rotation angle θ.
Structure deterioration detection system. In the structure deterioration detection system according to claim 1,
The computer system corrects the size of the plurality of images in the first image related to the target structure so that the resolution is constant, and calculates the first image based on the size of the plurality of images with the constant resolution. Determine and set the input size,
Structure deterioration detection system. In the structure deterioration detection system according to claim 1,
The second process includes
Processing for displaying, on the screen, a multi-tone image in which deterioration estimation probability values are expressed in multi-tone for each pixel of the first diagnosis result image or the second diagnosis result image as the second image;
A first part for variably setting a first threshold value for binarization related to the estimated deterioration probability value is displayed on the screen, and the first threshold value is set based on an operation of the first part by the user. The process to change,
A process of generating a binarized image obtained by binarizing the deterioration estimation probability value of the second image according to the changed first threshold, and displaying the binarized image on the screen;
Structure deterioration detection system including In the structure deterioration detection system according to claim 10,
The second process includes
The screen displays a second component for variably setting a second threshold related to the size of the degradation region adjacent to the pixel estimated to be degraded, and the second component is displayed based on the user's operation of the second component. 2 processing to change the threshold value;
A process of generating an image in which a degraded area having a small size is removed from the degraded areas in the binarized image of the second image according to the changed second threshold, and displaying the image on the screen;
Structure deterioration detection system including In the structure deterioration detection system according to claim 10,
The second process includes
The screen displays a third part related to the removal of the linearly shaped area among the degraded areas adjacent to the pixels estimated to be degraded, and removes the linearly shaped area based on the user's operation of the third part. Processing to accept,
A process of generating an image obtained by extracting and removing the linear shape region from the degraded region in the binarized image of the second image in response to acceptance of the removal of the linear shape region, and displaying the image on the screen When,
Structure deterioration detection system including