JP2021135903A - Defective product image generation program and quality determination device - Google Patents

Abstract

To generate a defective product image that is closer to a real article from a non-defective product image and a partial image including a defective portion.SOLUTION: A defective product image generation program causes a computer to execute: first acquisition processing of acquiring a non-defective product image of an industrial product and a composite image obtained by combining a partial image including a defective portion with the non-defective product image; and first image generation processing of generating a defective product image by repeatedly executing an image updating model on the basis of the input image, the non-defective product image and the composite image. The image updating model inputs the input image (the composite image in the first time, the output image that is output in the previous time from the image updating model in the second or subsequent time) and the non-defective product image to a convolution neural network that has been learned in advance, inputs the input image and the composite image to an edge extraction filter, and updates and outputs the input image so as to minimize the totalized loss obtained by totalizing the first loss function obtained with the convolution neural network and the second loss function obtained with the edge extraction filter.SELECTED DRAWING: Figure 7

Description

The present invention relates to a defective image generation program for industrial products and a quality determination device.

There is known a quality determination technique for determining the quality of an industrial product based on the presence or absence of defective parts in the captured image of the industrial product using an image recognition model that recognizes the image by machine learning such as deep learning. In the quality judgment technology using such an image recognition model, in order to improve the quality judgment accuracy of an industrial product, it is necessary to input a large amount of good and defective images of the industrial product into the image recognition model for learning. is necessary. However, in the manufacturing process of industrial products, the probability of occurrence of defective products is extremely low compared to the probability of occurrence of non-defective products, so it is practically difficult to collect a sufficient amount of images of defective products of industrial products for machine learning. ..

Therefore, conventionally, a technique has been known in which a sample image of one industrial product is artificially edited to generate a plurality of defective image images (see, for example, Patent Document 1). Specifically, a plurality of defective pattern images including defective portions are acquired in advance, and an editing process for synthesizing an arbitrary defective pattern image with a sample image is performed by operating an input device by a person. It is also possible to edit the position, rotation angle, etc. of the defective pattern image in the sample image by operating the input device by a person.

Japanese Unexamined Patent Publication No. 5-149728

In the conventional technique of generating a defective product image by the above-mentioned artificial editing, there is a problem that an unnatural defective product image is generated because the defective product image is generated by artificially editing.

An object of the present invention is to provide a defective product image generation program capable of solving the above-mentioned problems, and a quality determination device for determining quality using a defective product image generated by the defective product image generation program. And.

In order to achieve the above object, the defective product image generation program of the present invention is a defective product image generation program that generates a defective product image of an industrial product determined to be a defective product due to the presence of defective parts, and is a computer. In addition, a first acquisition process for acquiring a non-defective product image of the industrial product and a composite image in which a partial image including the defective portion is combined with the non-defective product image, and an input image, the non-defective product image, and the composite image Based on this, the image update model is repeatedly executed to execute the first image generation process for generating the defective product image, and the image update model is the input image (however, the first time is the composite image, the second and subsequent times). Is the output image previously output from the image update model) and the non-defective image are input to the pre-trained convolutional neural network, and the input image and the composite image are input to the edge extraction filter. , The input image is updated and output so that the total loss obtained by adding the first loss function obtained by the convolutional neural network and the second loss function obtained by the edge extraction filter is minimized. It is a model.

In this defective product image generation program, an image update model is used. The image update model is a model that updates and outputs an input image so that the total loss of the sum of the first loss function and the second loss function is minimized. Here, the first loss function is the error between the input image obtained by the convolutional neural network and the non-defective image, and updating the input image so that the first loss function is minimized is a feature of the non-defective image. It means that the input image is updated so as to approach. The second loss function is the error between the input image obtained by the edge extraction filter and the composite image, and updating the input image so that the second loss function is minimized is a feature (defectiveness) of the composite image. It means that the input image is updated so as to approach the place). Therefore, updating the input image so that the total loss is minimized means that the input image is updated so as to approach the characteristics of the non-defective image while maintaining the characteristics of the defective portion of the composite image. As a result, according to the defective product image generation program, it is possible to generate a defective product image closer to the real thing from the non-defective product image and the partial image.

In the defective product image generation program, the image update model further inputs the input image into a defect determination model for determining the presence or absence of a defective portion in the image, and the first loss function and the second loss. Further, the third loss function obtained by the defect determination model is added to the function to obtain the total loss. The third loss function is an error between the judgment criterion with a defect and the judgment result obtained by the defect judgment model, and updating the input image so that the third loss function is minimized is a defect judgment model. This means that the input image is updated so as to approach the image determined to have a defective part. As a result, according to the defective product image generation program, it is possible to generate a defective product image closer to the real thing as compared with the case where the defective product determination model is not used.

In the defective product image generation program, a second acquisition process for acquiring a non-defective product image of the industrial product and a partial image including the defective portion of the industrial product on the computer, and a non-defective product image of the industrial product. A designation process for designating a first image area that is a part of the image, and a combination of the partial image at an arbitrary position in the first image area, targeting the first image area of the non-defective image. The second image generation process for generating the composite image may be executed. That is, the partial image may be combined with the first image region designated in advance among the non-defective images. As a result, according to the defective product image generation program, the generation of a composite image in which the defective portion is arranged at an unnatural position is suppressed, and a defective product image closer to the real thing can be generated.

In the defective product image generation program, the industrial product may be a gear, and the first image region may be configured to be an edge portion of a tooth in the gear and a peripheral portion of the edge portion. That is, a composite image in which defective portions are present in particular at the edge portion and its peripheral portion (hereinafter, referred to as “edge portion or the like”) may be generated. Defective parts existing on the edge portion and the like are likely to affect the function of the gears, such as making an abnormal noise due to contact with other gears that mesh with each other. Therefore, in the present defective product image generation program, a more useful defective product image can be generated with high accuracy by setting the first image region as the edge portion of the tooth or the like.

In the defective product image generation program, the industrial product may be a gear, and the first image region may be a region different from the tooth bottom of the gear. That is, a composite image in which the defective portion exists only in the tooth bottom may not be generated. The tooth bottom of the gear does not come into contact with other gears that mesh with each other, and even if there is a defective portion on the tooth bottom, there is a high possibility that the function of the gear will not be affected. Therefore, in the present defective product image generation program, a more useful defective product image can be generated with high accuracy by setting the first image region to a region different from the tooth bottom of the gear.

In the defective product image generation program, in the designation process, the first image region may be designated based on the position of the edge portion extracted by the edge extraction process with respect to the non-defective product image. As a result, according to the defective product image generation program, the first image area can be automatically specified based on the position of the edge portion extracted from the non-defective product image.

In the defective image generation program, in the second image generation process, the saturation of the image obtained by synthesizing the partial image with the non-defective image is corrected based on the saturation of the non-defective image before synthesizing the partial image. Then, the composite image may be generated. As a result, according to the defective product image generation program, a composite image reflecting the saturation of the non-defective product image is generated even for the partial image, so that a defective product image closer to the real thing can be generated. ..

The quality determination device of the present invention is a quality determination device that determines the quality of the judgment target product based on the presence or absence of defective parts in the captured image of the judgment target product, and is generated by the defective product image generation program described above. The quality of the judgment target product is judged by inputting the captured image of the judgment target product into the image recognition model trained by using the defective product image as the teacher data. According to the quality determination device, it is possible to accurately determine whether or not the product to be determined is a defective product by using an image recognition model learned based on a defective product image that is closer to the real thing.

The defective image generation program of the present invention can also be realized in other forms such as a defective image generator, a defective image generation method, and a non-temporary recording medium on which the defective image generation program is recorded. Is.

Explanatory drawing which shows the schematic structure of the non-defective product inspection system 10 in this embodiment. Flowchart showing the contents of the baseline image generation process Flowchart showing the contents of mask image generation processing Explanatory drawing which shows the change of the image of the bevel gear W in the baseline image generation processing. Flowchart showing the contents of the saturation adjustment process Flowchart showing the contents of image optimization processing Explanatory drawing which shows the outline of image update model 141 Explanatory drawing which shows evaluation result of each image by FID score

A. Embodiment:
A-1. Configuration of non-defective product inspection system 10:
FIG. 1 is an explanatory diagram showing a schematic configuration of a non-defective product inspection system 10 according to the present embodiment. The non-defective product inspection system 10 determines whether an industrial product is a non-defective product or a defective product by performing a visual inspection on industrial products (which may be industrial parts) that are sequentially manufactured, for example, in a quality inspection process on a production line. It is a system for. In the following description, it is assumed that the industrial product is a bevel gear W used in, for example, a differential gear device. Further, a non-defective product of the bevel gear W means a bevel gear W in which there is no defective portion in a specific region affecting the function of the bevel gear W, and a defective product of the bevel gear W is a defective product of the bevel gear W in the specific region of the bevel gear W. It means that there is a defective part. The specific region is, for example, an edge portion of a tooth (particularly a tooth tip ridge line and a tooth profile ridge line) and a peripheral portion thereof (tooth tip surface and tooth surface). A defective part is a damage or a dent in an industrial product. Hereinafter, "dents" will be described as an example.

As shown in FIG. 1, the non-defective product inspection system 10 includes an imaging unit 12 (for example, a digital camera), a rotating stage 14, a transfer robot 16, an image processing device 100, and a quality determination device 200.

The rotary stage 14 has a configuration in which a table on which the bevel gear W is placed is rotatably supported. The imaging unit 12 is arranged at a position where the bevel gear W placed on the table of the rotating stage 14 can be imaged from the side, for example. The transfer robot 16 grips the bevel gears W sequentially manufactured on the production line one by one and places them on the table of the rotating stage 14, for example, a bevel gear W determined to be a non-defective product and a bevel gear W determined to be a defective product. And to different locations. The image pickup unit 12, the rotation stage 14, and the transfer robot 16 are communicably connected to the control unit 210 of the quality determination device 200 described below via the interface unit 250, and the image pickup unit 12, the rotation stage 14, and the rotation stage 14 are connected to each other. Each operation with the transfer robot 16 is controlled by the control unit 210. Further, the image processing device 100 and the quality determination device 200 are connected to each other so as to be able to communicate with each other. The image processing device 100 and the quality determination device 200 do not need to be provided on the same production line as the image pickup unit 12, the rotation stage 14, and the transfer robot 16, and are not required to be provided in a predetermined network (for example, a public line such as the Internet or a dedicated line). Etc.) may be provided in a cloud environment where communication is possible.

A-1-1. Configuration of Image Processing Device 100 The image processing device 100 includes a control unit 110, a storage unit 120, a display unit 130, an operation input unit 140, and an interface unit 150. Each of these parts is communicably connected to each other via a bus 190.

The display unit 130 is composed of, for example, a liquid crystal display or the like, and displays various images and information. The operation input unit 140 is composed of, for example, a keyboard, a mouse, a button, a microphone, or the like, and receives human operations and instructions. In the present embodiment, the display unit 130 is provided with a touch panel to function as an operation input unit 140. Further, the interface unit 150 is composed of, for example, a LAN interface, a USB interface, or the like, and communicates with another device by wire or wirelessly.

The storage unit 120 is composed of, for example, a ROM, a RAM, a hard disk drive (HDD), or the like, and stores various data, programs, or models, or temporarily executes a work area or data when executing various programs or models. It is used as a storage area.

Specifically, the storage unit 120 stores a dent image DB 121, a non-defective image DB 122, a defective image DB 123, and a baseline image DB 124. A plurality of patterns of dent images K (see FIG. 4) are stored in the dent image DB 121. The dent image K is a partial image including dents in the captured image of the defective product of the actual bevel gear W. In the present embodiment, it is a divided image obtained by extracting a specific region of the bevel gear W from the captured image of the defective product. In the present embodiment, a plurality of patterns (for example, about 30) of dent images K are acquired based on captured images of defective products of the plurality of bevel gears W. In the present embodiment, the dent image K is assumed to be a grayscale image. The non-defective product image DB 122 stores a pre-captured captured image of a plurality of patterns of the non-defective product of the bevel gear W (hereinafter, referred to as “non-defective product image GY”). In the defective product image DB 123, an image of a defective product of a plurality of patterns of the bevel gear W imaged in advance and an image of a defective product of the bevel gear W generated by executing the defective product image generation PGM 131 described below (hereinafter, “defective product”). Image GX ") is sequentially stored. The dent image K is an example of a partial image within the scope of claims. The baseline image DB 124 sequentially stores the baseline image GB generated in the execution process of the defective image generation PGM131.

In addition, the defective product image generation PGM 131 is stored in the storage unit 120. The defective image generation PGM131 is a computer program for executing a process for generating a defective image GX. The defective image generation PGM 131 includes a baseline image generation PGM 133 and an image optimization PGM 134. The baseline image generation PGM 133 is a computer program for executing the baseline image generation process (see FIG. 2) described later, and the image optimization PGM 134 is for executing the image optimization process (see FIG. 6) described later. Computer program. These programs are provided in a state of being stored in a computer-readable recording medium (not shown) such as a CD-ROM, a DVD-ROM, or a USB memory, and can be installed in the image processing apparatus 100 to store the programs. It is stored in 120. The storage unit 120 also stores an image update model 141, which will be described later.

The control unit 110 is composed of, for example, a CPU or the like, and controls the operation of the image processing device 100 by executing a computer program read from the storage unit 120. For example, the control unit 110 reads a defective image generation PGM 131 from the storage unit 120 and executes it to execute a process (baseline image generation process, image optimization process) for generating a defective image GX, which will be described later. do. The control unit 110 functions as a second acquisition unit 113, a designation unit 114, and a second image generation unit 115 when executing the baseline image generation processing, and the first acquisition unit 110 when executing the image optimization processing. It functions as a unit 111 and a first image generation unit 112. The functions of each of these parts will be described in accordance with the description of various processes described later.

A-1-2. Configuration of Pass / Fail Judgment Device 200 The pass / fail determination device 200 includes a control unit 210, a storage unit 220, a display unit 230, an operation input unit 240, and an interface unit 250. Each of these parts is communicably connected to each other via bus 290.

The display unit 230 is composed of, for example, a liquid crystal display or the like, and displays various images and information. The operation input unit 240 is composed of, for example, a keyboard, a mouse, a button, a microphone, or the like, and receives human operations and instructions. In the present embodiment, the display unit 230 functions as the operation input unit 240 by providing the touch panel. Further, the interface unit 250 is composed of, for example, a LAN interface, a USB interface, or the like, and communicates with other devices by wire or wirelessly.

The storage unit 220 is composed of, for example, a ROM, a RAM, a hard disk drive (HDD), or the like, and stores various data, programs, or models, or temporarily executes a work area or data when executing various programs or models. It is used as a storage area.

Specifically, the storage unit 220 stores the pass / fail determination PGM232. The quality determination PGM232 is a program for determining the quality of the bevel gear W based on the presence or absence of defective parts in the captured image of the bevel gear W. This program is provided in a state of being stored in a recording medium (not shown) readable by a computer such as a CD-ROM, a DVD-ROM, or a USB memory, and is installed in the quality determination device 200 to store the storage unit 220. Stored in. The storage unit 220 also stores an image recognition model 242, which will be described later.

The control unit 210 is composed of, for example, a CPU or the like, and controls the operation of the quality determination device 200 by executing a computer program read from the storage unit 220. For example, the control unit 210 reads the pass / fail determination PGM232 from the storage unit 220 and executes it to execute the pass / fail determination process described later. At this time, the control unit 210 functions as a learning unit 226 and a determination unit 227. The functions of each of these parts will be described in accordance with the description of the processing described later.

A-2. Processing to generate defective image GX:
Next, various processes for generating the defective image GX executed by the image processing apparatus 100 of the present embodiment will be described.

A2-1. Baseline image generation processing FIG. 2 is a flowchart showing the contents of the baseline image generation processing, FIG. 3 is a flowchart showing the contents of the mask image generation processing, and FIG. 4 is a bevel gear W in the baseline image generation processing. It is explanatory drawing which shows the change of the image of. FIG. 5 is a flowchart showing the contents of the saturation adjustment process. The baseline image generation process is a process of synthesizing the dent image K stored in the dent image DB 121 with the non-defective image GY stored in the non-defective image DB 122 to generate the baseline image GB.

When the baseline image generation process is started, as shown in FIG. 2, the second acquisition unit 113 randomly selects one of the plurality of non-defective image GYs stored in the non-defective image DB 122 by a uniform random number. Good product image GY (see FIG. 4A) is selected and acquired (S110). The process of S110 is an example of a second acquisition process within the scope of claims. The next processing after S112 may be performed on the non-defective image GY, or may be performed on the processed non-defective image GY in which a part of the non-defective image GY is trimmed. In the present embodiment, as shown in FIG. 4A, the processing after S112 is performed on the image in which the peripheral portion of the non-defective image GY where no dents cannot exist is trimmed. Hereinafter, for the sake of brevity of description, the good product image GY after trimming is also simply referred to as a good product image GY. Further, the non-defective image GY is assumed to be image data in RGB space.

Next, the designation unit 114 performs a mask image generation process on the acquired non-defective image GY (S112). In the mask image generation process, when the baseline image GB is generated, the dented image K is arranged in an image area (hereinafter, referred to as “mask image area”) in which dents cannot actually occur in the non-defective image GY. This is a process for avoiding the dents and arranging the dents in the specific image area R1 where the dents may actually occur. Specifically, the mask image generation process is a process for designating a specific image region R1 in the non-defective image GY. The processing of S112 is an example of the designation processing in the claims, the specific image area R1 is the area corresponding to the above-mentioned specific area, and is an example of the first image area in the claims.

When the mask image generation process is started, as shown in FIG. 3, the designation unit 114 applies grayscale conversion to the non-defective image GY to generate a grayscale image GY1 (see FIG. 4B). (S210). Next, the designation unit 114 applies a differential filter (for example, a first-order differential filter) to the grayscale image GY1 to extract the edge portion (contour) of the bevel gear W (see FIG. 4C). ) Is generated (S212). Next, the designation unit 114 applies a median filter (for example, window 5 × 5) to the edge image GY2 to generate an edge image GY3 (see FIG. 4D) with reduced noise (S214). A binarization process (for example, Otsu's binarization process) is applied to the edge image GY3 to generate a binarized image GY4 (see FIG. 4E) (S216). In the binarized image GY4, the edge portion and the peripheral portion of the bevel gear W are mainly white regions, and a part of the tooth bottom is also a white region.

In the present embodiment, the designation unit 114 further applies a shrinkage processing and smoothing filter (for example, a median filter) of a window (for example, 3 × 3) smaller than that at the time of processing of S214 to the binarized image GY4. Alternatively, it is applied a plurality of times to generate a mask image GM (see FIG. 4 (g)) (S218). By applying the median filter, the binarized image GY4 is smoothed and noise is removed. By applying the shrinkage treatment, noise in an isolated region (for example, the tooth bottom of the bevel gear W) away from the edge portion is removed in the binarized image GY4. In FIG. 4, the binarized image GY4 is subjected to the shrinkage treatment and the median filter once to generate the binarized image GY5 (see FIG. 4 (f)), and further, the binarized image GY5 is generated. An example is shown in which the shrinkage treatment and the median filter are applied once to obtain a mask image GM (see FIG. 4 (g)). In this way, by applying the shrinkage treatment and the median filter to the binarized image GY4 a plurality of times, noise can be removed more effectively. The “noise” referred to here is, in particular, a white region remaining on the tooth bottom of the bevel gear W. As shown in FIGS. 4 (e) to 4 (g), it can be seen that the white region remaining on the tooth bottom of the bevel gear W is reduced according to the number of times the shrinkage treatment and the median filter are applied. As will be described later, the white region is a region for designating the specific image region R1 in the non-defective image GY.

When the mask image generation process is completed, the second image generation unit 115 targets the specific image area R1 of the non-defective image GY, and synthesizes the dent image K at an arbitrary position in the specific image area R1. Generate a baseline image GB. This process is an example of a second image generation process in the claims, and the baseline image GB is an example of a composite image in the claims.

Specifically, as shown in FIG. 2, the second acquisition unit 113 is a uniform random number from a plurality of patterns of dent images K (see FIG. 4 (l)) stored in the dent image DB 121. Randomly select and acquire one or a plurality of patterns of dent images K (S114). Next, the second image generation unit 115 overwrites the dent image K at a position randomly selected by a uniform random number in the specific image area R1 (the area corresponding to the white area of the mask image GM) of the non-defective image GY. Then, an image GD with dents (see FIG. 4 (h)) is generated (S116).

Next, the second image generation unit 115 executes the saturation adjustment process on the dented image GD (S118). The saturation adjustment process is a process for reflecting the saturation of the non-defective image GY on the dented image K in the dented image GD. When the saturation adjustment process is started, as shown in FIG. 5, the second image generation unit 115 sets each of the dented image GD and the non-defective image GY before the dent image K is synthesized. , The image in the RGB space is converted into an image in a color space in which the saturation can be adjusted (for example, HSV space, HLS space, L * a * b space, YCbCr space, YUV space, etc.) (S310). Next, the second image generation unit 115 corrects the saturation of the dented image GD after the color conversion based on the saturation of the non-defective image GY after the color conversion (S320), and strikes after the saturation correction. The marked image GD is converted into an image in RGB space, a baseline image GB is generated (S330), the image is stored in the baseline image DB 124 of the storage unit 120, and the baseline image generation process is completed. As a result, the baseline image GB (see FIG. 4 (i)) in which the saturation of the non-defective image GY is reflected is also generated for the composite portion of the dented image K in the dented image GD.

A-2-2. Image optimization processing FIG. 6 is a flowchart showing the contents of the image optimization processing, and FIG. 7 is an explanatory diagram showing an outline of the image update model 141. The image optimization process is a process for bringing the baseline image GB generated by the baseline image generation process closer to a defective image that is closer to the real thing (more natural).

When the image optimization process is started, as shown in FIG. 6, the first acquisition unit 111 acquires the baseline image GB and the non-defective image GY that is the basis of the baseline image GB from the storage unit 120. (S410). The processing of S410 is an example of the first acquisition processing within the scope of claims. Next, the first image generation unit 112 repeatedly executes the image update model 141 based on the input image GS, the baseline image GB, and the non-defective image GY to generate the defective image GX (S420). The process of S420 is an example of the first image generation process within the scope of claims.

Specifically, as shown in FIG. 7, in the image update model 141, the input image GS (however, the first time is the baseline image GB, the second and subsequent times is the output image previously output from the image update model 141), and a good product. The image GY and the image GY are input to the pre-trained convolutional neural network M11 to obtain the content loss L1 (S510). The convolutional neural network M11 is a learning model that classifies images based on the feature amount of the image, and is, for example, a VGG19 model. The convolutional neural network M11 is not limited to this, and may be any learning model trained by ImageNet, and may be, for example, a VGG16 model, AlexNet, ResNet34, ResNet50, InceptionV3, or the like.
The content loss L1 is a loss function obtained by using the squared error of the feature amount between the input image GS and the non-defective image GY output from the image update model 141 as a loss function, and is represented by, for example, the following equation 1.
<Equation 1>
Content loss _{L1 = (1/2) (F i} (GY) -F i (GS)) 2
F _i: a ImageNet, which is the output of the i-th ConvLayer of VGG19 model.
In the content loss L1, the non-defective image GY is set as the true value. Therefore, minimizing the content loss L1 means that the input image GS is updated so as to approach the characteristics of the non-defective image GY. In addition, "the input image GS is updated" means that the pixel value of the input image GS is updated (hereinafter, the same applies). The content loss L1 is an example of the first loss function in the claims.

Further, in the image update model, the input image GS and the baseline image GB are input to the Laplacian filter M12 to obtain the Laplacian loss L2 (S520). The Laplacian loss L2 uses the squared error between the input image GS output from the Laplacian filter M12 and the baseline image GB as a loss function, and is represented by, for example, the following equation 2.
<Equation 2>
Laplacian loss L2 = (1/2) (L (GB) -L (GS)) ²
L: This is the output of the Laplacian filter M12.
The Laplacian filter M12 is a filter that extracts edges of an image, and the Laplacian loss L2 has the baseline image GB as a true value. Therefore, minimizing the Laplacian loss L2 means that the input image GS is updated so as to approach the characteristics of the baseline image GB (particularly the contour of the dent image K). The Laplacian loss L2 is an example of a second loss function in the claims.

Further, in the image update model 141, the input image GS is input to the dent classification model M13 to obtain a dent determination loss L3 (S530). The dent classification model M13 uses a pre-learned convolutional neural network as a classifier, and by performing transfer learning on a plurality of divided images of the bevel gear W described above, the presence or absence of dents in the image It is a model for discriminating, for example, ResNet50. The dent classification model M13 is not limited to this, and may be a learning model trained by ImageNet, and may be, for example, a VGG16 model, a VGG19 model, AlexNet, ResNet34, InceptionV3, or the like. The dent determination loss L3 is represented by the following equation 3.
<Equation 3>
Dust determination loss L3 = (1/2) (1-cnn (GS)) ²
cnn: A dent classification model for determining the presence or absence of dents, which outputs "0" for an image without dents and outputs "1" for an image with dents.
In the dent determination loss L3, "1" is set as the true value. Therefore, minimizing the dent determination loss L3 means that the input image GS is updated in the direction in which the dent classification model determines that there is a dent. The dent classification model M13 is an example of a defect determination model in the claims, and the dent determination loss L3 is an example of a third loss function in the claims.

Next, in the image update model 141, the total loss (= α ・ L1 + β ・ L2 + γ ・ L3) obtained by adding the obtained content loss L1, the Laplacian loss L2, and the dent determination loss L3 is calculated (S540), and the total loss is calculated (S540). The input image GS is updated so that is minimized, and the output image is output (S550). The constant term (α, β, γ) of the total loss can be changed as appropriate.

Next, the first image generation unit 112 determines whether or not the output image satisfies a predetermined optimum condition (S430). The optimum condition is, for example, that the number of epochs has reached a predetermined value (for example, 20). The optimum condition is not limited to this, and may be, for example, that the total loss is equal to or less than a predetermined value. When the first image generation unit 112 determines that the output image does not satisfy the optimum conditions (S430: NO), sets the output image as the input image GS (S440), returns to S420, and re-image update model 141. Repeat the execution of. On the other hand, when the first image generation unit 112 determines that the output image satisfies the optimum condition (S430: YES), it is assumed that the defective image GX with high accuracy is generated, and the latest output image is the defective image GX. The image is stored in the defective image DB 123 of the storage unit 120 (S450), and the image optimization process is completed.

A-2-3. Evaluation Results In the present embodiment, each of the baseline image GB and the defective image GX was evaluated by comparison with a real dent image (for example, 1000 images) using a FID (Frechet Information Distance) score. .. The FID score is a measure for evaluating the distance between the mean and variance of the feature expression obtained by inputting an image into an ImageNet trained Inception V3 model for two image data sets. The FID score is defined below by Equation 4.
<Equation 4>
_{^{d 2 = | m 1 -m 2}} | 2 + Tr (C 1 + C 2 -2 (C 1 C 2) 1/2)
m ₁ : Output of the last average pooling layer of the model when the image data set 1 is input to the InputV3 model.
m ₂ : Output of the last average pooling layer of the model when the image data set 2 is input to the InputV3 model.
Tr: Matrix trace.
C ₁ : A covariance matrix for the output 2048-dimensional vector of the InputV3 model with the image data set 1 as the input.
C ₂ : A covariance matrix for the output 2048-dimensional vector of the InputV3 model with the image data set 2 as the input.

FIG. 8 is an explanatory diagram showing the evaluation results of each image based on the FID score. The defective image GX in FIG. 8 (without the dent determination model) does not add the dent determination loss L3 to the total loss in S420 of the image optimization process (FIG. 6), and the content loss L1 and the Laplacian loss L2. It means the defective image GX generated by summing only with, and the defective image GX (with a dent judgment model) is the sum of the content loss L1, the Laplacian loss L2, and the dent judgment loss L3. It means the defective product image GX generated in the above. As shown in FIG. 8, the defective product image GX (without the dent determination model) has an evaluation result closer to the real dent image than the baseline image GB, and the defective product image GX (dent determination). In the case of (with model), the evaluation result was closer to that of the real dent image as compared with the defective image GX (without the dent determination model).

A-3. Process for determining the quality of the bevel gear W:
Next, a process for determining the quality of the bevel gear W executed by the quality determination device 200 of the present embodiment will be described. This process is a process for determining the quality of the bevel gear W based on the presence or absence of defective portions in the captured image of the bevel gear W by using the image recognition model 242 that recognizes the image by machine learning.

When the defective product image GX in a sufficient amount for machine learning is stored in the defective product image DB 123 of the storage unit 120 by the process for generating the defective product image GX described above, the learning unit 226 of the quality determination device 200 receives the defective product image GX. Using the plurality of patterns of non-defective image GY stored in the non-defective image DB 122 of the image processing device 100 and the plurality of patterns of defective image GX stored in the defective image DB 123 as teacher data, the quality of the bevel gear W is determined. Machine learning is performed to generate or update the image recognition model 242. The image recognition model 242 can be generated by controlling the weighting coefficient of the model formula prepared in advance, but the specific model formula used to generate the image recognition model 242 is not limited, and any model formula can be used. Applicable.

When the quality determination process is started, the determination unit 227 rotates the table of the rotation stage 14 and causes the image pickup unit 12 to take an image of the entire circumference of the bevel gear W placed on the table. As a result, a plurality of captured images in which different angles of one bevel gear W are captured are generated. Specifically, for example, 36 captured images are generated for one bevel gear W, and each captured image is divided into, for example, eight to generate a total of 36 × 8 = 288 determination target data. Next, the determination unit 227 determines the quality (presence / absence of defective portion) using the image recognition model 242 for each determination target data. That is, for one bevel gear W, quality determination is performed for 288 types of determination target data. The number of determination target data per bevel gear W is arbitrary.

A-4. Effect of this embodiment:
As described above, in the defective image generation PGM131 of the present embodiment, the image update model 141 is used in the image optimization process (S420 in FIG. 6). The image update model 141 is a model that updates and outputs the input image GS so that the total loss of the content loss L1 and the Laplacian loss L2 is minimized (see FIG. 7). Here, the content loss L1 is an error between the input image GS obtained by the convolutional neural network M11 and the non-defective image GY, and updating the input image GS so that the content loss L1 is minimized is the non-defective image GY. It means that the input image GS is updated so as to approach the feature. The Laplacian loss L2 is an error between the input image GS obtained by the edge extraction filter (Laplacian filter M12) and the baseline image GB, and updating the input image GS so that the Laplacian loss L2 is minimized is the base. It means that the input image GS is updated so as to approach the feature (defective part) of the line image GB. Therefore, updating the input image GS so as to minimize the total loss means that the input image GS is updated so as to approach the characteristics of the non-defective image GY while maintaining the characteristics of the defective portion of the baseline image GB. means. As a result, according to the present embodiment, it is possible to generate a defective image GX that is closer to the real thing from the non-defective image GY and the dent image K.

In the present embodiment, in the image optimization process, the input image GS is input to the dent classification model M13 so that the total loss is minimized and the dent classification model M13 determines that there is a defective portion. The image GS is updated (see FIG. 7). As a result, according to the present embodiment, it is possible to generate a defective image GX that is closer to the real thing than when the dent classification model M13 is not used.

In the present embodiment, in the baseline image generation process, the dent image K is synthesized with respect to the specific image region R1 designated in advance among the non-defective image GY (S116 in FIG. 2). As a result, the generation of the baseline image GB in which the defective portion is arranged at an unnatural position is suppressed, and the defective product image GX closer to the real thing can be generated. Here, a method is conceivable in which a non-defective product image is used as a content image and an image including a defective portion is used as a style image to perform style conversion to generate a defective product image by using NST (Neural Style Transfer) technology. However, in the method using this NST technique, the position of the defective portion is converted according to the overall style of painting, so that the position of the defective portion in the defective product image cannot be controlled. As a result, there is a high possibility that an unnatural defective product image is generated, for example, the defective portion is arranged at a position where it cannot exist. On the other hand, in the present embodiment, the position of the defective portion in the defective product image GX can be controlled by designating the specific image area R1 in advance. Therefore, it is possible to generate a defective image GX that is closer to the real thing.

Further, in the present embodiment, a baseline image GB in which a defective portion exists particularly in an edge portion and a peripheral portion thereof (hereinafter, referred to as “edge portion or the like”) is generated. Defective parts existing on the edge portion and the like are likely to affect the function of the gears, such as making an abnormal noise due to contact with other gears that mesh with each other. Therefore, in the present embodiment, by setting the specific image region R1 as the edge portion of the tooth or the like, a more useful defective image GX can be generated with high accuracy.

Further, in the present embodiment, it is difficult to generate the baseline image GB in which the defective portion exists only in the tooth bottom. The tooth bottom of the gear does not come into contact with other gears that mesh with each other, and even if there is a defective portion on the tooth bottom, there is a high possibility that the function of the gear will not be affected. Therefore, in the present embodiment, by setting the specific image region R1 to a region different from the tooth bottom of the gear, a more useful defective image GX can be generated with high accuracy.

In the present embodiment, in the mask image generation process (see FIGS. 2 and 3), the specific image area R1 is designated based on the position of the edge portion extracted by the edge extraction process (S212 in FIG. 3) for the non-defective image GY. .. Thereby, the specific image area R1 can be automatically specified based on the position of the edge portion extracted from the non-defective image GY.

In the present embodiment, in the saturation adjustment process (see FIG. 5), the saturation of the image obtained by synthesizing the dent image K with the good image GY is based on the saturation of the good image GY before synthesizing the dent image K. The baseline image GB is generated after correction. As a result, the baseline image GB that reflects the saturation of the non-defective image GY is generated for the dent image K as well, so that the defective image GX that is closer to the real thing can be generated.

In the quality determination device 200 of the present embodiment, the quality is determined by inputting the captured image of the product to be determined (bevel gear W) into the image recognition model 242 trained by using the above-mentioned defective product image GX as teacher data. .. As a result, by using the image recognition model 242 learned based on the defective product image GX that is closer to the real thing, it is possible to accurately determine whether or not the product to be determined is a defective product.

B. Modification example:
The present invention is not limited to the above-described embodiment, and can be transformed into various forms without departing from the gist thereof. For example, the following modifications are also possible.

The configuration of the non-defective product inspection system 10 in the above embodiment is merely an example and can be variously modified. For example, at least a part of the storage unit 120 of the image processing device 100 and at least a part of the storage unit 220 of the quality determination device 200 may be stored in an external device (for example, a cloud server). Further, the operations of the imaging unit 12, the rotating stage 14, and the transfer robot 16 may be controlled not by the quality determination device 200 but by a dedicated control device (programmable logic controller or the like).

In the above embodiment, the bevel gear W is exemplified as an industrial product, but other gears (for example, a helical gear or the like) may be used, or an industrial product other than the gear (such as a shaft or a screw) may be used. Industrial products are, for example, forged products obtained by pressing a metal material or the like with a mold or castings formed by pouring a metal material or the like into a mold, and the shape, size, material, etc. are arbitrary, and not only metal but also metal. , Resin or the like may be used.

The contents of various processes in the above embodiment are merely examples and can be variously modified. For example, in the baseline image generation process, the dent image K stored in the dent image DB 121 may be image data in the same color space (RGB color space) as the non-defective image GY. Further, the non-defective image GY may be image data in a color space other than the RGB space. Further, in S110 of the baseline image generation process, the non-defective image GYs selected in a predetermined order may be acquired, or one non-defective image GY may always be acquired. Further, in S114, the dent images K selected in a predetermined order may be acquired, or one dent image K may always be acquired.

In the baseline image generation process, at least one of the mask image generation process (S112) and the saturation adjustment process (S118) may not be executed, or an external device may execute the result and the image processing device 100 receives the result. May be good. In the baseline image generation process, the specific image area R1 may be designated by a human input operation. In the mask image generation process, at least one of the processes S214 to S218 may not be executed.

In the image optimization process, at least a part of the execution process by the image update model 141 may be executed by an external device, and the pass / fail determination device 200 may receive the result.

In the process for determining the quality of the bevel gear W, the quality determination device 200 may receive the result by causing an external device to execute at least a part of the process of generating or updating the image recognition model 242.

In the above embodiment, a part of the configuration realized by the hardware may be replaced with software, and conversely, a part of the configuration realized by the software may be replaced with hardware.

100: Image processing device 110, 210: Control unit (computer) 111: First acquisition unit 112: First image generation unit 113: Second acquisition unit 114: Designation unit 115: Second image generation unit 131: Defective product image generation program 141: Image update model 200: Good / bad product image 227: Judgment unit 242: Image recognition model GB: Baseline image (composite image) GS: Input image GX: Defective product image GY: Good product image K: Imprint Image (partial image) L1: Content loss (first loss function) L2: Laplacian loss (second loss function) M11: Convolutional neural network M12: Laplacian filter (edge extraction filter) M13: Dimple classification model (defective) Judgment model) R1: Specific image area (first image area) W: Bevel gear (industrial product, judgment target product)

Claims (8)

It is a defective product image generation program that generates a defective product image of an industrial product that is determined to be a defective product due to the presence of defective parts.
On the computer
A first acquisition process for acquiring a non-defective image of the industrial product and a composite image obtained by synthesizing a partial image including the defective portion with the non-defective image.
Based on the input image, the non-defective product image, and the composite image, the image update model is repeatedly executed to execute the first image generation process for generating the defective product image.
The image update model is
The input image (however, the first time is the composite image, the second and subsequent times is the output image previously output from the image update model) and the non-defective image are input to the pre-trained convolutional neural network.
The input image and the composite image are input to the edge extraction filter, and the input image is input to the edge extraction filter.
A model in which the input image is updated and output so that the total loss obtained by adding the first loss function obtained by the convolutional neural network and the second loss function obtained by the edge extraction filter is minimized. Is,
Defective product image generation program. The defective image generation program according to claim 1.
The image update model is
Further, the input image is input to a defect determination model for determining the presence or absence of a defective portion in the image, and the input image is input to the defect determination model.
The first loss function and the second loss function are further added to the third loss function obtained by the defect determination model to obtain the total loss.
Defective product image generation program. The defective image generation program according to claim 1 or 2.
On the computer
A second acquisition process for acquiring a non-defective image of the industrial product and a partial image including the defective portion of the industrial product.
A designation process for designating a first image area that is a part of a non-defective image of the industrial product, and
A second image generation process of targeting the first image region of the non-defective image and synthesizing the partial image at an arbitrary position in the first image region to generate the composite image.
Defective product image generation program to execute. The defective image generation program according to claim 3.
The industrial product is a gear
The first image region is an edge portion of a tooth in the gear and a peripheral portion of the edge portion.
Defective product image generation program. The defective image generation program according to claim 3 or 4.
The industrial product is a gear
The first image region is a region different from the tooth bottom in the gear.
Defective product image generation program. The defective image generation program according to claim 4 or 5.
In the designation process, the first image area is designated based on the position of the edge portion extracted by the edge extraction process on the non-defective image.
Defective product image generation program. The defective product image generation program according to any one of claims 3 to 6.
In the second image generation process, the saturation of the image obtained by synthesizing the partial image with the good product image is corrected based on the saturation of the good product image before synthesizing the partial image to generate the composite image. ,
Defective product image generation program. It is a quality judgment device that judges the quality of the judgment target product based on the presence or absence of defective parts in the captured image of the judgment target product.
The captured image of the product to be determined is applied to an image recognition model trained by using the defective image generated by the defective image generation program according to any one of claims 1 to 7 as teacher data. Input to judge the quality of the product to be judged.
Pass / fail judgment device.

Images