JP2021117152A - Image processing device, image processing method, and image processing program - Google Patents

Abstract

To determine the appropriate method and quantity of augmentation.SOLUTION: An image processing method includes: a step 701 for accepting an indication from a user that requests selection of a good image 31 from among a plurality of good images 31 and an indication from the user that pertains to the method and quantity of augmentation of the selected good image 31; a step 702 for performing augmentation of the selected good image 31 in accordance with the indication for the method and quantity of augmentation; a step 703 for displaying an image 32 of the product to be inspected and an augmented good image 35; and a step 705 for registering the good image 35, as a new good image 31, which has been augmented in accordance with an indication determined by the user among indications pertaining to the method and quantity of augmentation.SELECTED DRAWING: Figure 7

Description

The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

In the field of manufacturing industrial products, an inspection method for evaluating the presence or absence and degree of defects in a product has been put into practical use by visual inspection applying image processing technology. In the field of this kind of inspection method, attempts to improve inspection accuracy by applying machine learning are being studied. For example, there is known an algorithm that learns the characteristics of a non-defective product from an image of only a non-defective product of a product to be inspected, and determines whether the inspection target is a non-defective product or a defective product based on the learning result. In this algorithm, the features of a non-defective product are extracted from the image of the product to be inspected, and a restored image is generated from the extracted features. Even if the product to be inspected contains defects, the restored image does not have the characteristics of the defects, so the difference between the image of the product to be inspected and the restored image is compared. When is greater than or equal to the threshold value, it can be determined that the product to be inspected is a defective product containing defects.

In order to properly learn the characteristics of a non-defective product by such machine learning, a necessary and sufficient amount of images are required to analyze the variation in the characteristics of the non-defective product without omission. If there is a bias in the variation in the characteristics of the non-defective product represented by the image used for learning, the learning result will reflect the biased characteristics. There may be a deviation from the image of the product. Such a deviation can be a factor that is erroneously determined as a defect.

A method called augmentation is known as a means for obtaining a necessary and sufficient amount of images sufficient to analyze variations in the characteristics of non-defective products without omission. In this method, the image used for training is intentionally changed (for example, translation, rotation, enlargement / reduction, horizontal inversion, vertical inversion, etc.), and the changed image is added as an image for learning. This is a technique for increasing the number of images used for learning.

However, since the variation in the characteristics of non-defective products differs from product to product, what kind of variation and how much should be added to the image used for learning also differs from product to product. If an appropriate amount of variation is not added to the image used for learning, it can be a factor that causes false detection, as in the case where the variation of the characteristics of the non-defective product represented by the image used for learning is biased. In addition, there is a problem that the time spent on learning increases. In the past, it took time and effort to determine what kind of variation and how much should be added to the image used for learning by trial and error.

Therefore, it is an object of the present invention to propose an image processing apparatus, an image processing method, and an image processing program capable of solving such a problem and determining an appropriate method and amount of augmentation.

In order to solve the above-mentioned problems, the image processing device according to the present invention is a storage device for storing a plurality of non-defective images, and a user who requests to select one of the non-defective images from the plurality of non-defective images. Means for accepting instructions and instructions from the user regarding the method and quantity of augmentation of the selected good image, and means for augmenting the selected non-defective image according to the instructions regarding the method and quantity of augmentation. A display device that displays an image of the product to be inspected and an image of a non-defective product that has been augmented, and augmented according to one of the instructions regarding the method and amount of augmentation determined by the user. A means for registering a non-defective image as a new non-defective image in a storage device is provided. As a result, the user can determine an appropriate method and amount of augmentation by comparing the image of the product to be inspected with the image of the non-defective product that has been augmented.

The display device may further display a difference image between the image of the product to be inspected and the image of the non-defective product that has been augmented. From the difference image, the user can visually recognize the difference or difference between the image of the product to be inspected and the augmented non-defective image. This allows the user to determine the appropriate method and amount of augmentation, taking into account the difference between the image of the product to be inspected and the augmented non-defective image.

The display device may further display a composite image of the image of the product to be inspected and the augmented non-defective image. The user can visually recognize the difference or difference between the image of the product to be inspected and the augmented non-defective image from the composite image. This allows the user to determine the appropriate method and amount of augmentation, taking into account the difference between the image of the product to be inspected and the augmented non-defective image.

Here, the composite image may be an image in which the difference or difference between the image of the product to be inspected and the augmented non-defective image can be visually recognized. The composite image may be, for example, an image in which the pixel values of the image of the product to be inspected and the image of the non-defective product that has been augmented are converted into shades of different color components and then composited. The composite image may be, for example, an image obtained by synthesizing the edges extracted from each of the image of the product to be inspected and the image of the non-defective product that has been augmented. The composite image may be, for example, an image obtained by synthesizing the color densities of the image of the product to be inspected and the image of the non-defective product that has been augmented.

The display device may display a mark as a positioning mark on each of the image of the product to be inspected and the image of the non-defective product that has been augmented. This allows the user to determine the appropriate method and amount of augmentation using the mark as a marker so that the difference between the image of the product to be inspected and the image of the non-defective product that has been augmented is small. ..

The image processing apparatus according to the present invention further provides means for searching for a tentative solution of the augmentation method and quantity that minimizes the difference between the image of the product to be inspected and the image of the non-defective product that has been augmented. You may prepare. The display device may display a difference image between a non-defective product image augmented according to a tentative solution of the searched augmentation method and quantity and an image of the product to be inspected. This allows the user to determine whether the tentative solution of the searched augmentation method and quantity is correct.

The image processing method according to the present invention includes an instruction from a user requesting that the image processing apparatus select one of the non-defective images from a plurality of non-defective images, and a method of augmenting the selected non-defective image. And the step of accepting the user's instructions regarding the quantity, the step of augmenting the selected non-defective image according to the instructions regarding the augmentation method and quantity, the image of the product to be inspected, and the augmentation. A step of displaying the non-defective product image and a step of registering the non-defective product image augmented according to any instruction determined by the user among the instructions regarding the augmentation method and amount as a new non-defective product image are executed. As a result, the user can determine an appropriate method and amount of augmentation by comparing the image of the product to be inspected with the image of the non-defective product that has been augmented.

The image processing program according to the present invention is an instruction from a user requesting an image processing apparatus to select one of a plurality of non-defective images from a plurality of non-defective images, and a method of augmenting the selected non-defective image. And the step of accepting the user's instructions regarding the quantity, the step of augmenting the selected non-defective image according to the instructions regarding the augmentation method and quantity, the image of the product to be inspected, and the augmentation. The step of displaying the non-defective product image and the step of registering the non-defective product image augmented according to any instruction determined by the user among the instructions regarding the augmentation method and amount as a new non-defective product image are executed. As a result, the user can determine an appropriate method and amount of augmentation by comparing the image of the product to be inspected with the image of the non-defective product that has been augmented.

According to the present invention, an appropriate method and amount of augmentation can be determined.

It is the description which shows the outline of the augmentation procedure which concerns on embodiment of this invention. It is explanatory drawing which shows an example of augmentation which concerns on embodiment of this invention. It is explanatory drawing which shows an example of the partial screen which functions as the user interface which receives the instruction about the augmentation method and amount which concerns on embodiment of this invention. It is the description which shows an example of the hardware composition of the image processing apparatus which concerns on embodiment of this invention. It is explanatory drawing which shows an example of the machine learning in the image processing method which concerns on embodiment of this invention. It is explanatory drawing which shows an example of the process flow of the non-defective product determination in the image processing method which concerns on embodiment of this invention. It is a flowchart which shows an example of the processing flow of the image processing method which concerns on embodiment of this invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. This is an example of a screen capable of recognizing a difference or difference between an augmented non-defective product image and an image of a product to be inspected according to an embodiment of the present invention. It is a flowchart which shows an example of the processing flow of the image processing method which concerns on embodiment of this invention. It is a figure which shows an example of the setting screen which concerns on embodiment of this invention.

Hereinafter, embodiments relating to one aspect of the present invention will be described with reference to the drawings. The embodiments of the present invention are for facilitating the understanding of the present invention, and are not for limiting the interpretation of the present invention. The present invention can be modified or improved without departing from the spirit thereof, and the present invention also includes an equivalent thereof. It should be noted that the same reference numerals indicate the same components, and duplicate description will be omitted.

[Application example]
First, an application example of the present invention will be described. In the present embodiment, a class classified as a non-defective product and a class classified as a defective product are set in advance. In this specification, an image of a product classified into a non-defective product class is referred to as a non-defective product image. The image processing device according to the present embodiment registers a plurality of non-defective product images as learning images, and learns the characteristics of the non-defective product based on the registered non-defective product images. The image processing apparatus determines, for example, whether the image of the product to be inspected is classified into the non-defective product class or the defective product class by a known algorithm. In the present specification, such a judgment is referred to as a quality judgment.

Next, with reference to FIG. 1, the outline of the method of increasing the number of non-defective images used for learning by augmentation will be described. In FIG. 1, reference numeral 32 indicates an image of a product to be inspected. The product to be inspected may belong to the non-defective product class or the defective product class. The user selects a non-defective image 31 that most closely resembles the image 32 of the product to be inspected from the plurality of non-defective images 31 registered in advance. "Similarity" means the degree of similarity of the appearance features (eg, shape) of the product in the image. The user gives instructions regarding the augmentation method and amount so that the difference between the non-defective image 31 selected from the plurality of pre-registered non-defective images 31 and the image 32 of the product to be inspected is small. Give to the image processing device. Reference numeral 35 indicates a non-defective product that has been augmented according to instructions regarding the method and amount of augmentation input by the user so that the difference between the selected non-defective product image 31 and the image 32 of the product to be inspected is small. The image is shown. That is, the non-defective product image 35 is an image after the non-defective product image 31 has been augmented in response to an instruction from the user. When the non-defective image 35 does not match the image 32, the user gives the image processing apparatus, for example, a plurality of instructions regarding the method and amount of augmentation so that the difference between the non-defective image 35 and the image 32 is small. You can re-enter it times.

Here, the augmentation method means a method of changing the image by augmentation (for example, up movement, down movement, left movement, right movement, rotation, enlargement / reduction, left-right inversion, up-down inversion, etc.). do. The example shown in FIG. 2 shows an example in which the non-defective image 35 is generated by moving the non-defective image 31 up, down, left, or right. The amount of augmentation means a value that quantifies the degree of variation added to the image by augmentation. For example, when the non-defective product image 31 is moved to the right to generate the non-defective product image 35, the amount of augmentation means the amount of movement of the non-defective product image 31 in the right direction.

The difference between the non-defective product image 35 and the image 32 means the difference between the appearance characteristics of the product shown in the non-defective product image 35 and the appearance characteristics of the product shown in the image 32, and this difference is not necessarily limited to the area. It is desirable to take into consideration the difference between the shape of the product shown in the non-defective product image 35 and the shape of the product shown in the image 32, instead of making a judgment based on the above. The shape of the product shown in the non-defective image 35 and the product shown in the image 32 when the difference between the area of the product shown in the non-defective image 35 and the area of the product shown in the image 32 is minimized. When the difference from the shape of is, for example, a thin hollow circular shape, this may be erroneously recognized as a defect depending on the defect recognizing algorithm. Such misrecognition may result in a decrease in the accuracy of the quality determination.

FIG. 3 is an explanatory diagram showing an example of a partial screen that functions as a user interface for receiving instructions regarding an augmentation method and amount according to an embodiment of the present invention.
In FIG. 3, reference numeral 71 is a partial screen (for example, a window) that functions as a user interface for receiving an instruction from a user requesting that one of the non-defective product images 31 be selected from the plurality of non-defective product images 31. This is an example. The partial screen 71 presents a list of a plurality of non-defective images 31 registered in advance in the image processing apparatus. "Image 1", "Image 2", ..., "Image N" are file names of non-defective image 31. For example, the user can instruct the selection of the non-defective image 31 most similar to the image 32 from the list presented on the partial screen 71.

In FIG. 3, reference numeral 72 is an example of a partial screen (for example, a window) that functions as a user interface for receiving instructions regarding the augmentation method from the user. The partial screen 72 lists a plurality of options 80 for the augmentation method. Each of the plurality of options 80 is associated with a different augmentation method. The user can select a method of augmenting any non-defective image 31 selected from the list presented on the partial screen 71 from a plurality of options 80. The method of inputting instructions regarding the augmentation method is not limited to the form of choices, and may be, for example, text input or voice input.

In FIG. 3, reference numeral 73 is an example of a partial screen (for example, a window) that functions as a user interface for receiving an instruction regarding the amount of augmentation from the user. In this example, the partial screen 73 receives instructions regarding the amount of augmentation by inputting "numbers" from the user. For example, when "move up" is selected from the options 80 as the augmentation method and "10" is input as the amount of augmentation, it is selected from the list presented on the partial screen 71. An instruction to move the good product image 31 up by the amount of "10" is input to the image processing device. The method of inputting the instruction regarding the amount of augmentation is not limited to the numerical input, and may be, for example, a choice format or a voice input.

In one or more instructions regarding the augmentation method and amount entered through the partial screens 72 and 73, the user takes into account the area and shape of the difference between the non-defective image 35 and the image 32. Determine any one of the instructions when it will be the least moderately low. The image processing apparatus registers the non-defective image 35 obtained by augmenting the non-defective image 31 according to the instructions thus determined as a new non-defective image. In this way, the user can determine an appropriate method and amount of augmentation while comparing the image 32 of the product to be inspected with the augmented non-defective product image 35. As a result, the number of non-defective image 31 used for learning can be increased by augmentation, while appropriately considering the variation in the appearance characteristics of the non-defective product, which differs from product to product.

[Hardware configuration]
Next, an example of the hardware configuration of the image processing apparatus 10 according to the present embodiment will be described with reference to FIG.
The image processing device 10 is a computer system including a processor 11, a main memory 12, a camera interface 13, an input / output interface 14, a display interface 15, a communication interface 16, and a storage device 17 as hardware resources thereof. Is.

The storage device 17 is a computer-readable recording medium such as a disk medium (for example, a magnetic recording medium or an optical magnetic recording medium) or a semiconductor memory (for example, a volatile memory or a non-volatile memory). Such a recording medium can also be referred to as, for example, a non-transient recording medium. The storage device 17 stores an image processing program 20, a non-defective image 31, and a learning device 40. The image processing program 20 is a computer program for causing the processor 11 to execute the image processing method according to the present embodiment. The image processing program 20 is read from the storage device 17 into the main memory 12, interpreted and executed by the processor 11, and the image processing method according to the present embodiment is executed. The learner 40 is generated by machine learning of the features of the plurality of non-defective images 31.

A camera 51 is connected to the camera interface 13. The camera 51 may include, for example, an image sensor that captures a color image. The camera 51 may be built in the image processing device 10 or externally attached to the image processing device 10. When the product to be inspected is a non-defective product, the image taken by the camera 51 is stored in the storage device 17 as a non-defective product image 31.

The inspection target may be, for example, a work such as a device or a part conveyed on a production line by a belt conveyor. The image processing device 10 may be installed on the production line, or may be installed at a place different from the production line. When the image processing device 10 is installed in a place different from the production line, the image processing device 10 transmits the image obtained by photographing the inspection target by the camera installed in the production line through the communication interface 16. It may be received from a wired or wireless network.

An input device 52 and an output device 53 are connected to the input / output interface 14. The input device 52 is a device for the user to input an instruction regarding the selection of the non-defective image 31 and an instruction regarding the method and amount of augmentation of the selected non-defective image 31 to the image processing device 10. The input device 52 is, for example, a keyboard, a mouse, a touch pad, or the like. The output device 53 is a device that outputs various processing results (for example, quality determination results of products to be inspected) and the like. The output device 53 is, for example, a printer.

A display device 54 is connected to the display interface 15. The display device 54 displays various information used in the process for increasing the number of non-defective images 31 by augmentation. Examples of such information include partial screens 71, 72, 73 that function as a user interface for receiving instructions from the user, and various screens (for example, the screens shown in FIGS. 8 to 31).

[Learning process]
FIG. 5 is an explanatory diagram showing an example of machine learning in the image processing method according to the embodiment of the present invention. The learning device 40 learns the characteristics of the non-defective product from the plurality of non-defective product images 31 without using the image of the defective product.

[Good / bad judgment processing]
FIG. 6 is an explanatory diagram showing an example of a flow of processing for determining a non-defective product in the image processing method according to the embodiment of the present invention. The learning device 40 extracts the features of the non-defective product from the image 32 of the product to be inspected by, for example, the subspace method, and generates the restored image 33 from the extracted features. The image processing device 10 generates a difference image 34 which is a difference between the image 32 and the restored image 33, and obtains a result value (for example, an area value) obtained by binarizing and labeling the difference image 34 and a threshold value. Good or bad is judged by comparison.

[Image processing method]
FIG. 7 is a flowchart showing an example of the processing flow of the image processing method according to the embodiment of the present invention. This image processing method relates to a method of increasing the number of non-defective images 31 used for learning by augmentation.

In step 701, the image processing device 10 receives an instruction from the user requesting to select one of the non-defective product images 31 from the plurality of non-defective product images 31 through the partial screen 71, and also receives the selected non-defective product image 31. Instructions from the user regarding the method and amount of the augmentation of the above are received through the partial screens 72 and 73.

In step 702, the image processing apparatus 10 augments the non-defective image 31 selected in step 701 according to the instructions regarding the method and amount of augmentation specified in step 701.

In step 703, the image processing device 10 can recognize the difference or difference between the image 32 of the product to be inspected and the non-defective product image 35 augmented in step 702 (for example, shown in FIGS. 8 to 31). Screen) is displayed on the display device 54. By referring to such a screen, the user determines whether or not the difference between the image 32 of the product to be inspected and the non-defective image 35 augmented in step 702 is within the permissible range. Can be done. If the difference between the image 32 of the product to be inspected and the non-defective image 35 augmented in step 702 is not within the permissible range, the determination result in step 704, which will be described later, is a negative determination, and the process of FIG. 7 is performed. Return to step 701. In step 701, the user gives an instruction regarding the augmentation method and amount so that the difference between the non-defective image 31 selected from the plurality of non-defective images 31 and the image 32 of the product to be inspected is small. It can be re-input to the processing device 10.

In step 704, the image processing apparatus 10 was augmented with the image 32 of the product to be inspected and step 702 in one or more instructions regarding the method and amount of augmentation entered in step 701. It is determined whether or not any one instruction when the difference from the non-defective image 35 is most appropriately reduced in consideration of the area and the shape is determined by the user. The user can, for example, operate the input device 52 to instruct the image processing apparatus 10 of this determination.

In step 705, the image processing apparatus 10 registers the non-defective product image 35 augmented according to the instruction determined in step 704 as a new non-defective product image.

Next, a screen example shown in FIG. 8 and FIG. 31 will be described.
In the screen example shown in FIG. 8, the image 32 of the product to be inspected and the non-defective product image 35 augmented in step 702 are displayed in comparison with each other. In this example, as reference information when comparing the image 32 of the product to be inspected with the augmented non-defective image 35, for example, the pixel value of a certain point P in the image 32 is displayed on the partial screen ( For example, it may be displayed on the window (91) and the pixel value of a certain point Q in the non-defective image 35 may be displayed on the partial screen (for example, the window) 92. Here, the relative positional relationship of the point P with respect to the specific position (for example, the center position) of the image 32 and the relative positional relationship of the point Q with respect to the specific position (for example, the center position) of the non-defective image 35 are the same. Is. The user can accurately recognize the difference between the non-defective image 35 and the image 32 based on the pixel values displayed on the partial screens 91 and 92, respectively.

In the screen example shown in FIG. 9, in addition to the image 32 of the product to be inspected and the non-defective product image 35 augmented in step 702, the difference image 36 is further contrasted and displayed. The difference image 36 shows the difference between the image 32 and the non-defective image 35. The difference image 36 may be, for example, a binarized difference between the pixel values of the corresponding pixels of the image 32 and the non-defective image 35. The user can determine the appropriate method and amount of augmentation, taking into account the difference between the image 32 of the product to be inspected and the augmented good product image 35.

In the screen example shown in FIGS. 10 to 12, in addition to the image 32 of the product to be inspected and the non-defective image 35 augmented in step 702, any one of the composite images 37A, 37B, and 37C is composited. The image is further contrasted. The composite image 37A is, for example, an image in which the pixel values of the image 32 and the non-defective image 35 are converted into shades of different color components and then composited. For example, each pixel value of the image 32 is converted into a shade of the first color component (for example, red), and each pixel value of the non-defective image 35 is converted into a second color component (for example, blue) different from the first color component. ) May be converted into a shade of), and then the two may be combined to generate a composite image 37A. The composite image 37B is, for example, an image obtained by synthesizing the edges extracted from each of the image 32 and the non-defective image 35. The composite image 37C is, for example, an image obtained by synthesizing the color densities of the image 32 and the non-defective image 35.

From the composite images 37A, 37B, and 37C, the user can visually recognize the difference or difference between the image 32 of the product to be inspected and the non-defective image 35 that has been augmented. This allows the user to determine the appropriate method and amount of augmentation, taking into account the difference between the image 32 of the product to be inspected and the augmented non-defective image 35.

The image processing device 10 may switch and display the composite images 37A, 37B, and 37C to the display device 54 at regular time intervals, for example. For example, the image processing device 10 switches the display from the composite image 37A to the composite image 37B after displaying the composite image 37A for a certain period of time, displays the composite image 37B for a certain period of time, and then displays the composite image 37B to the composite image 37C. Switching may be performed. Further, such display switching may be performed by the image processing device 10 in response to an instruction from the user. The user can instruct the image processing device 10 by using the input device 52 which composite image should be displayed at which timing. The image processing device 10 may have a user interface that accepts such instructions.

In the screen example shown in FIG. 13, marks 61 and 62, which serve as positioning marks, are displayed on the image 32 of the product to be inspected and the non-defective product image 35 augmented in step 702, respectively. For example, the relative positional relationship of the mark 61 with respect to a specific position (for example, the center position) of the image 32 and the relative positional relationship of the mark 62 with respect to the specific position (for example, the center position) of the non-defective image 35 are the same. Marks 61 and 62 are displayed so as to become. As a result, the user can input instructions regarding the augmentation method and amount so that the difference between the non-defective image 35 and the image 32 is reduced by using the marks 61 and 62 as markers.

In the screen example shown in FIG. 14, the image 32 of the product to be inspected and the difference image 36 are displayed in comparison.

In the screen example shown in FIG. 15, in addition to the image 32 and the difference image 36 of the product to be inspected, the non-defective product image 35 augmented in step 702 is further contrasted and displayed. In this example, as reference information when comparing the image 32 of the product to be inspected with the augmented non-defective product image 35, for example, as in FIG. 8, at a certain point P in the image 32. The pixel value may be displayed on the partial screen (for example, window) 91, and the pixel value of a certain point Q in the non-defective image 35 may be displayed on the partial screen (for example, window) 92.

In the screen example shown in FIGS. 16 to 18, in addition to the image 32 and the difference image 36 of the product to be inspected, any one of the composite images 37A, 37B, and 37C is further contrasted and displayed.

In the screen example shown in FIG. 19, in addition to the image 32 of the product to be inspected and the difference image 36, the non-defective product image 35 augmented in step 702 is further contrasted and displayed. In this example, similarly to FIG. 13, marks 61 and 62 as positioning marks may be displayed on the image 32 of the product to be inspected and the non-defective product image 35 augmented in step 702, respectively. ..

In the screen example shown in FIGS. 20 to 22, the image 32 of the product to be inspected and the composite image of any one of the composite images 37A, 37B, and 37C are displayed in contrast.

In the screen example shown in FIGS. 23 to 25, in addition to the image 32 of the product to be inspected and the composite image of any one of the composite images 37A, 37B, and 37C, the non-defective product image augmented in step 702. 35 is further contrasted and displayed. In these examples, as reference information when comparing the image 32 of the product to be inspected with the augmented non-defective product image 35, for example, as in FIG. 8, a certain point P in the image 32 The pixel value of is displayed on the partial screen (for example, window) 91, and the pixel value of a certain point Q in the non-defective image 35 may be displayed on the partial screen (for example, window) 92.

In the screen example shown in FIGS. 26 to 28, in addition to the image 32 of the product to be inspected and the composite image of any one of the composite images 37A, 37B, and 37C, the non-defective product image augmented in step 702. 35 is further contrasted and displayed. In these examples, similarly to FIG. 13, even if the images 32 of the product to be inspected and the non-defective product image 35 augmented in step 702 are displayed with marks 61 and 62 as positioning marks, respectively. good.

In the screen example shown in FIGS. 29 to 31, in addition to the image 32 of the product to be inspected and the composite image of any one of the composite images 37A, 37B, and 37C, the difference image 36 is further contrasted and displayed.

FIG. 32 is a flowchart showing an example of the processing flow of the image processing method according to the embodiment of the present invention. This image processing method relates to an augmentation method that minimizes the difference between the image 32 of the product to be inspected and the non-defective image 35 that has been augmented, and a method that automatically searches for a tentative solution in quantity.

In step 3201, the image processing device 10 selects any one of the non-defective image 31 from the plurality of non-defective images 31 stored in the storage device 17.

In step 3202, the image processing apparatus 10 selects one method of augmenting the non-defective image 31 selected in step 3201.

In step 3203, the image processing apparatus 10 selects an amount of the non-defective image 31 selected in step 3201 to be augmented by the augmentation method selected in step 3202. For this selection, for example, any amount within the range of the amount of augmentation predetermined for each augmentation method may be arbitrarily selected.

In step 3204, the image processing apparatus 10 sets the non-defective image 31 selected in step 3201 together with the non-defective image 35 augmented according to the augmentation method and amount selected in steps 3202 and 3203. The difference A from the image 32 of the product is calculated. The difference A may be, for example, an area value obtained by binarizing the difference between the pixel values of the corresponding pixels of the non-defective image 35 and the image 32.

In step 3205, the image processing apparatus 10 determines whether or not the difference A obtained by the calculation in step 3204 is smaller than the tentative minimum value Amin.

In step 33206, the image processing apparatus 10 sets the difference A obtained by the calculation in step 3204 as a tentative minimum value Amin.

In step 3207, the image processing apparatus 10 determines whether or not to change the method of augmenting the non-defective image 31 selected in step 3201. This determination may be based on, for example, whether or not the selection of the amount of the non-defective image 31 selected in step 3201 to be augmented by the augmentation method selected in step 3202 has been made for the entire range. ..

In step 3208, the image processing apparatus 10 determines whether or not to change the non-defective image 31 to be selected in step 3201. This determination may be based on, for example, whether or not the non-defective image 31 selected in step 3201 has been augmented for all the augmentation methods and amounts.

The minimum value Amin obtained by steps 3201 to 208 above is a tentative solution of the augmentation method and amount that minimizes the difference between the image 32 of the product to be inspected and the non-defective image 35 that has been augmented. Become.

The display device 54 displays a difference image 36 between the non-defective product image 35 augmented according to the tentative solution of the augmentation method and quantity searched in steps 3201 to 208 above and the image 32 of the product to be inspected. indicate. For example, the difference images 36 shown in FIGS. 9, 14-19, and 29-31 are augmented according to a tentative solution of the augmentation method and quantity retrieved in steps 3201 to 208 above. It may be a difference image between the non-defective product image 35 and the image 32 of the product to be inspected. This allows the user to determine whether the tentative solution of the searched augmentation method and quantity is correct. For example, the shape of the product shown in the non-defective image 35, which is augmented according to the tentative solution of the searched augmentation method and quantity, and the shape of the product shown in the image 32 of the product to be inspected. If the difference between the two is misidentified as a defect, the user may determine that the tentative solution of the augmentation method and quantity is incorrect.

The image processing device 10 functions as a means for executing the above-mentioned image processing method (steps 701 to 705 in FIG. 7 and steps 3201 to 208 in FIG. 32), and such means is not necessarily the hardware of the image processing device 10. It does not have to be realized in collaboration with the resource and the image processing program 20, for example, a dedicated hardware resource for the image processing apparatus 10 (eg, an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA)). Etc.) may be used.

FIG. 33 shows an example of the setting screen 100 for receiving an instruction regarding the augmentation method and amount from the user. The setting screen 100 includes an inspection image display unit 101, a registered image display unit 102, an augmentation display unit 103, a difference image display unit 104, a registered image list display unit 105, and an augmentation parameter specification unit 106. The setting screen 100 is displayed on the display device 54, for example.

The registered image list display unit 105 displays a list of a plurality of non-defective image 31 registered in advance. When any one of the non-defective product images 31 is selected from this list by the user, the selected non-defective product image 31 is displayed on the registered image display unit 102. The user can specify the augmentation parameter from the augmentation parameter specification unit 106. The augmentation parameter is a parameter that determines the amount of augmentation of the non-defective image 31. In the example shown in FIG. 33, the horizontal movement amount and the rotational movement amount are exemplified as the augmentation parameters, but the example is not limited to this example. The augmentation display unit 103 displays the non-defective image 35 that has been augmented according to the designated augmentation parameters. On the other hand, the inspection image display unit 101 displays an image 32 of the product to be inspected. A difference image 36 showing the difference between the image 32 and the non-defective image 35 is displayed on the difference image display unit 104.

Some or all of the above embodiments may be described, but not limited to:
(Appendix 1)
A storage device 17 that stores a plurality of non-defective images 31 and
A means for receiving an instruction from a user requesting to select one of the good product images 31 from a plurality of good product images 31 and an instruction from the user regarding the method and amount of augmentation of the selected good product image 31. 701 and
Means 702 for augmenting the selected non-defective image 31 according to instructions regarding the method and quantity of augmentation, and
A display device 54 that displays an image 32 of the product to be inspected and an augmented non-defective image 35, and
Means 705 for registering the non-defective image 35 augmented according to any instruction determined by the user among the instructions regarding the augmentation method and amount in the storage device 17 as a new non-defective image 31.
An image processing device 10 comprising.
(Appendix 2)
The image processing apparatus 10 according to Appendix 1.
The display device 54 is an image processing device 10 that further displays a difference image 36 between the image 32 of the product to be inspected and the non-defective product image 35 that has been augmented.
(Appendix 3)
The image processing apparatus 10 according to Appendix 1.
The display device 54 is an image processing device 10 that further displays composite images 37A, 37B, and 37C of the image 32 of the product to be inspected and the non-defective image 35 that has been augmented.
(Appendix 4)
The image processing apparatus 10 according to Appendix 3, wherein the image processing apparatus 10 is used.
The composite image 37A is an image processed in which the pixel values of the image 32 of the product to be inspected and the augmented non-defective image 35 are converted into shades of different color components and then composited. Device 10.
(Appendix 5)
The image processing apparatus 10 according to Appendix 3, wherein the image processing apparatus 10 is used.
The composite image 37B is an image processing device 10 which is an image obtained by synthesizing edges extracted from each of an image 32 of a product to be inspected and an augmented non-defective image 35.
(Appendix 6)
The image processing apparatus 10 according to Appendix 3, wherein the image processing apparatus 10 is used.
The composite image 37C is an image processing device 10 which is an image obtained by synthesizing the color densities of the image 32 of the product to be inspected and the image 35 of the non-defective product which has been augmented.
(Appendix 7)
The image processing apparatus 10 according to Appendix 1.
The display device 54 is an image processing device 10 that displays marks 61 and 62 as positioning marks on the image 32 of the product to be inspected and the non-defective product image 35 that has been augmented, respectively.
(Appendix 8)
The image processing apparatus 10 according to Appendix 2, wherein the image processing apparatus 10 is used.
Further provided are means 3201 to 208 for searching for a tentative solution of the augmentation method and quantity that minimizes the difference between the image 32 of the product to be inspected and the non-defective image 35 of the augmented product.
The display device 54 is an image processing device that displays a difference image 36 between a non-defective product image 35 augmented according to a tentative solution of the searched augmentation method and quantity and an image 32 of the product to be inspected.
(Appendix 9)
The image processing device 10
A step of accepting an instruction from a user requesting to select one of the good product images 31 from a plurality of good product images 31 and an instruction from the user regarding the method and amount of augmentation of the selected good product image 31. 701 and
Step 702, which augments the selected non-defective image 31 according to instructions regarding the method and quantity of augmentation, and
Step 703 to display the image 32 of the product to be inspected and the augmented non-defective image 35, and
Step 705 of registering the non-defective image 35 augmented according to any instruction determined by the user among the instructions regarding the method and amount of augmentation as a new non-defective image 31.
Image processing method to execute.
(Appendix 10)
For image processing equipment
A step of accepting an instruction from a user requesting to select one of the good product images 31 from a plurality of good product images 31 and an instruction from the user regarding the method and amount of augmentation of the selected good product image 31. 701 and
Step 702, which augments the selected non-defective image 31 according to instructions regarding the method and quantity of augmentation, and
Step 703 to display the image 32 of the product to be inspected and the augmented non-defective image 35, and
Step 705 of registering the non-defective image 35 augmented according to any instruction determined by the user among the instructions regarding the method and amount of augmentation as a new non-defective image 31.
Image processing program 20 for executing.

10 ... Image processing device 11 ... Processor 12 ... Main memory 13 ... Camera interface 14 ... Input / output interface 15 ... Display interface 16 ... Communication interface 17 ... Storage device 20 ... Image processing program 31 ... Good product image 32 ... Image 33 ... Restored image 34 … Difference image 35… Good product image 36… Difference image 37A, 37B, 37C… Composite image 54… Display device 61, 62… Mark 71, 72, 73… Partial screen 80… Choices

Claims (30)

A storage device that stores multiple non-defective images,
A means for receiving an instruction from a user requesting to select one of the good product images from the plurality of non-defective product images and an instruction from the user regarding the method and amount of augmentation of the selected non-defective product image. When,
A means for augmenting the selected non-defective image according to the instructions regarding the method and amount of the augmentation, and
A display device that displays an image of the product to be inspected and the augmented image of the non-defective product.
A means for registering the non-defective product image augmented according to any instruction determined by the user among the instructions regarding the augmentation method and amount in the storage device as a new non-defective product image.
An image processing device comprising. The image processing apparatus according to claim 1.
The display device is an image processing device that further displays a difference image between the image of the product to be inspected and the augmented non-defective product image. The image processing apparatus according to claim 1.
The display device is an image processing device that further displays a composite image of the image of the product to be inspected and the augmented non-defective image. The image processing apparatus according to claim 3.
The composite image is an image processed after converting the pixel values of the image of the product to be inspected and the augmented non-defective image into shades of different color components. Device. The image processing apparatus according to claim 3.
The composite image is an image processing apparatus that is an image obtained by synthesizing edges extracted from each of the image of the product to be inspected and the augmented non-defective image. The image processing apparatus according to claim 3.
The composite image is an image processing apparatus that is an image obtained by synthesizing the color densities of the image of the product to be inspected and the augmented non-defective image. The image processing apparatus according to claim 1.
The display device is an image processing device that displays a mark as a positioning mark on each of the image of the product to be inspected and the image of the non-defective product that has been augmented. The image processing apparatus according to claim 2.
Further provided with means for searching for a tentative solution of augmentation method and quantity that minimizes the difference between the image of the product to be inspected and the image of the non-defective product that has been augmented.
The display device is an image processing device that displays a difference image between the non-defective product image augmented according to a tentative solution of the searched augmentation method and quantity and the image of the product to be inspected. The image processing device
A step of accepting an instruction from a user requesting to select one of the good product images from a plurality of non-defective product images, and an instruction from the user regarding the method and amount of augmentation of the selected non-defective product image. ,
The step of augmenting the selected non-defective image according to the instructions regarding the method and amount of the augmentation, and
A step of displaying an image of the product to be inspected and the augmented image of the non-defective product.
A step of registering the non-defective product image augmented according to any instruction determined by the user among the instructions regarding the augmentation method and amount as a new non-defective product image, and
Image processing method to execute. For image processing equipment
A step of accepting an instruction from a user requesting to select one of the good product images from a plurality of non-defective product images, and an instruction from the user regarding the method and amount of augmentation of the selected non-defective product image. ,
The step of augmenting the selected non-defective image according to the instructions regarding the method and amount of the augmentation, and
A step of displaying an image of the product to be inspected and the augmented image of the non-defective product.
A step of registering the non-defective product image augmented according to any instruction determined by the user among the instructions regarding the augmentation method and amount as a new non-defective product image, and
An image processing program that executes. A storage device that stores multiple non-defective images,
A means for receiving an instruction from a user requesting to select one of the good product images from the plurality of non-defective product images and an instruction from the user regarding the method and amount of augmentation of the selected non-defective product image. When,
A means for augmenting the selected non-defective image according to the instructions regarding the method and amount of the augmentation, and
A display device that displays an image of the product to be inspected and a difference image between the image of the product to be inspected and the augmented non-defective product image.
A means for registering the non-defective product image augmented according to any instruction determined by the user among the instructions regarding the augmentation method and amount in the storage device as a new non-defective product image.
An image processing device comprising. The image processing apparatus according to claim 11.
Further provided with means for searching for a tentative solution of augmentation method and quantity that minimizes the difference between the image of the product to be inspected and the image of the non-defective product that has been augmented.
The display device is an image processing device that displays a difference image between the non-defective product image augmented according to a tentative solution of the searched augmentation method and quantity and the image of the product to be inspected. The image processing apparatus according to claim 11.
The display device is an image processing device that further displays the augmented non-defective image. The image processing apparatus according to claim 11.
The display device is an image processing device that further displays a composite image of the image of the product to be inspected and the augmented non-defective image. The image processing apparatus according to claim 14.
The composite image is an image processed after converting the pixel values of the image of the product to be inspected and the augmented non-defective image into shades of different color components. Device. The image processing apparatus according to claim 14.
The composite image is an image processing apparatus that is an image obtained by synthesizing edges extracted from each of the image of the product to be inspected and the augmented non-defective image. The image processing apparatus according to claim 14.
The composite image is an image processing apparatus that is an image obtained by synthesizing the color densities of the image of the product to be inspected and the augmented non-defective image. The image processing apparatus according to claim 13.
The display device is an image processing device that displays a mark as a positioning mark on each of the image of the product to be inspected and the image of the non-defective product that has been augmented. The image processing device
A step of accepting an instruction from a user requesting to select one of the good product images from a plurality of non-defective product images, and an instruction from the user regarding the method and amount of augmentation of the selected non-defective product image. ,
The step of augmenting the selected non-defective image according to the instructions regarding the method and amount of the augmentation, and
A step of displaying an image of the product to be inspected and a difference image between the image of the product to be inspected and the augmented non-defective product image.
A step of registering the non-defective product image augmented according to any instruction determined by the user among the instructions regarding the augmentation method and amount as a new non-defective product image, and
Image processing method to execute. For image processing equipment
A step of accepting an instruction from a user requesting to select one of the good product images from a plurality of non-defective product images, and an instruction from the user regarding the method and amount of augmentation of the selected non-defective product image. ,
The step of augmenting the selected non-defective image according to the instructions regarding the method and amount of the augmentation, and
A step of displaying an image of the product to be inspected and a difference image between the image of the product to be inspected and the augmented non-defective product image.
A step of registering the non-defective product image augmented according to any instruction determined by the user among the instructions regarding the augmentation method and amount as a new non-defective product image, and
An image processing program that executes. A storage device that stores multiple non-defective images,
A means for receiving an instruction from a user requesting to select one of the good product images from the plurality of non-defective product images and an instruction from the user regarding the method and amount of augmentation of the selected non-defective product image. When,
A means for augmenting the selected non-defective image according to the instructions regarding the method and amount of the augmentation, and
A display device that displays an image of the product to be inspected and a composite image of the image of the product to be inspected and the augmented non-defective product image.
A means for registering the non-defective product image augmented according to any instruction determined by the user among the instructions regarding the augmentation method and amount in the storage device as a new non-defective product image.
An image processing device comprising. The image processing apparatus according to claim 21.
The composite image is an image processed after converting the pixel values of the image of the product to be inspected and the augmented non-defective image into shades of different color components. Device. The image processing apparatus according to claim 21.
The composite image is an image processing apparatus that is an image obtained by synthesizing edges extracted from each of the image of the product to be inspected and the augmented non-defective image. The image processing apparatus according to claim 21.
The composite image is an image processing apparatus that is an image obtained by synthesizing the color densities of the image of the product to be inspected and the augmented non-defective image. The image processing apparatus according to any one of claims 21 to 24.
The display device is an image processing device that further displays the augmented non-defective image. The image processing apparatus according to claim 25.
The display device is an image processing device that displays a mark as a positioning mark on each of the image of the product to be inspected and the image of the non-defective product that has been augmented. The image processing apparatus according to claim 21.
The display device is an image processing device that further displays a difference image between the image of the product to be inspected and the augmented non-defective product image. The image processing apparatus according to claim 27.
Further provided with means for searching for a tentative solution of augmentation method and quantity that minimizes the difference between the image of the product to be inspected and the image of the non-defective product that has been augmented.
The display device is an image processing device that displays a difference image between the non-defective product image augmented according to a tentative solution of the searched augmentation method and quantity and the image of the product to be inspected. The image processing device
A step of accepting an instruction from a user requesting to select one of the good product images from a plurality of non-defective product images, and an instruction from the user regarding the method and amount of augmentation of the selected non-defective product image. ,
The step of augmenting the selected non-defective image according to the instructions regarding the method and amount of the augmentation, and
A step of displaying an image of a product to be inspected and a composite image of the image of the product to be inspected and the augmented non-defective product image.
A step of registering the non-defective product image augmented according to any instruction determined by the user among the instructions regarding the augmentation method and amount as a new non-defective product image, and
Image processing method to execute. For image processing equipment
A step of accepting an instruction from a user requesting to select one of the good product images from a plurality of non-defective product images, and an instruction from the user regarding the method and amount of augmentation of the selected non-defective product image. ,
The step of augmenting the selected non-defective image according to the instructions regarding the method and amount of the augmentation, and
A step of displaying an image of a product to be inspected and a composite image of the image of the product to be inspected and the augmented non-defective product image.
A step of registering the non-defective product image augmented according to any instruction determined by the user among the instructions regarding the augmentation method and amount as a new non-defective product image, and
An image processing program that executes.

Images