JP2020187654A - Image inspection device - Google Patents

Abstract

To allow a user to easily determine a type of inspection with a higher correct answer rate when both normal inspection and deep learning processing-based inspection are made applicable.SOLUTION: An image inspection device provided herein is configured to: apply normal inspection processing and deep learning processing to both newly acquired good product images given with a good product attribute and defective product images given with a defective product attribute; and display correct answer rates for the normal inspection processing and for the deep learning processing in a comparable manner.SELECTED DRAWING: Figure 4

Description

The present invention relates to an image inspection apparatus that determines the quality of an inspection object based on an image of the inspection object.

For example, as disclosed in Patent Document 1, in a general inspection process using an image of an inspection object, it is based on various feature quantities (color, edge, position, etc.) of the inspection object in the image. The quality of the inspection target is judged (hereinafter referred to as normal inspection). In normal inspection, the user selects the feature amount to be inspected when setting the image inspection device, sets a threshold value as a reference for quality judgment for the selected feature amount, and newly operates it. The feature amount selected at the time of setting is extracted from the inspection image input to, and the quality of the inspection object is judged by comparing the extracted feature amount with the threshold value. It is easy to judge the quality of an image with clear features such as color and edges, but the appearance of edges changes depending on the surrounding environment, such as inspection objects with many color irregularities and metal parts. For easy inspection objects, the feature amount is likely to change depending on the imaging conditions, etc., and even if it is easy to judge whether it is good or bad by inspection with the human eye, it is difficult to judge by an image inspection device, and eventually the judgment result is not stable. is there.

As an inspection processing technology that can handle such difficult inspections, a new machine learning device such as a neural network is used to learn the characteristics of a good product image in which a good product is imaged and a defective product image in which a defective product is imaged. There is known a technique for identifying whether an image of an inspection object input to is a non-defective product or a defective product by a machine learning device (see, for example, Patent Document 2).

Japanese Unexamined Patent Publication No. 2013-12050 JP-A-2018-5640

By the way, learning with a neural network having multiple layers and greatly improving the discriminating ability is generally called deep learning. In deep learning, an image to which multiple good product attributes are given in advance and an image to which bad product attributes are given are input to a multi-layer neural network so that the good product image and the defective product image can be distinguished from each other in the network. Adjust multiple parameters to complete the learning. This deep learning is used in many fields.

However, the deep learning process has a problem that the behavior is unstable when unknown data not used for learning is input. That is, when the deep learning process is applied to the inspection process of the inspection object and the two-class discrimination of whether the inspection object is a non-defective product or a defective product is trained, for example, it does not exist in the training data. If such an unexpectedly defective product is input, it may be mistaken for a non-defective product, and this must be prevented in the inspection process at a factory or the like. Further, the deep learning process requires a longer processing time for determination than the normal inspection, which is also a problem to be improved in the inspection process in factories and the like.

In this way, deep learning processing has the advantage of enabling inspection of inspection objects that are difficult to handle in normal inspection, but the behavior of unknown data is unstable and the processing time is long. There was a problem. On the other hand, if the feature quantity is clear, a sufficiently stable inspection can be performed even by a normal inspection, and in such a case, there is almost no merit to use the deep learning process. However, there is a problem that it is difficult for the user to know which test is more suitable.

The present invention has been made in view of the above points, and an object of the present invention is to allow a user to easily perform a test having a high correct answer rate when making a normal test and a test by deep learning processing applicable. It is to be able to distinguish.

In order to achieve the above object, the first invention is to set a feature amount to be used for inspection in an image inspection apparatus that determines the quality of an inspection object based on an image of the inspection object acquired by imaging the inspection object. And the normal inspection setting means for setting the normal inspection process by accepting from the user the setting of the threshold value which is the standard of the quality judgment to be compared with the feature amount, and a plurality of good product images to which the good product attribute is given. And / or a plurality of defective product images to which the defective product attribute is given are input to the input layer of the neural network and trained by the neural network, and the newly input inspection target image is classified into a non-defective product image and a defective product image. A deep learning setting means for setting the deep learning process to be performed, an inspection execution unit that applies the normal inspection process and the deep learning process to both the newly acquired good product image and the defective product image, and the inspection. A display means for displaying the correct answer rate by the normal inspection process and the correct answer rate by the deep learning process executed by the execution unit, the normal inspection process, any one of the deep learning processes, or a combination of these processes. It is characterized by having an inspection selection means capable of selecting an inspection process.

According to this configuration, the feature amount and the threshold value used for the inspection can be set by the normal inspection setting means, and the good product image and the defective product image are learned by the neural network by the deep learning setting means. And you will be able to classify defective images. That is, it becomes possible to perform a normal inspection process and a deep learning process. Only the non-defective image may be input to the input layer of the neural network as the learning image, or only the defective image may be input. Further, by setting the number of layers of the neural network to three or more, the discrimination ability is greatly improved by so-called deep learning.

After setting the normal inspection process and the deep learning process, the correct answer rate when the normal inspection process is applied to the newly acquired non-defective image and the defective image, and the deep learning process for the newly acquired non-defective image and the defective image. The percentage of correct answers when is applied is displayed on the display means. The correct answer rate can be said to be, for example, the certainty (accuracy) of the judgment.

Since the correct answer rate when the normal inspection process is applied and the correct answer rate when the deep learning process is applied are displayed on the display means, the user can use either the normal inspection process or the deep learning process. Is suitable for inspection of the inspection object, it becomes possible to easily determine based on the display content of the display means. In addition, it becomes possible to easily determine whether or not a process that combines a normal inspection process and a deep learning process is suitable for inspection of an inspection object. The correct answer rate when the normal inspection process is applied and the correct answer rate when the deep learning process is applied may be displayed in a comparable form, and the comparable form is, for example, a graph display form or a numerical value. The display form can be mentioned.

Based on the comparison result of the correct answer rate, either one of the normal inspection process and the deep learning process, or an inspection process combining these processes can be selected. For example, when a sufficiently stable inspection is possible only by the normal inspection process, selecting the normal inspection process eliminates the unstable behavior peculiar to the deep learning process and shortens the processing time. Will be done. On the other hand, in the case of an inspection object that is difficult to handle only by the normal inspection process, the inspection accuracy is improved by selecting the deep learning process. The user may select the normal inspection process, the deep learning process, and the inspection process that combines these processes, or the image inspection device may automatically perform the selection.

The second invention is characterized in that the display means is configured to simultaneously display the correct answer rate by the normal inspection process and the correct answer rate by the deep learning process on the same graph.

According to this configuration, it becomes possible to easily compare the correct answer rate by the normal inspection process and the correct answer rate by the deep learning process.

In the third invention, in the normal inspection process, a pass / fail judgment is made using a normalized correlation, and the display means is configured to display the correct answer rate by the normal inspection process in association with the correlation value. It is characterized by.

According to this configuration, when a pass / fail judgment is performed using a normalized correlation such as a pattern search in a normal inspection process, a correlation value can be obtained. The higher the correlation value, the higher the correct answer rate in the normal inspection process. Therefore, by displaying the correct answer rate in the normal inspection process in association with the correlation value, it is possible to make an accurate judgment in selecting the inspection process.

The fourth invention is characterized in that when the correlation value obtained in the normal inspection process is equal to or less than a predetermined correlation value, it is determined that the product is defective in the normal inspection process.

According to this configuration, when the correlation value obtained by the normal inspection process is lower than the predetermined correlation value, it is clearly a defective product because an unexpected substance is mixed in or the product is oriented in an unexpected direction. There is a high possibility, and in such a case, it is judged as a defective product by the normal inspection process. As a result, it is possible to prevent the unstable behavior shown for unknown data in the deep learning process.

A fifth aspect of the invention is characterized in that the inspection selection means is configured to select the deep learning process when the correlation value obtained in the normal inspection process is higher than a predetermined correlation value. ..

That is, for example, when the correlation value obtained in the normal inspection process is near the middle level, it is considered difficult to distinguish between a non-defective product and a defective product in the normal inspection process. There are various causes for this, and examples thereof include those caused by disturbances such as shape distortion and uneven lighting. In this case, by combining with a deep learning process having a higher separation ability than the normal inspection process, it is possible to distinguish between a non-defective product and a defective product.

In the sixth invention, in the normal inspection process, a pass / fail judgment is made by a difference inspection that detects a difference between a pre-registered registered image and a newly acquired inspection target image, and the display means is the normal inspection. It is characterized in that the correct answer rate by processing is displayed in association with the blob area of the difference.

According to this configuration, when a pass / fail judgment is performed using blob detection in the normal inspection process, the blob area of the difference can be obtained. The smaller the blob area of the difference, the higher the correct answer rate in the normal inspection process. Therefore, by displaying the correct answer rate in the normal inspection process in association with the blob area of the difference, it is possible to make an accurate judgment when selecting the inspection process. Become.

The seventh invention is characterized in that when the blob area of the difference obtained in the normal inspection process is equal to or larger than a predetermined area value, it is determined to be a defective product in the normal inspection process. To do.

According to this configuration, when the blob area of the difference obtained by the normal inspection process is larger than the predetermined area value, an unexpected substance is mixed or the product is oriented in an unexpected direction, and the product is clearly defective. In such a case, it is determined to be a defective product by the normal inspection process. As a result, it is possible to prevent the unstable behavior shown for unknown data in the deep learning process.

An eighth invention is characterized in that the inspection selection means is configured to select the deep learning process when the blob area of the difference obtained in the normal inspection process is smaller than the predetermined area value. And.

That is, for example, when the blob area of the difference obtained in the normal inspection process is near the middle level, it is considered that it is difficult to distinguish between a non-defective product and a defective product in the normal inspection process. There are various causes for this, and examples thereof include those caused by disturbances such as shape distortion and uneven lighting. In this case, by combining with a deep learning process having a higher separation ability than the normal inspection process, it is possible to distinguish between a non-defective product and a defective product.

The ninth invention is configured such that the normal inspection setting means can be compared with the feature amount and can accept from the user a threshold for determining a non-defective product or a threshold for determining a defective product. In the normal inspection process, for an inspection target image having a feature amount capable of determining a non-defective product or a defective product based on the feature amount and the threshold for determining the non-defective product determination or the threshold for determining the defective product determination. It is characterized in that the deep learning process is applied to an inspection target image having a feature amount in which the non-defective product determination or the defective product determination is confirmed and the non-defective product determination or the defective product determination cannot be determined.

That is, in the actual inspection site of the inspection object, in general, most of the inspection objects (for example, 99% or more) are often non-defective products. In the present invention, the throughput is greatly improved by inspecting an inspection object that can be clearly determined to be a non-defective product or an inspection object that can be clearly determined to be a defective product by a normal inspection process having a high processing speed. Since only the remaining few inspection objects are inspected by the deep learning process, it is possible to improve the inspection accuracy while suppressing the decrease in the processing speed.

According to the tenth invention, in the normal inspection process, a non-defective product determination is determined for an inspection target image having a feature amount capable of determining a non-defective product based on the feature amount and the threshold value, and a defective product determination is performed. For the inspection target image having a possible feature amount, the defective product judgment is confirmed, and for the inspection target image having the feature amount that the non-defective product judgment cannot be determined and the inspection target image having the feature amount that the defective product determination cannot be determined. It is characterized in that it is configured to apply the deep learning process only to the user.

According to this configuration, both an inspection object that can be clearly determined to be a non-defective product and an inspection object that can be clearly determined to be a defective product can be inspected by a normal inspection process having a high processing speed.

According to the present invention, the normal inspection process and the deep learning process are applied to both the newly acquired non-defective product image to which the non-defective product attribute is given and the defective product image to which the defective product attribute is given, and the normal inspection process is applied. Since the correct answer rate by the above and the correct answer rate by the deep learning process can be displayed in a comparable form, the user can easily determine the test having a high correct answer rate.

It is a schematic diagram which shows the structure of the image inspection apparatus which concerns on embodiment of this invention. It is a figure which shows the hardware structure of the image inspection apparatus. It is a block diagram of an image inspection apparatus. It is a figure which shows the user interface which displays when the pattern search and the deep learning process are combined. It is a figure which shows the user interface which displays when the difference inspection and the deep learning process are combined. It is a figure which shows the user interface which displays when the learning inspection and the deep learning process are combined. It is a figure which shows the user interface which displays when the scratch detection and the deep learning process are combined. It is a flowchart which shows the procedure when the defective product judgment is performed by a normal inspection process, and the quality is judged by a deep learning process for the rest. It is a figure which shows the cumulative histogram of only the deep learning process, and the cumulative histogram when the normal inspection process and the deep learning process are combined. It is a flowchart which shows the procedure when the non-defective product is judged by a normal inspection process, and the rest is judged by a deep learning process. It is a flowchart which shows the procedure in the case of confirming a good product judgment and a defective product judgment by a normal inspection process, and judging the quality of the rest by a deep learning process. FIG. 6 is a diagram corresponding to FIG. 6 showing a case where a non-defective product image and a defective product image are the first distribution examples. FIG. 6 is a diagram corresponding to FIG. 6 showing a case where a non-defective product image and a defective product image are the second distribution example. FIG. 7 is a diagram corresponding to FIG. 7 showing a case where a non-defective product image and a defective product image are the first distribution examples. It is a figure corresponding to FIG. 7 which shows the case where a good product image and a defective product image are the second distribution example. It is a figure which shows the user interface when displaying the cumulative histogram. FIG. 16 is a diagram corresponding to FIG. 16 showing an example of distribution when a good product image and a defective product image cannot be separated by both the normal inspection process and the deep learning process. It is a figure corresponding to FIG. 16 when the threshold value display line for normal inspection is displayed. It is a figure which shows the distribution of only defective products, the distribution of only non-defective products, and the distribution of a mixture of defective products and non-defective products in the cumulative histogram. FIG. 16 is a diagram corresponding to FIG. 16 showing an example in the case of displaying numerical values. FIG. 20 is a diagram corresponding to FIG. 20 showing an example of distribution when a good product image and a defective product image cannot be separated by both the normal inspection process and the deep learning process. It is a figure corresponding to FIG. 20 when the threshold value display line for normal inspection is displayed.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. It should be noted that the following description of the preferred embodiment is essentially merely an example and is not intended to limit the present invention, its application or its use.

FIG. 1 is a schematic view showing a configuration of an image inspection device 1 according to an embodiment of the present invention. The image inspection device 1 is a device for determining the quality of an inspection object based on an image of the inspection object acquired by imaging an inspection object such as various parts or products, and is used at a production site such as a factory. Can be used. The entire inspection target may be the inspection target, or only a part of the inspection target may be the inspection target. Further, one inspection object may include a plurality of inspection objects. Further, the inspection target image may include a plurality of inspection target objects. The object to be inspected can also be called a work.

The image inspection device 1 includes a control unit 2 which is a main body of the device, an image pickup unit 3, a display device (display unit) 4, and a personal computer 5. The personal computer 5 is not essential and may be omitted. A personal computer 5 can be used instead of the display device 4 to display various information and images. In FIG. 1, as an example of the configuration of the image inspection device 1, the control unit 2, the imaging unit 3, the display device 4, and the personal computer 5 are described as separate devices, but any plurality of these may be combined. Can also be integrated. For example, the control unit 2 and the imaging unit 3 can be integrated, or the control unit 2 and the display device 4 can be integrated. Further, the control unit 2 can be divided into a plurality of units and a part thereof can be incorporated into the image pickup unit 3 or the display device 4, or the image pickup unit 3 can be divided into a plurality of units and a part thereof can be incorporated into another unit. ..

(Configuration of Imaging Unit 3)
As shown in FIG. 2, the image pickup unit 3 includes a camera module (imaging unit) 14 and an illumination module (illumination unit) 15, and is a unit that executes acquisition of an image to be inspected. The camera module 14 includes an AF motor 141 for driving an imaging optical system and an imaging substrate 142. The AF motor 141 is a portion that automatically performs focus adjustment by driving the lens of the imaging optical system, and can perform focus adjustment by a method such as contrast autofocus, which has been well known in the past. The imaging field of view (imaging area) of the camera module 14 is set to include an inspection object. The image pickup substrate 142 includes a CMOS sensor 143, an FPGA 144, and a DSP 145 as light receiving elements that receive light incident from the image pickup optical system. The CMOS sensor 143 is an imaging sensor configured to be able to acquire a color image. Instead of the CMOS sensor 143, a light receiving element such as a CCD sensor can also be used. The FPGA 144 and the DSP 145 are for executing image processing inside the image pickup unit 3, and the signal output from the CMOS sensor 143 is also input to the FPGA 144 and the DSP 145.

The lighting module 15 includes an LED (light emitting diode) 151 as a light emitting body that illuminates an imaging region including an inspection object, and an LED driver 152 that controls the LED 151. The light emission timing, light emission time, and light emission amount of the LED 151 can be arbitrarily set by the LED driver 152. The LED 151 may be provided integrally with the image pickup unit 3, or may be provided as an external lighting unit separately from the image pickup unit 3. Although not shown, the lighting module 15 is provided with a reflector that reflects the light emitted from the LED 151, a lens through which the light emitted from the LED 151 passes, and the like. The irradiation range of the LED 151 is set so that the light emitted from the LED 151 irradiates the inspection target and the peripheral area of the inspection target. A light emitter other than the light emitting diode can also be used.

(Configuration of control unit 2)
The control unit 2 includes a main board 13, a connector board 16, a communication board 17, and a power supply board 18. The FPGA 131, the DSP 132, and the memory 133 are mounted on the main board 13. The FPGA 131 and DSP 132 constitute a control unit 13A, and a main control unit in which these are integrated can be provided.

The control unit 13A of the main board 13 controls the operation of each connected board and module. For example, the control unit 13A outputs a lighting control signal for controlling the lighting / extinguishing of the LED 151 to the LED driver 152 of the lighting module 15. The LED driver 152 switches the lighting / extinguishing of the LED 151 and adjusts the lighting time according to the lighting control signal from the control unit 13A, and also adjusts the amount of light of the LED 151 and the like.

Further, the control unit 13A outputs an image pickup control signal for controlling the CMOS sensor 143 to the image pickup board 142 of the camera module 14. The CMOS sensor 143 starts imaging in response to the imaging control signal from the control unit 13A, and adjusts the exposure time to an arbitrary time to perform imaging. That is, the image pickup unit 3 takes an image within the visual field range of the CMOS sensor 143 in response to the image pickup control signal output from the control unit 13A, and if there is an inspection object within the visual field range, the image pickup unit 3 takes an image of the inspection object. However, if an object other than the inspection object is within the field of view, it can also be imaged. For example, at the time of setting the image inspection device 1, it is possible to take an image of a non-defective product to which an attribute as a non-defective product is given by the user and an image of a defective product to which the attribute as a defective product is given. When the image inspection device 1 is in operation, the inspection object can be imaged. Further, the CMOS sensor 143 is configured so that a live image, that is, a currently captured image can be output at a short frame rate at any time.

When the imaging by the CMOS sensor 143 is completed, the image signal output from the imaging unit 3 is input to the FPGA 131 of the main board 13, processed by the FPGA 131 and the DSP 132, and stored in the memory 133. The details of the specific processing contents by the control unit 13A of the main board 13 will be described later.

The connector board 16 is a portion that receives power from the outside via a power connector (not shown) provided in the power interface 161. The power supply board 18 is a portion that distributes the electric power received by the connector board 16 to each board, a module, and the like. Specifically, the electric power is distributed to the lighting module 15, the camera module 14, the main board 13, and the communication board 17. To do. The power supply board 18 includes an AF motor driver 181. The AF motor driver 181 supplies drive power to the AF motor 141 of the camera module 14 to realize autofocus. The AF motor driver 181 adjusts the power supplied to the AF motor 141 in response to the AF control signal from the control unit 13A of the main board 13.

The communication board 17 outputs the quality determination signal, image data, user interface, etc. of the inspection object output from the control unit 13A of the main board 13 to the display device 4, the personal computer 5, an external control device (not shown), or the like. To do. The display device 4 and the personal computer 5 have a display panel including, for example, a liquid crystal panel, and image data, a user interface, and the like are displayed on the display panel.

Further, the communication board 17 is configured to be able to receive various operations of the user input from the touch panel 41 of the display device 4, the keyboard 51 of the personal computer 5, and the like. The touch panel 41 of the display device 4 is, for example, a conventionally known touch-type operation panel equipped with a pressure-sensitive sensor, and detects a touch operation by the user and outputs the touch operation to the communication board 17. The personal computer 5 includes a keyboard 51 and a mouse 52 as operation devices. The personal computer 5 may include a touch panel (not shown) as an operation device. The personal computer 5 is configured to be able to receive various operations of the user input from these operation devices. The communication may be wired or wireless, and any communication form can be realized by a conventionally known communication module.

The control unit 2 is provided with a storage device 19 such as a hard disk drive. The storage device 19 stores a program file 80, a setting file, and the like (software) for enabling each control and process described later to be executed by the hardware. The program file 80 and the setting file can be stored in a storage medium 90 such as an optical disk, and the program file 80 and the setting file stored in the storage medium 90 can be installed in the control unit 2. Further, the storage device 19 can store the image data, the quality determination result, and the like.

(Specific configuration of image inspection device 1)
FIG. 3 is a block diagram of the image inspection device 1, and each part shown in FIG. 3 is configured by the control unit 2 in which the program file 80 and the setting file are installed. That is, the image inspection device 1 includes an image input unit 21, a normal inspection setting unit (an example of a normal inspection setting means) 22, a deep learning setting unit (an example of a deep learning setting means) 23, and an inspection execution unit (an inspection execution unit). An example of means) 24, a display control unit 25, a threshold value adjusting unit (an example of a threshold value adjusting means) 26, and an inspection selection unit (an example of an inspection selection means) 27 are provided. A display means is composed of a display control unit 25 and a display device 4. Each of these parts 21 to 27 and means may be composed of only hardware, or may be composed of a combination of hardware and software. For example, the threshold value adjusting unit 26 may include a keyboard 51 and a mouse 52.

Further, although the parts 21 to 27 shown in FIG. 3 are shown as conceptually independent, they may be in a form in which any two or more parts are integrated, and are not limited to the illustrated form. Absent. Further, each of the parts 21 to 27 and the means may be configured by independent hardware, or may be configured so that a plurality of functions are realized by one hardware or software. There may be. Further, the functions and operations of the respective units 21 to 27 and the means shown in FIG. 3 can be realized by the control by the control unit 13A of the main board 13.

The image inspection apparatus 1 is configured to be capable of performing at least two types of inspections, that is, inspection of an inspection object by a normal inspection process and inspection of an inspection object by a deep learning process. The normal inspection process is a general inspection process using an image of an image to be inspected, and is an inspection target based on various feature quantities (color, edge, position, etc.) of the object to be inspected in the image. This is a process for determining the quality of an object. The normal inspection process includes, for example, pattern search, difference inspection, learning inspection, scratch detection, and the like, but inspection processes other than these may be included.

On the other hand, in the deep learning process, at least one of the image to which a plurality of good product attributes are given in advance and the image to which the bad product attribute is given is input to the multi-layer neural network, and the good product image and the defective product image are obtained. It is an inspection process using a trained neural network obtained by adjusting a plurality of parameters in the network so that the image can be identified. The neural network used here has three or more layers, and is a neural network capable of so-called deep learning.

Details will be described later, but if a stable inspection can be performed by only one of the normal inspection process and the deep learning process, only one of the inspection processes can be performed. Further, by using both the normal inspection process and the deep learning process together, for example, an inspection that is difficult to determine by the normal inspection process can be determined with high accuracy by the deep learning process. In addition, unstable behavior peculiar to deep learning processing that may occur when unknown data not used for learning is input can be avoided by inspection processing only for normal inspection processing.

In addition, the image inspection device 1 has a setting mode for setting various parameters such as imaging settings, registering a master image, setting a normal inspection process, setting a deep learning process, and the like, and an inspection in which an inspection object is imaged at an actual site. It is possible to switch to an operation mode (Run mode) in which the quality of the inspection target is determined based on the target image. In the setting mode, it is possible to perform pre-work to enable the user to distinguish between non-defective products and defective products in the desired product inspection, and this work is a neural network learning work. It is included. Switching between the setting mode and the operation mode can be performed on a user interface (not shown), and the operation mode can be automatically switched to the operation mode as soon as the setting mode is completed. The transition from the operation mode to the setting mode can also be performed arbitrarily. It is also possible to relearn the neural network in the operation mode.

(Structure of image input unit 21)
The image input unit 21 shown in FIG. 3 is a portion for inputting a plurality of non-defective product images to which the non-defective product attribute is given and / or a plurality of defective product images to which the defective product attribute is assigned to the deep learning setting unit 23 in the setting mode. At the same time, it is a part for inputting a registered image to the normal inspection setting unit 22. Further, the image input unit 21 is also a part for inputting the newly acquired inspection target image to the inspection execution unit 24 in the operation mode.

Specifically, when the user places the inspection object in the field of view of the CMOS sensor 143 of the image pickup unit 3 in the setting mode, the control unit 13A displays the live image captured by the CMOS sensor 143 as an image input user interface. It is incorporated in a part (not shown), and the image input user interface in which the live image is incorporated is displayed on the display device 4. When the user performs an image capture operation while the inspection object is displayed on the image input user interface, the image displayed on the image input user interface at that time, that is, the image that the user wants to capture is displayed. Captured as a still image. The captured image is stored in the memory 133 or the storage device 19. Examples of the image capture operation by the user include a button operation incorporated in the image input user interface, an operation of the keyboard 51 and the mouse 52, and the like.

When capturing an image, the user can give one of a non-defective product attribute and a defective product attribute. For example, a "non-defective product import button" and a "defective product import button" are provided in the image input user interface, and the "non-defective product import button" is operated when the image displayed on the image input user interface is imported. In that case, the image captured at that time can be captured as a non-defective image with the non-defective attribute, while if the "defective product capture button" is operated, it is captured at that time. The image can be imported as a non-defective product image to which the defective product attribute is added. By repeating this, a plurality of non-defective product images and a plurality of defective product images can be captured. When a plurality of non-defective product images are input, it may be an image obtained by capturing different non-defective products, or may be an image captured a plurality of times by changing the angle or position of one non-defective product. Further, a plurality of non-defective product images and a plurality of defective product images may be generated inside the image inspection device 1, for example, by changing the brightness of the image. For example, about 100 images of non-defective products and images of defective products are prepared. For example, when detecting a scratch, a defective image with a scratch is prepared, but this may be created by the user or may be automatically created by the image inspection device 1. Good.

As an image for deep learning, only a good product image or only a defective product image can be captured. Further, the method of imparting the non-defective product attribute and the defective product attribute to the image is not limited to the above-mentioned method, and may be, for example, a method of imparting the image after capturing the image. It is also possible to configure it so that it is possible to accept modification of the attribute after assigning the non-defective product attribute and the defective product attribute.

The user can import the registered image used in the normal inspection process as the master image. The registered image can be used, for example, when performing a difference inspection for determining quality by detecting a blob (lump) of a difference from a newly acquired image to be inspected. In addition, the registered image can be used when making a pass / fail judgment using the normalized correlation. For example, if a "registered image capture button" is provided in the image input user interface and the "registered image capture button" is operated when the image displayed on the image input user interface is captured, the operation is performed. The image captured at that time can be used as a registered image. After importing the image, the imported image can be set as a registered image.

In the operation mode, the control unit 13A takes an image of the inspection object by the CMOS sensor 143 and captures the inspection target image while the inspection object is in the field of view of the CMOS sensor 143. The signal that triggers the capture of the image to be inspected is well known from the past, and may be, for example, a signal input from the outside of the image inspection device 1 or a signal generated inside the image inspection device 1. There may be.

(Structure of normal inspection setting unit 22)
The normal inspection setting unit 22 sets the normal inspection process by accepting from the user the setting of the feature amount used for the normal inspection and the setting of the threshold value for the normal inspection which is a standard for quality judgment to be compared with the feature amount. Is the part to do. Examples of the feature amount used in the normal inspection include the color of the inspection target, the edge of the inspection target, the position of the inspection target, and the like. The edge information of the inspection object includes the position, shape, length, etc. of the edge. The position of the inspection object includes not only the position of the inspection object itself but also the position of a part of the inspection object. The feature amount used for the usual inspection may be one or two or more.

When setting the feature amount used for the normal inspection, the feature amount setting user interface (not shown) generated by the control unit 13A is displayed on the display device 4, and the user interface is used to display the feature amount setting user interface. Accept operations. The feature amount setting user interface is provided with a feature amount setting unit for inputting and selecting the above-mentioned feature amount, and the user performs an input operation on the feature amount setting unit with the keyboard 51 or the mouse 52. Then, the input operation is accepted by the control unit 13A, and the setting of the feature amount used for the inspection is completed. The set feature amount is stored in the memory 133 or the storage device 19.

In addition, the feature amount set as described above is compared with the threshold value for normal inspection, which is a criterion for determining the quality, and as a result, it is determined whether the image to be inspected is a good product or a defective product, which will be described later. The unit 24 will determine. When setting the normal inspection threshold value used at this time as a reference for pass / fail judgment, the threshold value setting user interface (not shown) generated by the control unit 13A is displayed on the display device 4, and the threshold value setting is performed. Accept user operations on the user interface. The threshold value setting user interface is provided with a threshold value input unit for inputting the above-mentioned normal inspection threshold value. When the user performs a threshold value input operation on the threshold value input unit with the keyboard 51 or the mouse 52, the input operation is accepted by the control unit 13A, and the threshold value input and setting are completed. The set normal inspection threshold value is stored in the memory 133 or the storage device 19. The threshold value for normal inspection may be set automatically by the image inspection apparatus 1 and then adjusted by the user to complete the final input. The threshold value for normal inspection is a threshold value used in the normal inspection process and is not used in the inspection of the deep learning process.

When accepting the normal inspection threshold value from the user, it can be configured to accept the non-defective product confirmation threshold value for confirming the non-defective product determination and the defective product confirmation threshold value for confirming the defective product determination. The non-defective product determination threshold is a threshold value for determining a non-defective product such as a non-defective product if it is above the threshold value or a non-defective product if it is below the threshold value, and is set to a threshold value with high accuracy so that the non-defective product can be determined. be able to. On the other hand, the threshold value for determining defective products is a threshold value for determining defective products such as defective products if it is above the threshold value or defective products if it is below the threshold value, and to the extent that defective products can be determined. It can be set to a highly accurate threshold. Only one of the non-defective product confirmation threshold value and the defective product confirmation threshold value may be accepted, or both may be accepted. When the non-defective product determination threshold value and the defective product determination threshold value are input, they are stored in the memory 133 or the storage device 19 in an identifiable state.

(Structure of deep learning setting unit 23)
The deep learning setting unit 23 inputs a plurality of non-defective product images to which the non-defective product attribute is given and / or a plurality of defective product images to which the defective product attribute is given to the input layer of the neural network and causes the neural network to learn. This is the part that sets the deep learning process that classifies the newly acquired image to be inspected into a non-defective image and a defective image. The neural network can be constructed on the control unit 13A and has at least an input layer, an intermediate layer and an output layer.

Since the non-defective product image and the defective product image are acquired by the image input unit 21, the deep learning setting unit 23 inputs the non-defective product image and the defective product image acquired by the image input unit 21 to the input layer of the neural network. Only the non-defective product image may be input to the input layer of the neural network, only the defective product image may be input, or both the non-defective product image and the defective product image may be input. This may be automatically changed according to the image acquisition status, or may be selectable by the user.

The deep learning setting unit 23 also provides correct answer information (whether the input image is a good product or a defective product) to the neural network, and obtains a plurality of good product images and / or a plurality of defective product images and correct answer information. Let the neural network learn by using it. As a result, a plurality of initial parameters of the neural network are changed to become parameters with a high correct answer rate. The learning of this neural network can also be automatically performed when a good product image or a defective product image is input. By training the neural network, a classifier capable of distinguishing between a non-defective image and a defective image is generated, and the newly acquired image to be inspected can be classified into a non-defective image and a defective image by the classifier. Will be. This classifier can be generated by the classifier generation unit 23a shown in FIG.

The neural network may be an identification type network based on a CNN (Convolutional Neural Network), or may be a restoration type neural network typified by an autoencoder. In the case of an identification type network, a value obtained by normalizing the output value (generally, a normalization function called a softmax function is used) can be used as a threshold value (threshold value for deep learning processing) as a criterion for quality judgment. .. The threshold value for deep learning processing can include a threshold value for determining a non-defective product which is a standard for determining a non-defective product and a threshold value for determining a defective product which is a standard for determining a defective product, and a normalized value can be used for each threshold value. it can. The threshold value for deep learning processing and the threshold value for normal inspection are independent of each other.

In the case of a restoration type neural network, particularly an autoencoder, as an example, it is possible to input non-defective image data in the setting mode and train in advance to restore and output the input data as it is. In the operation mode, the newly acquired image to be inspected is input to the trained neural network to obtain a restored image. The difference between the image input to the neural network and the restored image can be taken, and if the difference is more than a certain value, it is judged as a defective product, and if it is less than a certain value, it is judged as a good product. There are a method of summing up the difference between the gradation values of the image input to the neural network and the restored image, and a method of summing up the number of pixels with a certain difference or more, and any method can be used. Good. As described above, the threshold value for deep learning processing may be determined by using the number of pixels (number of pixels) or the area of the image.

(Structure of inspection execution unit 24)
The inspection execution unit 24 is configured to apply the normal inspection process and the deep learning process to both the newly acquired non-defective product image with the non-defective product attribute and the defective product image with the defective product attribute. ing. The newly acquired non-defective product image to which the non-defective product attribute is added and the newly acquired defective product image to which the defective product attribute is assigned are the setting of the normal inspection process by the normal inspection setting unit 22 and the deep learning setting unit 23. It is an image acquired after the setting of the deep learning process by is completed. The inspection execution unit 24 is configured to be capable of performing both an inspection in the setting mode and an inspection in the operation mode.

The inspection execution unit 24 applies the normal inspection process to the newly acquired non-defective product image to calculate the correct answer rate when the normal inspection process is performed on the non-defective product image, and newly acquires the correct answer rate. By applying the normal inspection process to the defective product image, the correct answer rate when the normal inspection process is performed on the defective product image is calculated. Further, the inspection execution unit 24 applies the deep learning process to the newly acquired good product image to calculate the correct answer rate when the deep learning process is performed on the good product image, and also newly By applying the deep learning process to the defective product image acquired in, the correct answer rate when the deep learning process is performed on the defective product image is calculated.

The correct answer rate is, for example, the certainty (accuracy) of whether or not the product is good. In the case of deep learning processing, based on the normalized output value, it is determined whether the image is close to a good product image or a defective product image. The closer to a good product, the higher the degree of good product, and the closer to a defective product, the better the product. It can be determined that the degree is low. Based on this, it is possible to calculate the correct answer rate when deep learning processing is performed, and for example, as shown on the horizontal axis of FIG. 4, it is possible to display the degree of good quality.

On the other hand, in the case of the normal inspection process, the method of calculating the correct answer rate differs depending on the inspection method. As described above, the inspection methods include pattern search, difference inspection, learning inspection, scratch detection, and the like. In the case of pattern search, the part (pattern) to be inspected is registered in the setting mode, and the registered pattern and the pattern included in the newly acquired image to be inspected are normalized and correlated. Judgment is made using the above, and good / bad judgment is made based on the obtained correlation value. If this correlation value is higher than a certain level, it is determined to be a non-defective product, while if the correlation value is low, it is determined to be a defective product. Therefore, as shown on the vertical axis of FIG. 4, the correct answer rate can be displayed according to the level of the correlation value, and the correct answer rate can be displayed in association with the correlation value. In the case of pattern search, the correlation value can be a threshold. The degree of good quality when deep learning processing is performed may be indicated by the vertical axis, and the correlation value of the pattern search may be indicated by the horizontal axis. The same applies to the following examples.

In the case of the difference inspection, an image is registered in advance in the setting mode, and a pass / fail judgment is made by detecting the difference between the registered image and the newly acquired inspection target image thereafter. If the difference blob area is smaller than a certain level, it is judged as a non-defective product, while if the difference blob area is large, it is judged as a defective product. Therefore, as shown on the vertical axis of FIG. 5, the correct answer rate can be displayed according to the magnitude of the difference blob area, and the correct answer rate can be displayed in association with the difference blob area. In the case of difference inspection, the blob area of the difference can be the threshold.

The learning test is, for example, a method described in Japanese Patent No. 5767963, which automatically defines a range of non-defective products by learning a plurality of non-defective product images and detects “non-defective products”. This is a method of detecting defects based on the sum of gradation values (defect amount) deviating from the pixel variation range based on statistics and the area such as the number of pixels (area of the defect portion).

In the case of learning inspection, if the amount of defects or the area of the defective part is smaller than a certain level, it is judged as a non-defective product, while if the amount of defects or the area of the defective part is large, it is judged as a defective product. Therefore, as shown on the vertical axis of FIG. 6, the correct answer rate can be displayed according to the amount of defects and the size of the area of the defective portion. In the case of a learning test, the amount of defects and the area of the defective portion can be threshold values.

Scratch detection is a method described in Japanese Patent No. 4544578, for example, an inspection area is determined in an image, an N × N segment is set with respect to pixels included in this inspection area, and the segment is shifted in a predetermined direction. A method of calculating the average density of each pixel contained in each segment and calculating the difference between the average density of one segment and the average density of other segments adjacent to it as the defect level. Can be adopted. As a result, it is possible to detect scratches on the inspection object.

In the case of scratch detection, if the defect level or the area of the defective portion is smaller than a certain level, it is determined to be a non-defective product, while if the defect level or the area of the defective portion is large, it is determined to be a defective product. Therefore, as shown on the vertical axis of FIG. 7, the correct answer rate can be displayed according to the amount of defects and the size of the area of the defective portion. In the case of scratch detection, the defect level or the area of the defect portion can be a threshold value.

(Configuration of display control unit 25)
By reading each correct answer rate calculated by the inspection execution unit 24, the display control unit 25 determines the correct answer rate by the normal inspection process executed by the inspection execution unit 24 and the deep learning process executed by the inspection execution unit 24. The correct answer rate acquisition unit 25a for acquiring the correct answer rate is provided. The display control unit 25 causes the display device 4 to display the correct answer rate by the normal inspection process executed by the inspection execution unit 24 and the correct answer rate by the deep learning process executed by the inspection execution unit 24 in a comparable form. Is configured to allow

Specifically, the display control unit 25 generates a first user interface 60 to be displayed when the pattern search and the deep learning process are combined as shown in FIG. The first user interface 60 is provided with a graph display area 60a. The vertical axis of the graph displayed in the graph display area 60a indicates the level of the correlation value obtained by the pattern search as the normal inspection process, and the higher the value, the higher the correlation value, that is, the higher the degree of non-defectiveness. Is set to. That is, the correct answer rate by the normal inspection process can be displayed in association with the correlation value of the pattern search.

The horizontal axis of the graph displayed in the graph display area 60a indicates the high and low degree of good quality obtained by deep learning processing, and the degree of good quality is set to be higher toward the right, that is, the degree of good quality is higher. There is. In the graph, "○" indicates a non-defective product image, and "x" indicates a defective product image. In this way, by simultaneously displaying the correct answer rate by the normal inspection process and the correct answer rate by the deep learning process on the same graph, the two can be easily compared. The display form of the correct answer rate by the normal inspection process and the correct answer rate by the deep learning process is not limited to the graph shown in FIG. 4, and may be a display form in which both can be compared by numerical values, for example. It may be in another display form.

Further, in the case of the difference inspection, the display control unit 25 generates a second user interface 61 to be displayed when the difference inspection and the deep learning process are combined as shown in FIG. The second user interface 61 is provided with a graph display area 61a similar to that of the first user interface 60. The vertical axis of the graph displayed in the graph display area 61a indicates the magnitude of the difference blob area obtained by the difference inspection as a normal inspection process, and the difference blob area becomes smaller as it goes up, that is, the degree of good quality. Is set to be high. That is, the correct answer rate by the normal inspection process can be displayed in association with the blob area of the difference. The horizontal axis of the graph displayed in the graph display area 61a is the same as that of the first user interface 60.

Further, in the case of the learning test, the display control unit 25 generates a third user interface 62 to be displayed when the learning test and the deep learning process are combined as shown in FIG. The third user interface 62 is provided with a graph display area 62a similar to that of the first user interface 60. The vertical axis of the graph displayed in the graph display area 62a indicates the magnitude of the defect amount obtained by the learning inspection as the normal inspection process, and the defect amount becomes smaller as it goes up, that is, the degree of good quality increases. Is set to. The horizontal axis of the graph displayed in the graph display area 62a is the same as that of the first user interface 60.

Further, in the case of scratch detection, the display control unit 25 generates a fourth user interface 63 to be displayed when the scratch detection and the deep learning process are combined as shown in FIG. 7. The fourth user interface 63 is provided with a graph display area 63a similar to that of the first user interface 60. The vertical axis of the graph displayed in the graph display area 63a indicates the magnitude of the defect level obtained by scratch detection as a normal inspection process, and the defect level becomes smaller as it goes up, that is, the degree of good quality increases. Is set to. The horizontal axis of the graph displayed in the graph display area 63a is the same as that of the first user interface 60.

(Structure of threshold value adjusting unit 26)
A threshold display line 60b for normal inspection extending parallel to the horizontal axis is displayed on the first user interface 60 shown in FIG. 4, and the determination result of the normal inspection process is displayed together with the threshold value in the normal inspection process. The normal inspection threshold display line 60b is a line indicating a normal inspection threshold. Instead of the normal inspection threshold value display line 60b, or together with the normal inspection threshold value display line 60b, a display form capable of showing the normal inspection threshold value may be adopted, for example, the background color may be painted separately or the threshold value may be set. A form or the like displayed by a numerical value can also be used.

The user can move the normal inspection threshold value display line 60b in the vertical direction (in the direction of increasing / decreasing the normal inspection threshold value) by using the keyboard 51, the mouse 52, or the like. The correlation value that becomes the normal inspection threshold increases or decreases according to the position of the normal inspection threshold display line 60b, and the correlation value increases when the normal inspection threshold display line 60b is moved upward, while the normal inspection threshold display line increases. Moving 60b down reduces the correlation value. The threshold value adjusting unit 26 changes the normal inspection threshold value by detecting the moving state of the normal inspection threshold value display line 60b by the user. The method for adjusting the threshold value for normal inspection is not limited to the above-mentioned method, and may be adjusted by inputting a numerical value, for example.

In the second user interface 61 shown in FIG. 5, a threshold display line 61b for normal inspection similar to that of the first user interface 60 is displayed. When the user moves the normal inspection threshold display line 61b in the vertical direction, the blob area value of the difference that becomes the normal inspection threshold increases or decreases, and when the normal inspection threshold display line 60b is moved upward, the difference becomes While the blob area value decreases, moving the normal inspection threshold display line 60b downward increases the difference blob area value.

In the third user interface 62 shown in FIG. 6, a threshold display line 62b for normal inspection similar to that of the first user interface 60 is displayed. By moving the normal inspection threshold display line 62b in the vertical direction, the user increases or decreases the amount of defects that become the normal inspection threshold, and moves the normal inspection threshold display line 60b upward to decrease the amount of defects. On the other hand, when the threshold display line 60b for normal inspection is moved downward, the amount of defects increases.

In the fourth user interface 63 shown in FIG. 7, a threshold display line 63b for normal inspection similar to that of the first user interface 60 is displayed. By moving the normal inspection threshold display line 63b in the vertical direction, the user increases or decreases the defect level that becomes the normal inspection threshold, and moves the normal inspection threshold display line 60b upward to decrease the defect level. On the other hand, if the threshold display line 60b for normal inspection is moved downward, the defect level becomes large.

(Combination of normal inspection processing and deep learning processing)
In this embodiment, the inspection execution unit 24 shown in FIG. 3 is configured to be able to perform an inspection by combining a normal inspection process and a deep learning process. Below, there are cases where defective products are judged by the normal inspection process and the rest is judged as good or bad by the deep learning process, cases where good products are judged by the normal inspection process and the rest are judged by the deep learning process, and cases where the normal inspection process is used. The procedure of three cases of confirming the non-defective product determination and the defective product determination and determining the quality of the rest by the deep learning process will be described.
1. 1. When a defective product is judged by the normal inspection process and the rest is judged as good or bad by the deep learning process. FIG. 8 is a flowchart showing a procedure when a defective product is judged by the normal inspection process and the rest is judged as good or bad by the deep learning process. is there. This is a processing procedure that can be applied when the pattern search and the deep learning process shown in FIG. 4 are combined and when the difference inspection and the deep learning process shown in FIG. 5 are combined.

After the start of the flowchart shown in FIG. 8, in step SA1, an image of the inspection target is taken. This can be done by the control unit 13A controlling the CMOS sensor 143, specifically, the image input unit 21 shown in FIG.

In step SA2, the inspection execution unit 24 inspects the inspection target image acquired in step SA1 by the normal inspection process. In step SA3, as a result of the inspection by the normal inspection process performed in step SA2, it is determined whether or not the image is clearly defective. This can be determined based on the threshold value for normal inspection. For example, in the case of the pattern search shown in FIG. 4, the correlation value corresponding to the threshold value for normal inspection (indicated by the threshold display line 60b for normal inspection) and step SA2 When the correlation value obtained in step SA2 is equal to or less than the normal inspection threshold value, it is clearly determined that the image is defective. Similarly, in the case of the difference inspection shown in FIG. 5, it can be determined by the size of the blob area.

The normal inspection threshold value used in step SA3 is a threshold value for determining the defective product determination. Therefore, in step SA3, the defective product determination is determined for the inspection target image having a feature amount capable of determining the defective product, while the defective product determination is performed for the inspection target image for which the defective product determination cannot be determined. Try to proceed to the next step without confirming.

If YES is determined in step SA3, the process proceeds to step SA7. If YES is determined in step SA3, the defective product determination is confirmed. Therefore, it is clearly a defective product image, and in this case, the defective product image is finally obtained in step SA7. That is, for an image that can be clearly determined to be a defective image, the determination result can be determined without performing deep learning processing, so that a high processing speed is maintained.

On the other hand, if NO is determined in step SA3, the process proceeds to step SA4. If NO is determined in step SA3, it means that the defective product determination cannot be confirmed, and it is not clear whether the image is a defective product or a non-defective product. In this case, in step SA4, an inspection by a deep learning process having a higher discriminating ability than the normal inspection process is executed. Since the image on which the deep learning process is executed does not include an image that is clearly determined to be a defective image, it is unknown that an unexpected defect peculiar to the deep learning process is mixed. The instability of the behavior when data is input is eliminated, and the determination accuracy can be improved.

After that, in step SA5, it is determined whether or not the degree of good quality obtained by the deep learning process exceeds the threshold value for the deep learning process. If NO is determined in step SA5, it means that the image is determined to be a defective product by the deep learning process having high discriminating ability, and the process proceeds to step SA7 to confirm the determination that the image is defective. On the other hand, if NO is determined in step SA5, it is a good product image, so the determination that it is a good product image is confirmed in step SA6.

By going through such a procedure, the effect shown in FIG. 9 can be obtained. The figure shown on the upper side of FIG. 9 is a histogram of the output values when the quality is judged only by the deep learning process. When the quality is determined only by the deep learning process, there is a region A in which the defective image and the defective image are mixed. That is, in the deep learning process, due to the unstable behavior described above, the image that is clearly determined to be a defective image is determined to be a non-defective image, and the defective image and the defective image are separated. I can't. On the other hand, by going through the procedure of the flowchart shown in FIG. 8, the image that is clearly determined to be a defective image is not subjected to deep learning processing, so that it is not a defective image as shown in the lower part of FIG. It becomes possible to separate it from a non-defective image.

That is, as shown in the flowchart of FIG. 8, depending on the result of the normal inspection process of the image to be inspected, the inspection is completed only by the normal inspection process, or the inspection process is performed by combining the normal inspection process and the deep learning process. Can be selected. This is performed by the inspection selection unit 27 shown in FIG. For example, in the case of pattern search, if the correlation value obtained in the normal inspection process is equal to or less than the threshold value (predetermined correlation value) used in step SA3, it can be determined that the product is defective in the normal inspection process. The inspection selection unit 27 ends the inspection only by the normal inspection process. On the other hand, when the correlation value obtained by the normal inspection process is higher than the predetermined correlation value, the inspection selection unit 27 selects the deep learning process.

Further, in the case of the difference inspection, the procedure is the same as that of the flowchart shown in FIG. 8, and when the blob area of the difference obtained in the normal inspection process is equal to or larger than the predetermined area value, it is considered as a defective product in the normal inspection process. Since the determination can be made, the inspection selection unit 27 ends the inspection only by the normal inspection process. On the other hand, when the blob area of the difference obtained by the normal inspection process is smaller than the predetermined area value, the inspection selection unit 27 selects the deep learning process.

2. 2. When the non-defective product is judged by the normal inspection process and the rest is judged by the deep learning process. FIG. 10 is a flowchart showing a procedure when the non-defective product is judged by the normal inspection process and the rest is judged by the deep learning process. This is a processing procedure that can be applied when the learning test shown in FIG. 6 and the deep learning process are combined, and when the scratch detection and the deep learning process shown in FIG. 7 are combined.

After the start of the flowchart shown in FIG. 10, steps SB1 and SB2 are the same as steps SA1 and SA2 of the flowchart shown in FIG. In step SB3, as a result of the inspection by the normal inspection process performed in step SB2, it is determined whether or not the image is clearly a good product. This can be determined based on the threshold value for normal inspection. For example, in the case of the learning test shown in FIG. 6, the defect amount corresponding to the threshold value for normal inspection (indicated by the threshold display line 62b for normal inspection) and step SB2 When the defect amount obtained in step SB2 is equal to or less than the normal inspection threshold value, it is clearly determined that the image is a good product. In the case of scratch detection shown in FIG. 7, the same determination can be made.

The normal inspection threshold value used in step SB3 is a threshold value for confirming the non-defective product determination. Therefore, in step SB3, the non-defective product determination is determined for the inspection target image having a feature amount capable of determining the non-defective product, but the non-defective product determination is not determined for the inspection target image for which the non-defective product determination cannot be determined. , Go to the next step.

If YES is determined in step SB3, the process proceeds to step SB7. The fact that YES is determined in step SB3 confirms that the product is non-defective. Therefore, it is clearly a good product image, and in this case, the good product image is finally obtained in step SB7. That is, for an image that can be determined to be a clearly non-defective image with a small amount of fluctuation, the determination result can be determined without performing deep learning processing, so that a high processing speed is maintained.

On the other hand, if NO is determined in step SB3, the process proceeds to step SB4. If NO is determined in step SB3, it means that the non-defective product determination cannot be confirmed, and it is not clear whether the image is a defective product or a non-defective product image. In this case, in step SB4, an inspection by a deep learning process having a higher discriminating ability than the normal inspection process is executed. Since the image to which the deep learning process is executed does not include an image that is clearly determined to be a good product image, the throughput can be maintained in a high state.

After that, in step SB5, it is determined whether or not the degree of good quality obtained by the deep learning process exceeds the threshold value for the deep learning process. If YES is determined in step SB5, it means that the image is determined to be a good product by the deep learning process having high discriminating ability, and the process proceeds to step SB7 to confirm the determination that the image is a good product. On the other hand, if NO is determined in step SB5, it is a defective product image, so the determination that it is a defective product image is confirmed in step SB6.

Considering that most of the work is generally non-defective (for example, 99% or more), it is difficult to judge what is clearly non-defective by the existing processing that can process at high speed first. By applying the deep learning process only to, it is possible to maintain the overall throughput at a high speed while using the deep learning process having high discriminating ability.
3. 3. When the non-defective product judgment and the defective product judgment are confirmed by the normal inspection process and the rest is judged as good or bad by the deep learning process. It is a flowchart which shows the procedure in the case of doing. In this procedure, when the non-defective product image and the defective product image have a distribution as in the first distribution example shown in FIGS. 12 and 14, and a distribution as in the second distribution example shown in FIGS. 13 and 15. Can be applied to.

After the start of the flowchart shown in FIG. 11, steps SC1 and SC2 are the same as steps SA1 and SA2 of the flowchart shown in FIG. Further, step SC3 of the flowchart shown in FIG. 11 is the same as step SA3 of the flowchart shown in FIG. 8, and step SC8 of the flowchart shown in FIG. 11 is the same as step SA7 of the flowchart shown in FIG.

In step SC3, for example, in the case of the learning inspection shown in FIG. 12, the defect amount corresponding to the normal inspection threshold value (indicated by the normal inspection threshold value display line 62c) is compared with the defect amount obtained in step SC2, and step SC2 If the amount of defects obtained in 1 is larger than the threshold value for normal inspection and the difference is larger than a certain level, it is clearly determined that the image is defective. The same determination can be made in the case of the distribution state shown in FIG. 13, and the same determination can be made in the case of the scratch detection shown in FIGS. 14 and 15.

The normal inspection threshold value used in step SC3 is a threshold value for determining the defective product determination. Therefore, in step SC3, the defective product determination is determined for the inspection target image having a feature amount capable of determining the defective product, while the defective product determination is performed for the inspection target image for which the defective product determination cannot be determined. Try to proceed to the next step without confirming.

If NO is determined in step SC3, it is a non-defective image or an image that cannot be clearly determined to be a defective image. In this case, the process proceeds to step SC4. In step SC4, as a result of the inspection by the normal inspection process performed in step SC2, it is determined whether or not the image is clearly a good product.

In step SC4, for example, in the case of a learning inspection, the defect amount corresponding to the normal inspection threshold value (normal inspection threshold value display line 62b shown in FIG. 12) is compared with the defect amount obtained in step SC2, and is obtained in step SC2. If the amount of defects is larger than the normal inspection threshold value and the difference is larger than a certain level, it is clearly determined that the image is a good product. The same determination can be made in the case of the distribution state shown in FIG. 13, and the same determination can be made in the case of the scratch detection shown in FIGS. 14 and 15.

The normal inspection threshold value used in step SC4 is a threshold value for confirming the non-defective product determination. Therefore, in step SC4, the non-defective product determination is determined for the inspection target image having a feature amount capable of determining the non-defective product, but the non-defective product determination is not determined for the inspection target image for which the non-defective product determination cannot be determined. , Go to the next step.

If YES is determined in step SC4, the process proceeds to step SC7. If YES is determined in step SC4, it means that the image is clearly a good product, and in this case, the determination that the image is a good product is confirmed in step SC7. That is, for an image that can be determined to be a clearly non-defective image with a small amount of fluctuation, the determination result can be determined without performing deep learning processing, so that a high processing speed is maintained.

On the other hand, if NO is determined in step SC4, the process proceeds to step SC5. The fact that NO is determined in step SC4 is an image that cannot be clearly determined to be a non-defective image or an image that cannot be clearly determined to be a defective image. In this case, in step SC5, it is more than the normal inspection process. Perform an inspection by deep learning processing with high discriminating ability.

After that, in step SC6, it is determined whether or not the degree of good quality obtained by the deep learning process exceeds the threshold value for the deep learning process. If YES is determined in step SC6, it means that the image is determined to be a good product by the deep learning process having high discriminating ability, and the process proceeds to step SC7 to confirm the determination that the image is a good product. On the other hand, if NO is determined in step SC6, it is a defective image, so the determination that it is a defective image is confirmed in step SC8. By going through such a procedure, it is possible to judge what is clearly good by the existing processing that can be processed at high speed first, and apply deep learning processing only to those that are difficult to judge, so that the discrimination ability is high. While using deep learning processing, the overall throughput can be maintained at a high speed. In addition, the effects shown in FIG. 9 can be obtained at the same time.

(Structure of inspection selection unit 27)
As shown in the flowchart described above, the inspection selection unit 27 is configured to automatically select only the normal inspection process or the combined use of the normal inspection process and the deep learning process by a determination based on a certain threshold value. The present invention is not limited to this, and for example, the user can select any process from the normal inspection process only, the deep learning process only, and the combination of the normal inspection process and the deep learning process. For example, it is also possible to display the inspection process selection user interface on the display device 4 so that the user can accept an arbitrary inspection process selection operation.

When selecting the inspection process, by displaying the user interfaces 60 to 63 shown in FIGS. 4 to 7 on the display device 4, the correct answer rate of the normal inspection process and the correct answer rate of the deep learning process are compared. It is possible to select an inspection process that is effective for classification.

(Example of user interface)
The user interface is not limited to the one described above, and may be another user interface. The user interface 70 shown in FIG. 16 includes a graph display area 70a similar to the graph display area 60a of the first user interface 60, and a first histogram display area 70b for displaying a cumulative histogram of output values in deep learning processing. A second histogram display area 70c for displaying a cumulative histogram of output values in the normal inspection process is provided. By providing the first histogram display area 70b and the second histogram display area 70c, it is possible to compare which process can separate the good product image and the defective product image, and select the one with the better degree of separation. can do.

Further, a first separated state display area 70d is provided in the vicinity of the first histogram display area 70b, and a second separated state display area 70e is provided in the vicinity of the second histogram display area 70c. In the first separated state display area 70d, the separated state of the good product image and the defective product image of the cumulative histogram displayed in the first histogram display area 70b, that is, whether or not the good product image and the defective product image can be separated. Is displayed. Similarly, in the second separated state display area 70e, the separated state of the good product image and the defective product image of the cumulative histogram displayed in the second histogram display area 70c is displayed. In the example shown in FIG. 16, it can be seen that the non-defective product image and the defective product image cannot be separated by the normal inspection process, but the non-defective product image and the defective product image can be separated by the deep learning process.

In addition, when a product that is clearly defective in the normal inspection process is determined to be defective and the rest is inspected by the deep learning process, deep learning is performed according to the threshold value for determining the product as clearly defective in the normal inspection process. By updating the cumulative histogram of the output values in the processing, it is possible to adjust the threshold value for separating the non-defective product image and the defective product image.

For example, in the distribution example shown in FIG. 17, when looking at the cumulative histograms displayed in the first histogram display area 70b and the second histogram display area 70c, a good product image and a defective product image are seen in both the normal inspection process and the deep learning process. It turns out that and cannot be separated.

In such a case, as shown in FIG. 18, the normal inspection threshold display line 70f for displaying the threshold value of the normal inspection process is displayed on the user interface 70, and the normal inspection threshold display line 70f is displayed on the keyboard 51 or the mouse. By moving the image in the vertical direction by 52 or the like, the non-defective product image and the defective product image can be separated by both the normal inspection process and the deep learning process. By moving the threshold display line 70f for normal inspection in the vertical direction, the range shown by the diagonal line in FIG. 18, that is, the range to be inspected by the deep learning process can be changed. By changing the range to be inspected by the deep learning process, the cumulative histogram displayed in the second histogram display area 70c is updated. The user can adjust the normal inspection threshold value while looking at the cumulative histogram displayed in the second histogram display area 70c, and set the non-defective product image and the defective product image to a separable threshold value.

When the cumulative histogram is as shown in FIG. 19, the number of images belonging to the range B of the distribution of only defective products is defined as the "number of defective products", and the number of images belonging to the range C of the distribution of only non-defective products is defined as "the number of defective products". The number of non-defective products can be defined as the number of images belonging to the range D of the distribution in which defective products and non-defective products coexist. By displaying the numerical values in the ranges B to D on the display device 4, the numerical values in the ranges B to D can be compared, and thereby, the one having a good degree of separation can be selected.

For example, in the example shown in FIG. 20, the user interface 70 is provided with a first numerical value display area 70g and a second numerical value display area 70h. In the first numerical display area 70g, the number of defective products, the number of non-defective products, and the number of unknown products calculated based on the output value in the deep learning process are displayed. In the second numerical value display area 70h, the number of defective products, the number of non-defective products, and the number of unknown products calculated based on the output value in the normal inspection process are displayed. As shown in this figure, can the good product image and the defective product image be separated by the normal inspection process and the deep learning process by looking at the numerical values displayed in the first numerical value display area 70g and the second numerical value display area 70h? You can know whether or not.

In addition, when a product that is clearly defective in the normal inspection process is determined to be defective and the remaining products are inspected by deep learning processing, depending on the threshold value for determining that the product is clearly defective in the normal inspection process, By updating the "numerical display of the degree of separation", the threshold value for separating the non-defective product image and the defective product image can be adjusted.

For example, in the distribution example shown in FIG. 21, the non-defective product image and the defective product image cannot be separated by both the normal inspection process and the deep learning process.

In such a case, as shown in FIG. 22, the normal inspection threshold display line 70f for displaying the threshold value of the normal inspection process is displayed on the user interface 70, and the normal inspection threshold display line 70f is displayed on the keyboard 51 or the mouse. By moving the image in the vertical direction by 52 or the like, the non-defective product image and the defective product image can be separated by both the normal inspection process and the deep learning process. When the threshold display line 70f for normal inspection is moved in the vertical direction, the numerical display of the degree of separation in the deep learning process with respect to the range shown by the diagonal line downward to the left in FIG. 22, that is, the range to be inspected in the deep learning process is updated. At the same time, the numerical display of the degree of separation in the normal inspection process with respect to the range shown by the diagonal line downward to the right in the figure, that is, the range inspected in the normal inspection process is updated. The user adjusts the normal inspection threshold value while observing the numerical values displayed in the first numerical value display area 70g and the second numerical value display area 70h, and sets the non-defective product image and the defective product image to separable threshold values. Can be done.

(Action and effect of the embodiment)
As described above, according to the image inspection apparatus 1 according to this embodiment, the normal inspection process and the deep learning process are set in the setting mode, and then the normal inspection process is performed on the newly acquired non-defective image and the defective image. The correct answer rate when the above is applied and the correct answer rate when the deep learning process is applied to the newly acquired non-defective product image and defective product image can be calculated and displayed on the display device 4. Since the correct answer rate when the normal inspection process is applied and the correct answer rate when the deep learning process is applied are displayed on the display device 4 in a form that can be compared, the user can use the normal inspection process and the normal test process. Which of the deep learning processes is suitable for the inspection of the inspection object can be easily determined based on the display content of the display device 4. In addition, it becomes possible to easily determine whether or not a process that combines a normal inspection process and a deep learning process is suitable for inspection of an inspection object.

Based on the comparison result of the correct answer rate, it is possible to select either one of the normal inspection process and the deep learning process, or an inspection process combining these processes. For example, when a sufficiently stable inspection is possible only by the normal inspection process, selecting the normal inspection process eliminates the unstable behavior peculiar to the deep learning process and shortens the processing time. Will be done. On the other hand, in the case of an inspection object that is difficult to handle only by the normal inspection process, the inspection accuracy is improved by selecting the deep learning process.

Further, since the user can adjust the threshold value for normal inspection while observing the determination result of the normal inspection process displayed on the display device 4, the non-defective product and the defective product can be appropriately separated. Further, since the degree of separation between the non-defective product and the defective product can be confirmed by the cumulative histogram or the numerical value, the separated state between the non-defective product and the defective product can be easily grasped.

In addition, after the setting mode, when the normal inspection process is applied to the newly acquired inspection target image, the feature amount in the inspection target image and the threshold value for determining the non-defective product judgment can be used to determine the non-defective product. It is possible to confirm the non-defective product judgment for the inspection target image having a quantity. Further, it is possible to determine the defective product determination for the inspection target image having the feature amount capable of determining the defective product based on the feature amount in the inspection target image and the threshold value for determining the defective product determination.

Therefore, by inspecting an inspection object that can be clearly determined to be a non-defective product or an inspection object that can be clearly determined to be a defective product by a normal inspection process having a high processing speed, the throughput is greatly improved. Since only the remaining few inspection objects are inspected by the deep learning process, it is possible to improve the inspection accuracy while suppressing the decrease in the processing speed.

The above embodiments are merely exemplary in all respects and should not be construed in a limited way. Furthermore, all modifications and modifications that fall within the equivalent scope of the claims are within the scope of the present invention.

As described above, the image inspection apparatus according to the present invention can be used when determining the quality of an inspection object based on the image of the inspection object acquired by imaging the inspection object.

1 Image inspection device 2 Control unit 4 Display device 13A Control unit 14 Camera module 15 Lighting module 21 Image input unit 22 Normal inspection setting unit 23 Deep learning setting unit 24 Inspection execution unit 25 Display control unit 26 Threshold adjustment unit 27 Inspection selection unit

Claims (10)

In an image inspection device that determines the quality of an inspection object based on the image of the inspection object acquired by imaging the inspection object.
A normal inspection setting means for setting a normal inspection process by accepting from a user the setting of a feature amount used for inspection and the setting of a threshold value as a reference for quality judgment to be compared with the feature amount.
A plurality of non-defective product images with non-defective product attributes and / or a plurality of defective product images with defective product attributes are input to the input layer of the neural network and trained by the neural network, and the newly input inspection target is input. Deep learning setting means for setting deep learning processing to classify images into non-defective products and defective products,
An inspection execution unit that applies the normal inspection process and the deep learning process to both the newly acquired non-defective image and the defective image.
A display means for displaying the correct answer rate by the normal inspection process and the correct answer rate by the deep learning process executed by the inspection execution unit.
An image inspection apparatus comprising the inspection selection means capable of selecting any of the normal inspection processing and the deep learning processing, or an inspection processing combining these processes. In the image inspection apparatus according to claim 1,
The display means is an image inspection apparatus characterized in that the correct answer rate by the normal inspection process and the correct answer rate by the deep learning process are simultaneously displayed on the same graph. In the image inspection apparatus according to claim 1 or 2.
In the normal inspection process, a pass / fail judgment is made using a normalized correlation.
The display means is an image inspection apparatus characterized in that the correct answer rate by the normal inspection process is displayed in association with a correlation value. In the image inspection apparatus according to claim 3,
An image inspection apparatus characterized in that when the correlation value obtained in the normal inspection process is equal to or less than a predetermined correlation value, it is determined to be a defective product in the normal inspection process. In the image inspection apparatus according to claim 4,
The image inspection apparatus is characterized in that the inspection selection means is configured to select the deep learning process when the correlation value obtained in the normal inspection process is higher than a predetermined correlation value. In the image inspection apparatus according to claim 1 or 2.
In the normal inspection process, pass / fail judgment is performed by a difference inspection that detects the difference between the pre-registered registered image and the newly acquired inspection target image by blob detection.
The display means is an image inspection apparatus characterized in that the correct answer rate by the normal inspection process is displayed in association with the blob area of the difference. In the image inspection apparatus according to claim 6,
An image inspection apparatus characterized in that when the blob area of the difference obtained in the normal inspection process is equal to or larger than a predetermined area value, it is determined to be a defective product in the normal inspection process. In the image inspection apparatus according to claim 7,
The image inspection apparatus is characterized in that the inspection selection means is configured to select the deep learning process when the blob area of the difference obtained in the normal inspection process is smaller than a predetermined area value. In the image inspection apparatus according to any one of claims 1 to 8.
The normal inspection setting means is configured to be able to receive from the user a threshold value for determining a non-defective product determination or a threshold value for determining a defective product determination by being compared with the feature amount.
In the normal inspection process, for an inspection target image having a feature amount capable of determining a non-defective product or a defective product based on the feature amount and the threshold for determining the non-defective product determination or the threshold for determining the defective product determination. The image is characterized in that the deep learning process is applied to an inspection target image having a feature amount in which the non-defective product determination or the defective product determination is confirmed and the non-defective product determination or the defective product determination cannot be determined. Inspection equipment. In the image inspection apparatus according to claim 9,
In the normal inspection process, for an inspection target image having a feature amount capable of determining a non-defective product based on the feature amount and the threshold value, the non-defective product determination is confirmed and the feature amount capable of determining a defective product is provided. For the inspection target image, the defective product determination is confirmed, and the deep learning process is performed only for the inspection target image having a feature amount for which the non-defective product determination cannot be determined and the inspection target image having a feature amount for which the defective product determination cannot be determined. An image inspection device characterized in that it is configured to apply.

Images