JP2023077058A - Appearance inspection device and appearance inspection method - Google Patents

Abstract

To reduce a time required to learn a defective product by naturally creating a defective product image for learning, which has various defective states that may be actually assumed in a workpiece.SOLUTION: A new defective product image is generated by performing an extraction process for extracting a defective point of a defective product image, a pasting process for pasting a defective point, extracted in the extraction process, to a reference image, and a process for making a boundary between the defective point, pasted in the pasting process, and the reference image inconspicuous. An inference model is generated by causing a machine learning network to learn a good product image and defective product image stored in a storage unit in advance and a new defective product image.SELECTED DRAWING: Figure 4

Description

The present invention relates to an appearance inspection apparatus and an appearance inspection method for inspecting the appearance of a work.

For example, Patent Literature 1 discloses a processing device that uses computer-based machine learning to determine whether a work is good or bad. The processing device of Patent Document 1 performs supervised machine learning on non-defective product data to generate a non-defective product learning model, and performs supervised machine learning on defective product data to generate a defective product learning model. After generation, the data of the work to be judged is input, and it is configured to be able to judge whether the work is a good product or a defective product by the non-defective product learning model and the defective product learning model. is also called a work visual inspection device.

JP 2019-204321 A

By the way, when performing supervised learning on defective product data, for example, tens or hundreds of defective product images are required. However, it takes a long time to collect such a large number of images of defective products because defective products are rarely produced at the work production site. Therefore, learning of non-defective products using non-defective product data is assumed as a solution. I want to learn defective products because the performance is inferior compared to other models.

In contrast to this, if it is possible to create defective product images for learning inside the visual inspection apparatus, a large number of defective product images can be acquired in a relatively short period of time. However, even if it is said that defective product images are created inside the appearance inspection device, for example, it is limited to image contrast conversion, simple rotation, and simple deformation that changes the aspect ratio, etc., and it is difficult to create a wide range of defective product images. Met. Specifically, for example, assuming the case of generating an inference model that detects "cracks" in a workpiece, the shape, position, and size of the "crack" in the actual workpiece are assumed to vary, so a wide range of defective products An image is required, but it is difficult to deal with an image whose contrast is simply converted, a rotated image, an image whose aspect ratio is changed, or the like.

The present disclosure is made in view of this point, and the purpose thereof is to naturally generate defective product images for learning having various defect states that are actually assumed in the work, and to perform defective product learning. to shorten the time required for

In order to achieve the above object, in one aspect of the present disclosure, a visual inspection apparatus inputs a workpiece image obtained by photographing a workpiece to be inspected into a machine learning network, and determines the quality of the workpiece based on the input workpiece image. can be assumed. The visual inspection apparatus includes a storage unit for storing a non-defective product image corresponding to a non-defective product and a defective product image corresponding to a defective product; a pasting process of pasting the defective portion extracted by the extracting process onto a reference image including the workpiece; and a process of making the boundary between the defective portion pasted by the pasting process and the reference image inconspicuous. machine learning of the image processing unit that generates a new defective product image, the non-defective product image and the defective product image that are pre-stored in the storage unit, and the new defective product image that is generated by the image processing unit. and a learning unit that trains the network to generate an inference model. Furthermore, the visual inspection apparatus also includes an inspection unit that inputs a workpiece image obtained by photographing a workpiece to be inspected into the inference model generated by the learning unit, and determines whether the workpiece is acceptable based on the input workpiece image. there is

According to this configuration, when the defective portion of the defective product image is extracted, the extracted defective portion is attached to the reference image. The reference image to which the defective portion is pasted may be a non-defective product image or a defective product image. As a result, a new defective product image having a defective portion is generated. In the newly generated defective product image, the boundary between the pasted defective portion and the reference image is not conspicuous, so the defective product image becomes a natural defective product image. By repeating this process, it is possible to acquire a plurality of defective product images for learning having various defect states actually assumed in the workpiece.

By having a machine learning network learn a new defective product image and non-defective product images and defective product images stored in advance in the storage unit, an inference model with a high ability to detect defective products is generated. Therefore, appearance inspection performance can be improved.

The image processing unit according to another aspect acquires the feature amount of the defective product image stored in the storage unit in the extraction process, and treats a part having a feature amount different from the surrounding feature amount as the defective part. can be extracted.

According to this configuration, since the defective part of the defective product image stored in the storage unit is automatically extracted, it is possible to save the user's trouble.

The image processing unit according to another aspect can receive correction of the range of the extracted defective portion in the extraction process.

That is, it is assumed that the range of the defective portion automatically extracted based on the feature amount of the defective product image is incorrect. In this case, when the user corrects the range of the defective portion, the correction is accepted, so the range of the defective portion can be appropriately set.

The image processing unit according to another aspect receives, in the extraction process, a user's selection operation of the defective part from among the defective product images stored in the storage unit, and extracts the defective part based on the received selection operation. can be extracted.

In other words, there may be cases where extraction based on the feature amount is difficult depending on the defective product image or the defective portion. In such a case, if the user selects a defective portion from the defective product image, the selected defective portion can be extracted, so that the defective portion is less likely to be erroneously extracted.

The image processing unit according to another aspect can paste the defective portion extracted in the extraction process to the reference image in a state in which image conversion is performed on the defective portion.

According to this configuration, instead of changing the contrast of the entire defective product image or rotating it, image conversion is applied only to the defective portion, so that the variation of new defective product images to be generated can be greatly increased. can be done.

The image processing unit according to another aspect can specify a workpiece portion included in the reference image as a pasting target area for the defective portion, and paste the defective portion to the specified pasting target region. , can generate defective images suitable for learning. Further, after pasting the defective portion, a trimming process for trimming a portion of the defective portion protruding from the pasting target area can be executed.

According to this configuration, since an unnatural defective product image in which the defective part is located outside the work portion is not generated, the accuracy of learning can be improved.

Since the display control unit is further provided for displaying the new defective product image generated by the image processing unit according to another aspect on the display unit, what kind of defective product image is generated by pasting and blurring the defective portion. The user can see and confirm whether the As a result, the user can decide whether or not to use the image for learning, such as not using it for learning, if the image gives a sense of discomfort.

The image processing unit according to another aspect can add, to the new defective product image generated by the image processing unit, identification information for distinguishing the defective product image from the defective product image obtained by photographing the defective workpiece.
According to this configuration, it is possible to distinguish between the defective product image generated by the image processing unit and the defective product image obtained by photographing the defective workpiece. In this stage, the defective product image generated by the image processing unit can be excluded from the learning target. Also, when learning the defective product image generated by the image processing unit, the learning weight can be made lighter than the defective product image obtained by photographing the defective workpiece.

As described above, after pasting the defective part extracted from the defective product image on the reference image, by making the boundary between the defective part and the reference image inconspicuous, various defect states actually assumed in the workpiece can be detected. It is possible to naturally create a new defective product image with By having the machine learning network learn the new defective product image and the non-defective product image and the defective product image stored in advance in the storage unit, an inference model with high defective product detection capability can be generated in a short time.

It is a mimetic diagram showing the composition of the appearance inspection device concerning the embodiment of the present invention. It is a block diagram which shows the hardware constitutions of the said visual inspection apparatus. 6 is a flow chart showing an example of a procedure of learning processing; FIG. 11 is a flowchart showing an example of a procedure of internal generation processing of a defective product image; FIG. FIG. 5A shows a defective product image of an actual defective product, and FIG. 5B shows an image obtained by extracting a defective portion from the defective product image. FIG. 6A shows a reference image, FIG. 6B shows an image specifying a pasting target area. FIG. 7A shows an example of a doughnut-shaped defective product image, FIG. 7B and 7C are diagrams showing how the defective portions are pasted. FIG. 8A shows an example of a long-shaped defective product image, FIG. 8B and 8C are diagrams showing how the defective portions are pasted. FIG. 9A shows an example of a defective product image with a complicated shape, FIG. 9B and 9C are diagrams showing how the defective portion is pasted. FIG. 10A shows how the defective part is pasted within the pasting target area, and FIG. 10B shows a state in which the defective portion is pasted out of the region to be pasted. 4 is a flow chart showing an example of a procedure during operation of the visual inspection apparatus;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail based on the drawings. It should be noted that the following description of preferred embodiments is essentially merely illustrative, and is not intended to limit the invention, its applications, or its uses.

FIG. 1 is a schematic diagram showing the configuration of an appearance inspection apparatus 1 according to an embodiment of the present invention. The visual inspection apparatus 1 is a device for determining the quality of a work image obtained by imaging a work to be inspected, such as various parts or products, and is used at a production site such as a factory. can be done. Specifically, a machine learning network is built inside the visual inspection apparatus 1 . A work image obtained by imaging a work to be inspected is input to the generated machine learning network, and the machine learning network can judge whether the work image is good or bad.

The entire workpiece may be inspected, or only part of the workpiece may be inspected. Also, one workpiece may include a plurality of inspection targets. Also, the workpiece image may include a plurality of workpieces.

A visual inspection apparatus 1 includes a control unit 2 serving as an apparatus main body, an imaging unit 3 , a display device (display section) 4 , and a personal computer 5 . The personal computer 5 is not essential and can be omitted. The personal computer 5 can be used instead of the display device 4 to display various information and images, and the functions of the personal computer 5 can be incorporated into the control unit 2 or into the display device 4 .

In FIG. 1, the control unit 2, the imaging unit 3, the display device 4, and the personal computer 5 are described as an example of the configuration of the appearance inspection apparatus 1, but any plurality of these may be combined and integrated. can also For example, the control unit 2 and the imaging unit 3 can be integrated, or the control unit 2 and the display device 4 can be integrated. Also, the control unit 2 can be divided into a plurality of units and some of them can be incorporated into the imaging unit 3 and the display device 4, or the imaging unit 3 can be divided into a plurality of units and some of them can be incorporated into other units. .

(Configuration of imaging unit 3)
As shown in FIG. 2, the imaging unit 3 includes a camera module (imaging section) 14 and an illumination module (illumination section) 15, and is a unit that acquires a workpiece image. The camera module 14 includes an AF motor 141 that drives an imaging optical system, and an imaging board 142 . The AF motor 141 is a part that automatically performs focus adjustment by driving the lens of the imaging optical system, and can perform focus adjustment by conventionally known methods such as contrast autofocus. The imaging substrate 142 includes a CMOS sensor 143 as a light receiving element that receives light incident from the imaging optical system. The CMOS sensor 143 is an imaging sensor configured to acquire a color image. Instead of the CMOS sensor 143, for example, a light receiving element such as a CCD sensor can also be used.

The illumination module 15 includes LEDs (light emitting diodes) 151 as light emitters that illuminate an imaging area including a workpiece, and an LED driver 152 that controls the LEDs 151 . The light emission timing, light emission time, and light emission amount of the LED 151 can be arbitrarily controlled by the LED driver 152 . The LED 151 may be provided integrally with the imaging unit 3, or may be provided separately from the imaging unit 3 as an external lighting unit.

(Configuration of display device 4)
The display device 4 has a display panel such as a liquid crystal panel or an organic EL panel. A work image, a user interface image, and the like output from the control unit 2 are displayed on the display device 4 . Moreover, when the personal computer 5 has a display panel, the display panel of the personal computer 5 can be used as a substitute for the display device 4 .

(operating device)
The operation device for the user to operate the visual inspection apparatus 1 includes, for example, the keyboard 51 and the mouse 52 of the personal computer 5, but is not limited to these, and accepts various operations by the user. Any device can be used as long as it is configured to be able to For example, a pointing device such as the touch panel 41 of the display device 4 is also included in the operating device.

The user's operation of the keyboard 51 and mouse 52 can be detected by the control unit 2 . Further, the touch panel 41 is a conventionally well-known touch-type operation panel equipped with, for example, a pressure-sensitive sensor, and the user's touch operation can be detected by the control unit 2 . The same is true when using other pointing devices.

(Configuration of control unit 2)
The control unit 2 includes a main board 13 , a connector board 16 , a communication board 17 and a power supply board 18 . The main substrate 13 is provided with a processor 13a. The processor 13a controls the operation of each connected board and module. For example, the processor 13 a outputs a lighting control signal for controlling lighting/lighting out of the LED 151 to the LED driver 152 of the lighting module 15 . The LED driver 152 adjusts the light amount of the LED 151 and the like while switching between lighting and extinguishing of the LED 151 and adjusting the lighting time according to the lighting control signal from the processor 13a.

The processor 13a also outputs an imaging control signal for controlling the CMOS sensor 143 to the imaging board 142 of the camera module 14 . The CMOS sensor 143 starts imaging in response to an imaging control signal from the processor 13a, adjusts the exposure time to an arbitrary time, and performs imaging. That is, the image pickup unit 3 picks up an image within the visual field range of the CMOS sensor 143 according to the image pickup control signal output from the processor 13a. If an object is within the field of view, it can also be imaged. For example, the visual inspection apparatus 1 can capture a non-defective product image corresponding to a non-defective product and a defective product image corresponding to a defective product using the imaging unit 3 as images for learning a machine learning network. The learning image may not be an image captured by the imaging unit 3, but may be an image captured by another camera or the like.

On the other hand, during operation of the visual inspection apparatus, the image of the work can be imaged by the imaging unit 3 . In addition, the CMOS sensor 143 is configured so as to be able to output a live image, that is, a currently captured image at a short frame rate at any time.

When the CMOS sensor 143 finishes imaging, the image signal output from the imaging unit 3 is input to the processor 13a of the main board 13 and processed, and is also stored in the memory 13b of the main board 13. . Details of specific processing contents by the processor 13a of the main substrate 13 will be described later. Note that the main board 13 may be provided with a processing device such as FPGA or DSP. A processor 13a integrated with a processing device such as FPGA or DSP may be used.

The main substrate 13 is provided with a display control section 13c. The display control unit 13c is a part that generates a display screen, controls the display device 4, and causes the display device 4 to display the display screen. A specific operation of the display control unit 13c will be described later.

The connector board 16 is a part that receives power supply from the outside via a power connector (not shown) provided on the power interface 161 . The power supply board 18 is a part that distributes the power received by the connector board 16 to each board and modules. do. The power supply board 18 has an AF motor driver 181 . The AF motor driver 181 supplies drive power to the AF motor 141 of the camera module 14 to achieve autofocus. The AF motor driver 181 adjusts the power supplied to the AF motor 141 according to the AF control signal from the processor 13a of the main substrate 13. FIG.

The communication board 17 is a part that executes communication between the main board 13 and the display device 4 and the personal computer 5, communication between the main board 13 and an external control device (not shown), and the like. The external control device can be, for example, a programmable logic controller or the like. Communication may be wired or wireless, and either form of communication can be realized by conventionally known communication modules.

The control unit 2 is provided with a storage device (storage section) 19 such as a solid state drive, a hard disk drive, or the like. The storage device 19 stores program files 80, setting files, and the like (software) for enabling the hardware to execute various controls and processes, which will be described later. The program files 80 and setting files can be stored in a storage medium 90 such as an optical disk, for example, and the program files 80 and setting files stored in this storage medium 90 can be installed in the control unit 2 . The program file 80 may be downloaded from an external server using a communication line. In addition, the storage device 19 can store, for example, the image data and the like, and can also store parameters and the like for constructing an inference model obtained after the learning process described later.

In FIG. 2 , the storage device 19 is shown as being integrated with the control unit 2 , but the storage device 19 may be separate from the control unit 2 . As an example of such a storage device 19, a network compatible storage (NAS: Network Attached Storage) or the like can be cited. The NSA and the control unit 2 are connected by a communication line such as wired LAN or wireless LAN.

That is, in the visual inspection apparatus 1, by learning a machine learning network using images for learning, the parameters of the machine learning network are adjusted and an inference model is generated. A work image obtained by photographing a work to be inspected can be input to the inference model, and the quality of the work can be determined based on the input work image. By using this visual inspection apparatus 1, it is possible to perform a visual inspection method for determining the quality of a workpiece based on the image of the workpiece.

(Processor configuration)
As shown in FIG. 2, the processor 13a is provided with an image processing section 13d, a learning section 13e, and an inspection section 13f. The image processing unit 13d, the learning unit 13e, and the inspection unit 13f may be parts configured by hardware, or may be parts configured by executing software. Also, the image processing unit 13d, the learning unit 13e, and the inspection unit 13f may not necessarily be provided on the main substrate 13, and some or all of them may be provided on a substrate other than the main substrate 13.

The details of each part will be described later, but the outline is as follows. The image processing unit 13d is a part that internally generates a defective product image, which will be described later. Also, the learning unit 13e is a part that inputs learning data to a machine learning network, causes the machine learning network to learn, and generates an inference model for determining whether an input image is good or bad. The learning unit 13e may be composed of a learning computer separate from the control unit 2, for example. The learning computer is configured to be able to perform machine learning at high speed. By communicably connecting the learning computer and the control unit 2, the parameters for constructing the inference model generated by the learning computer can be transmitted to the storage device 19 and stored. Thereby, an inference model can be constructed in the control unit 2 . The inspection unit 13f is a part that inputs the work image to the inference model and judges the quality of the work.

(learning process)
Next, the learning process of the machine learning network will be specifically described. As types of learning, there are defective product learning in which defective product images are learned, and non-defective product learning in which non-defective product images are learned. At the work production site, most of the products are non-defective products, so it is easy to collect many non-defective product images for learning non-defective products. Insufficient and difficult inspections are inferior in performance to defective product learning, so there is a demand for defective product learning. However, since the number of defective products produced is extremely small at work production sites, it is difficult to collect defective product images for learning in a short period of time. In the present embodiment, the appearance inspection apparatus 1 is configured so that a large number of defective product images with a wide variety of variations can be generated naturally. Details will be described below.

The flowchart in FIG. 3 shows an example of the procedure of the learning process. At step SA1 after the start, the learning unit 13e prepares an unlearned machine learning network. An unlearned machine learning network is, for example, one in which the initial values of parameters are randomly determined. At step SA2, the storage device 19 stores the non-defective product image and the defective product image for learning to be used for learning of the machine learning network. This step SA2 is a storage step. The storing step may be performed by the user prior to step SA1. Also, the non-defective product image and the defective product image for learning may be images previously photographed and accumulated by the user, or may be images acquired by newly photographing the work for learning. Alternatively, it may be an image acquired by the imaging unit 3 during operation of the visual inspection apparatus 1 . Also, non-defective product images and defective product images stored in a storage device (not shown) other than the storage device 19 can be used as non-defective product images and defective product images for learning.

Annotate the defective product image. That is, the user performs in advance a process of clearly indicating that the defective product image is a defective product image, a process of designating a defective portion of the defective product image, and the like. The information attached by the annotation is stored in the storage device 19 in association with the corresponding defective product image.

After that, the process proceeds to step SA3. At step SA3, it is determined whether or not the internal generation process of the defective product image is required. The internal defective product image generation process, which will be described later in detail, is a process of using an existing defective product image to create another defective product image inside the visual inspection apparatus 1 . The determination in step SA3 can be made by the user. For example, if only one or several defective product images can be collected, the number of defective product images for defective product learning is not sufficient. determine that it is necessary. On the other hand, if several tens or hundreds of defective product images have been collected for defective product learning, the number of defective product images is sufficient for defective product learning. It is determined that internal generation processing of the defective product image is unnecessary. The user can input the necessity of the internal generation process of the defective product image using the operation device such as the mouse 52 or the like. For example, the display control unit 13c generates a user interface screen on which the user can select whether or not to internally generate defective product images, and causes the display device 4 to display the screen, thereby allowing the user to select whether or not to internally generate defective product images. can be easily entered. Since the input by the user is accepted by the learning unit 13e, it can be treated as if the learning unit 13e determines step SA3 inside the visual inspection apparatus 1. FIG.

If the result of step SA3 is YES and the internal generation process of the defective product image is necessary, the process proceeds to step SA4. Proceed to step SA5.

At step SA4, internal generation processing of defective product images is executed in order to increase the number of defective product images for learning. An example of the procedure of internal generation processing is shown in the flowchart shown in FIG. The flowchart shown in FIG. 4 starts when YES is determined in step SA3 of the flowchart shown in FIG. In step SB1 after the start of the flow chart of FIG. 4, the learning unit 13e reads the non-defective product image and the defective product image stored in the storage device 19, and displays the non-defective product image and the defective product image read by the learning unit 13e on the display control unit 13c. is displayed on the display device 4. Any number of images may be displayed on the display device 4, but all defective product images can be displayed. In this case, defective product images can be displayed in a list format. This makes it possible to present various defective points to the user.

At step SB2, the learning unit 13e receives the user's selection of a defective product image. For example, in the previous step SB1, the display control unit 13c generates a user interface screen that can display defective product images in a list format, and causes the display device 4 to display it. A selection button or the like is incorporated in the user interface screen so that the user can select a desired defective product image by operating the mouse 52 or the like. When the learning unit 13e detects the user's selection operation, the learning unit 13e temporarily stores the defective product image corresponding to the selection operation as being selected.

Also, at step SB3, the learning unit 13e receives the selection of the reference image by the user. A reference image is an original image that is used for internal generation. In many cases, a non-defective product image is suitable, but a defective product image may be used. For example, a selection button or the like is incorporated into the user interface screen displayed in step SB1 so that the user can select a desired image by operating the mouse 52 or the like. When the learning unit 13e detects the user's selection operation, the learning unit 13e temporarily stores the selected image corresponding to the selection operation. The order of step SB2 and step SB3 may be interchanged.

At step SB4, the image processing unit 13d reads the defective product image selected at step SB2, and executes extraction processing for extracting a defective portion of the defective image. The extraction of the defective portion may be performed using a rule-based algorithm, or may be performed manually by the user. The image processing unit 13d acquires the feature amount of the defective product image and executes an algorithm for extracting a portion having a feature amount different from the surrounding feature amount as the defective area in the defective area extraction process.

A specific example of the process of extracting a defective portion will be described below with reference to FIG. FIG. 5A shows the defective product image 200 selected at step SB2. The defective product image 200 is acquired by imaging the workpiece W having the defective portion W1. FIG. 5B is an extracted image 201 obtained by extracting a defective portion W1 from the defective product image 200 using a rule-based algorithm. Examples of algorithms for extracting only the defective portion W1 include, for example, a color extraction algorithm and a GrabCut algorithm, but are not limited to these, and other algorithms can also be used.

In the case of the color extraction algorithm, when the user designates a color corresponding to the defect location W1 on the defective product image 200, the area having that color is automatically extracted as the defect location W1. In the case of the GrabCut algorithm, when the user encloses the defective part W1 and the periphery of the defective part W1 on the defective product image 200, the area is specified, and the defective part W1 is automatically extracted from the specified area. do. That is, in the extraction process, the image processing unit 13d receives a selection operation of the defective part W1 from the defective product image 200 by the user, and extracts the defective part W1 based on the received selection operation.

After that, at step SB5, the image processing unit 13d accepts correction of the range of the defective portion W1 extracted at step SB4. If the area extracted by the color extraction algorithm is much wider or narrower than the defective spot W1, the user re-specifies the color or makes corrections such as changing the threshold value. When the image processing unit 13d accepts this correction, the defect location W1 is extracted again. In the case of the GrabCut algorithm, the user finely designates the foreground and background by correcting strokes or correcting clicks, and each time, the defective portion W1 is re-extracted.

Further, when the user manually extracts the defective portion W1 on the defective product image 200, for example, an operation of encircling the defective portion W1 with the mouse 52, an operation of filling in the defective portion W1, and the like can be given. For example, a freehand tool or the like is used to generate a frame surrounding the defective portion W1, and the defective portion W1 can be extracted using this frame. In this case, the image processing unit 13d extracts the enclosed area or the filled area as the defective portion W1.

Alternatively, the defective portion W1 may be extracted by specifying AI Assisted. In the case of AI Assisted designation, after roughly designating and extracting the contour of the defective portion W1, the interior of the extracted portion is designated by a Fill tool or the like. Thereby, the defective point W1 can be automatically extracted. After automatically extracting the defective portion W1, it is also possible to make a fine correction.

When the user confirms that the defective portion W1 has been extracted as intended and performs an operation to proceed to the next step, the process proceeds to step SB6. At step SB6, the image processing unit 13d determines whether or not to perform image conversion processing, which will be described later, on the defective portion W1 extracted at steps SB4 and SB5. For example, the image processing unit 13d generates a user interface screen for selecting application/non-application of image conversion processing, and the display control unit 13c causes the display device 4 to display the generated user interface screen. The user selects application/non-application of image conversion processing on the user interface screen by operating the mouse 52 or the like. When the image processing unit 13d detects that "apply" has been selected, the process proceeds to step SB7. When detecting that "non-apply" has been selected, the image processing unit 13d proceeds to step SB8.

At step SB7, the image processing unit 13d performs image conversion processing on the defective portion W1 extracted at steps SB4 and SB5. The image conversion processing includes, for example, aspect ratio change processing for changing the aspect ratio of the defect location W1, rotation processing for rotating the defect location W1, and rotation of the defect location W1 in the horizontal direction (X direction) and vertical direction (Y direction). ), contrast conversion processing for converting the contrast of the defect location W1, color change processing for changing the color of the defect location W1, and the like, but are not limited to these, and other image transformations. Treatment may be applied. Also, any plurality of image conversion processes may be performed in any order on the defective portion W1. By going through step SB7, it is possible to paste the defective portion W1 in the state of image conversion to the reference image.

It is also possible to allow the user to select which image conversion process is to be performed. For example, the image processing unit 13d generates a user interface screen for selecting image conversion processing, and the display control unit 13c causes the display device 4 to display the generated user interface screen. When the user selects a desired image conversion process on the user interface screen by operating the mouse 52 or the like, the image processing section 13d detects this and applies the detected image conversion process to the defective portion W1. Also, which image conversion process is to be performed may be automatically set by the visual inspection apparatus 1 . Incidentally, steps SB6 and SB7 may be omitted.

After step SB7 or after step SB6 returns NO, the process proceeds to step SB8. At Step SB8, a pasting target area for pasting the extracted defective portion W1 is specified from the reference image. For example, FIG. Suppose a reference image 202 as shown in 6A is selected. This reference image 202 is a non-defective product image including a non-defective work W. FIG. FIG. 6B is an area identification image 204 after the pasting target area 203 has been identified. The pasting target area 203 is shown in FIG. It is specified by extracting the workpiece W portion included in the reference image 202 shown in 6A. As a method for extracting the workpiece W portion, a method conventionally used in the technical field of image processing can be applied. FIG. In 6B, in order to distinguish the pasting target area 203 from other areas, a binarized mask image is used, the white part is the pasting target area 203, and the black part is the other area.

After that, the process proceeds to step SB9, and it is determined whether or not a rule to be described later is applied when the defective portion W1 is pasted on the pasting target area 203 or not. For example, the image processing unit 13d generates a user interface screen for selecting application/non-application of the rule, and the display control unit 13c causes the display device 4 to display the generated user interface screen. The user selects application/non-application of the rule on the user interface screen by operating the mouse 52 or the like. When the image processing unit 13d detects that "apply" has been selected, the process proceeds to step SB10. When detecting that "non-apply" has been selected, the image processing unit 13d proceeds to step SB11.

In step SB10, a pasting process is executed in which the defective part W1 is pasted to the pasting target area 203 by applying the rule. The rule can be, for example, a rule of pasting the defective part W1 in the pasting target area 203. For example, FIG. Assume that a defective product image 200 as shown in 7A is selected and a doughnut-shaped work W is to be inspected. In this case, it is known that the defective portion W1 exists somewhere in the workpiece W in the circumferential direction. Therefore, FIG. As shown in 7B and 7C, on the image 204 after the pasting target area 203 is specified, the defective portion W1 is moved in the direction corresponding to the circumferential direction of the workpiece W and pasted. That is, a circumferential constraint (an example of a predetermined constraint condition) is applied to the sticking position of the defective portion W1.

Also, for example, FIG. Assume that a defective product image 200 as shown in 8A is selected and a linear workpiece W is to be inspected. In this case, it is known that the defective portion W1 exists somewhere in the longitudinal direction of the workpiece W. As shown in FIG. Therefore, FIG. As shown in 8B and 8C, on the image 204 after the pasting target area 203 is specified, the defective portion W1 is moved in the direction corresponding to the longitudinal direction of the workpiece W and pasted. That is, a longitudinal constraint (an example of a predetermined constraint condition) is applied to the sticking position of the defective portion W1.

Also, for example, FIG. Assume that a defective product image 200 as shown in 9A is selected and an annular work W having a complicated shape is to be inspected. In this case, the defective spot W1 is somewhere on the workpiece W, but it is difficult to simply identify it as in the cases shown in FIGS. Therefore, for example, the workpiece W is identified by detecting the outer and inner edges of the workpiece W, the normal direction of the detected edges is obtained, and the defect spot W1 is moved in the normal direction and pasted. wear.

For example, FIG. As shown in 10A, the defective portion W1 may be randomly attached to a portion corresponding to the workpiece W, which is the attachment target area 203 . Randomly pasting the defective portion W1 results in FIG. As shown in 10B, it is conceivable that part of the defective portion W1 protrudes from the portion corresponding to the workpiece W, which is the pasting target area 203, but the protruding portion can be trimmed by the trimming process described later.

In step SB11, to which the rule has been determined not to be applied in step SB9, the user pastes the defective portion W1 onto the paste target area 203. FIG. For example, after the user operates the mouse 52 or the like to select the pasting position of the defective portion W1, the user can paste the defective portion W1 at a desired position by performing a paste confirming operation. In step SB11, the defective portion W1 may be randomly attached to the attachment target area 203. FIG.

After the pasting process of the defective portion, in step SB12, a trimming process of trimming the part (shown in FIG. 10B of FIG. 10) protruding from the pasting target region 203 of the defective portion W1 is executed. Specifically, the image processing unit 13d acquires information indicating the size, shape, and position of the pasting target area 203, and acquires information indicating the size, shape, and position of the pasted defective portion W1. do. Based on the acquired information, it is determined whether or not the entire defective portion W1 is positioned inside the pasting target region 203 . If the entire defective portion W1 is positioned inside the pasting target region 203, it is determined that the trimming process is unnecessary. On the other hand, if a portion of the defective portion W1 is located outside the pasting target area 203, the portion located outside is regarded as a protruding portion, and is trimmed. As a result, the internally generated defective product image reflects the actual defective state of the workpiece, thereby improving the learning efficiency.

At step SB13, the image processing unit 13d executes a process of making the boundary between the defective portion W1 and the reference image 202 pasted in the pasting processes of steps SB10 and SB11 inconspicuous. That is, since the image from which the defective portion W1 is extracted is different from the reference image 202, the boundary between the defective portion W1 and the reference image 202 may be conspicuous simply by pasting the defective portion W1 onto the reference image 202. be. In addition, since the position where the defective portion W1 is extracted is different from the position where the defective portion W1 is pasted, the boundary between the defective portion W1 and the reference image 202 may be conspicuous. If the boundary is conspicuous, it is not preferable as an image for learning. Therefore, the blurring process of step SB13 is executed. For example, blurring processing or the like is used as the processing for making the image inconspicuous.

Blur processing may include, for example, GAN (Generative Adversarial Network), Deep Image Harmonization, or other rule-based processing, and any blur processing may be used. Blur processing is also called Harmonize processing of so-called deep learning. By executing the blurring process, the edge portion of the defective portion W1 blends nicely with the reference image 202 . As described above, the image processing unit 13d internally generates a new defective product image. This internal generation processing is the image processing step. The internally generated defective product image can be stored in the storage device 19, for example.

Steps SB10 to SB13 are executed multiple times. At this time, the pasting position of the defective portion W1 is made different for each image. Thereby, it is possible to internally generate a plurality of defective product images in which the pasting positions of the defective portion W1 are different from each other. Also, the defective portion W1 having a different image conversion may be pasted on the reference image. Thereby, it is possible to internally generate a plurality of defective product images having different defective parts W1.

At step SB14, the display control unit 13c acquires the internally generated defective product image and causes the display device 4 to display it. Specifically, the display control unit 13c generates a user interface screen for displaying the internally generated defective product image. The user interface screen is provided with a display area in which one or a plurality of internally generated defective product images can be displayed as a list. This allows the user to view and check the internally generated defective product image. As a result, it is possible to set such defective product images that should not be used for learning not to be used for learning. This setting can be set for each defective product image on the user interface screen. The information “not used for learning” is also stored in the storage device 19 .

In step SB15, the new defective product image generated by the image processing unit 13d is provided with identification information for distinguishing it from the defective product image obtained by photographing the defective workpiece. An identification flag is added to the internally generated defective product image, and the defective product image is stored in the storage device 19 together with the added information.

As described above, step SA4 in the flowchart of FIG. 3 is completed. At Step SA6, settings are made so that the internally generated defective product images are not used for learning (exclusion settings), and weighting is performed when they are used for learning.

For example, the learning unit 13e generates a user interface screen for selecting whether or not to use the internally generated defective product image for learning, and the display control unit 13c displays the generated user interface screen on the display device 4. Let The user selects use/not use on the user interface screen by operating the mouse 52 or the like. This selection information is stored, for example, in the memory 13b.

Also, on this user interface screen, it is possible to set the weighting when using internally generated defective product images for learning. Since the internally generated defective product image is different from the actually captured defective product image, it may be difficult to say that it is suitable for learning. In such a case, it is better to make the learning weight by the defective product image to which the identification information is given in step SB15 shown in FIG. There is good news. The weighting can be freely set by the user, and the learning weight based on internally generated defective product images and the learning weight based on defective product images obtained by photographing defective workpieces may be made the same.

At step SA7, the learning unit 13e inputs the non-defective product image and the defective product image pre-stored in the storage device 19 and the defective product image internally generated by the image processing unit 13d to the machine learning network. Then, in step SA8, the input image is learned by a machine learning network to generate an inference model. Steps SA7 and SA8 are learning steps.

At this time, learning is performed based on the information set in step SA6. For example, if the setting is to use internally generated defective product images for learning, the machine learning network is also trained with internally generated defective product images as described above. When the setting is made so that images are not used for learning, internally generated defective product images are excluded from learning targets and are not learned by the machine learning network. For example, at the stage when a large number of defective product images of defective workpieces have been accumulated, internally generated defective product images can be excluded from learning targets.

In addition, the weighting set in step SA6 is also reflected in the learning of the machine learning network. can be made lighter than the weight of

If NO is determined in step SA3 and the defective product image is not internally generated, the process proceeds to step SA5. At step SA5, the non-defective product image and the defective product image pre-stored in the storage device 19 are input to the machine learning network. Then, in step SA8, the input image is learned by a machine learning network to generate an inference model.

A step SA9 stores in the storage device 19 the parameters for constructing the inference model for which learning has been completed.

Although the learning method of the machine learning network is not particularly limited, for example, the following method can be used. That is, the machine learning network can be learned by minimizing the Loss function. Although there are various definitions of Loss, Mean Square Error (MSE) can be cited as an example.

Here, T is the target anomaly degree map, 0 is the output image (abnormality degree map), n is the number of 0 pixels in the image T, and x and y are the pixel positions. A Loss function such as Binary Cross Entropy can also be used. The above is just an example.

(During operation of appearance inspection device 1)
Next, operation of the visual inspection apparatus 1 will be described based on the flowchart shown in FIG. At step SC1 after the start, the processor 13a reads the parameters and the like stored in the storage device 19 to prepare a learned machine learning network. This is the inference model generated by the learning unit 13e. In step SC2, the image of the workpiece to be inspected is captured by the imaging unit 3 to acquire the workpiece image. After that, the process proceeds to step SC3, and the workpiece image acquired in step SC2 is input to the inference model.

Next, at step SC4, the inference model executes inference processing of the work image input at step SC3. Thereafter, at step SC5, the inference model outputs a map image as a result of inference processing.

After that, at step SC6, the quality of the workpiece is determined based on the map image output at step SC5. Steps SC2-SC6 can be executed each time the workpiece changes. The processing shown in this flowchart corresponds to the inspection step executed by the inspection unit 13f.

(Action and effect of the embodiment)
As described above, according to this embodiment, it is possible to extract a defective portion from a defective product image obtained by photographing a defective workpiece. Since the extracted defective portion is pasted on the reference image, a new defective product image having the defective portion is generated. A plurality of defective product images with different pasting positions of defective parts and a plurality of defective product images with different image conversions for defective parts can be automatically and internally generated in a short time. In the newly generated defective product image, blurring processing is performed so that the boundary between the pasted defective portion and the reference image is inconspicuous, so the defective product image becomes a natural defective product image.

By having a machine learning network learn a new defective product image and non-defective product images and defective product images stored in advance in the storage device 19, an inference model with a high ability to detect defective products is generated. Therefore, appearance inspection performance can be improved.
is merely an example in all respects and should not be construed in a restrictive manner. Furthermore, all modifications and changes within the equivalent scope of claims are within the scope of the present invention.

INDUSTRIAL APPLICABILITY As described above, the present invention can be used for inspecting the appearance of workpieces.

1 visual inspection device 4 display device (display unit)
13a processor 13b memory 13c display control unit 13d image processing unit 13e learning unit 13f inspection unit

Claims (12)

A visual inspection apparatus for inputting a workpiece image obtained by photographing a workpiece to be inspected into a machine learning network and determining the quality of the workpiece based on the input workpiece image,
a storage unit for storing non-defective product images corresponding to non-defective products and defective product images corresponding to defective products;
an extraction process for extracting the defective part of the defective product image pre-stored in the storage unit, a pasting process for pasting the defective part extracted by the extraction process to a reference image including the workpiece, and pasting by the pasting process an image processing unit that generates a new defective product image by executing a process that obscures the boundary between the attached defective portion and the reference image;
a learning unit for generating an inference model by causing a machine learning network to learn non-defective product images and defective product images stored in advance in the storage unit and new defective product images generated by the image processing unit;
and an inspection unit that inputs a work image obtained by photographing a work to be inspected into the inference model generated by the learning unit, and performs quality determination of the work based on the input work image. In the visual inspection apparatus according to claim 1,
In the extraction process, the image processing unit acquires the feature amount of the defective product image stored in the storage unit, and extracts a part having a feature amount different from the surrounding feature amount as the defective part. Device. In the appearance inspection apparatus according to claim 2,
The visual inspection apparatus, wherein the image processing unit receives correction of the range of the extracted defective portion in the extraction process. In the visual inspection apparatus according to any one of claims 1 to 3,
In the extracting process, the image processing unit receives a user's selection operation of the defective part from among the defective product images stored in the storage unit, and extracts the defective part based on the received selection operation. inspection equipment. In the visual inspection apparatus according to any one of claims 1 to 4,
The image processing unit is an appearance inspection device that pastes the defective portion extracted in the extraction process on the reference image in a state in which image conversion is performed on the defective portion. In the visual inspection apparatus according to any one of claims 1 to 5,
The image processing unit identifies a workpiece portion included in the reference image as a pasting target area of the defective portion, pastes the defective portion onto the identified pasting target region, and pastes the defective portion. After that, the appearance inspection apparatus performs a trimming process for trimming the part of the defective portion protruding from the pasting target area. In the visual inspection apparatus according to any one of claims 1 to 6,
The visual inspection apparatus further comprising a display control unit that causes a display unit to display the new defective product image generated by the image processing unit. In the visual inspection apparatus according to any one of claims 1 to 7,
The image processing unit provides a new defective product image generated by the image processing unit with identification information for distinguishing it from a defective product image obtained by photographing a defective workpiece. In the appearance inspection device according to claim 8,
The visual inspection apparatus, wherein the learning unit is configured to be capable of performing defective product learning by excluding the defective product image to which the identification information is assigned from a learning object. In the appearance inspection device according to claim 8 or 9,
The learning unit is configured such that the learning weight of the defective product image to which the identification information is assigned can be made lighter than the learning weight of the defective product image obtained by photographing the defective workpiece. inspection equipment. In the visual inspection apparatus according to any one of claims 1 to 5,
The image processing unit is a visual inspection device that affixes a defective portion to a work portion included in the reference image according to a predetermined constraint condition. A visual inspection method for inputting a workpiece image obtained by photographing a workpiece to be inspected into a machine learning network and determining the quality of the workpiece based on the input workpiece image,
a storage step of storing a non-defective product image corresponding to a non-defective product and a defective product image corresponding to a defective product;
an extraction process for extracting a defective part of the defective product image stored in advance in the storage step; a pasting process for pasting the defective part extracted by the extracting process onto a reference image including a workpiece; an image processing step of generating a new defective product image by executing a blurring process that obscures the boundary between the attached defective portion and the reference image;
a learning step of generating an inference model by causing a machine learning network to learn the non-defective product image and the defective product image stored in advance in the storage step and the new defective product image generated in the image processing step;
and an inspection step of inputting a work image obtained by photographing a work to be inspected into the inference model generated in the learning step, and determining the quality of the work based on the input work image.

Images