WO2022153564A1 - 部品検査装置 - Google Patents
部品検査装置 Download PDFInfo
- Publication number
- WO2022153564A1 WO2022153564A1 PCT/JP2021/009188 JP2021009188W WO2022153564A1 WO 2022153564 A1 WO2022153564 A1 WO 2022153564A1 JP 2021009188 W JP2021009188 W JP 2021009188W WO 2022153564 A1 WO2022153564 A1 WO 2022153564A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- defective
- defective product
- generated
- user
- Prior art date
Links
- 238000007689 inspection Methods 0.000 title claims abstract description 105
- 230000002950 deficient Effects 0.000 claims abstract description 212
- 238000003860 storage Methods 0.000 claims abstract description 32
- 238000010801 machine learning Methods 0.000 claims abstract description 21
- 230000007547 defect Effects 0.000 claims description 59
- 238000000034 method Methods 0.000 claims description 58
- 238000004590 computer program Methods 0.000 claims 1
- 238000012360 testing method Methods 0.000 description 16
- 238000010586 diagram Methods 0.000 description 14
- 238000012545 processing Methods 0.000 description 9
- 238000007667 floating Methods 0.000 description 8
- 229910000679 solder Inorganic materials 0.000 description 8
- 238000003384 imaging method Methods 0.000 description 7
- 238000004519 manufacturing process Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 238000009736 wetting Methods 0.000 description 5
- 238000001514 detection method Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000003825 pressing Methods 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000000513 principal component analysis Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000011282 treatment Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
- G06T7/001—Industrial image inspection using an image reference approach
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/84—Systems specially adapted for particular applications
- G01N21/88—Investigating the presence of flaws or contamination
- G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
- G01N2021/8883—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges involving the calculation of gauges, generating models
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/84—Systems specially adapted for particular applications
- G01N21/88—Investigating the presence of flaws or contamination
- G01N21/95—Investigating the presence of flaws or contamination characterised by the material or shape of the object to be examined
- G01N21/956—Inspecting patterns on the surface of objects
- G01N21/95684—Patterns showing highly reflecting parts, e.g. metallic elements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30141—Printed circuit board [PCB]
Definitions
- the present invention relates to a parts inspection device.
- a parts inspection device is used to inspect the mounting condition of the mounted parts soldered to the printed circuit board.
- it is common to adjust the threshold value (tuning) of the inspection logic in order to suppress erroneous detection (oversight or oversight) (Patent Document 1 and the like).
- the non-defective product image and defective product image actually generated in the inspection process are registered as model images, the inspection logic is virtually applied to the model image (called a model test), and the non-defective product image and defective product image are displayed. This is done by manually adjusting the threshold so that it can be detected correctly.
- Defective product images are indispensable in tuning work, and a lot of time and skill are required to mature the inspection logic by repeating fine threshold adjustment work many times while accumulating defective product images.
- An object of the present invention is to provide a technique for supporting the effective threshold adjustment work in a parts inspection device.
- the present invention makes it possible to effectively adjust the threshold value for component inspection by generating a defective product image using a machine learning model and performing a teaching process using the generated defective product image.
- the parts inspection apparatus includes at least a storage means for storing a non-defective product image, a generation means for generating a defective product image using a machine learning model, and parameters for parts inspection.
- a setting means that can be set by the user
- an output means that inspects the non-defective product image stored in the storage means and the defective product image generated by the generation means using the parameters and outputs the inspection result.
- the defective product image that can be used for the teaching process can be increased, and the threshold value can be adjusted appropriately.
- the generation means may generate a defective image by applying a machine learning model to at least one of the non-defective images stored in the storage means.
- a machine learning model for example, a generative model based on a hostile generative network (GAN) can be used.
- GAN hostile generative network
- a designating means capable of specifying the type of defect of the defective image generated by the generating means, and the generating means produces a defective image having the type of defect specified by the designated means. It may be generated.
- the designation means displays a graphical user interface (GUI) such as a designation screen in which the user can specify a plurality of features representing defects, and causes the user to specify the type of defect to be generated using this GUI. It is good.
- GUI graphical user interface
- the designation screen may be configured so that the user can specify an area in the feature space composed of a plurality of feature quantities, and further, at least one of the areas in the feature space has a defect corresponding to the feature space.
- the image to be represented should be displayed. Alternatively, instead of having the area in the feature amount space specified, a plurality of feature amount values may be individually specified by the user.
- the proposing means proposes, for example, to generate a defective image of a specific type when the number of registered defective images of the specific type is small.
- the proposed means generates a defective image for each of a plurality of types of defects by the generation means, and performs a clustering process for classifying the generated defective image into a plurality of clusters. It is determined which cluster each of the defective images stored in the storage means belongs to, and the defects corresponding to the clusters in which the existing defective images are less than a predetermined number are proposed as the types of defective images to be generated. be able to. Further, when generating a defective image corresponding to a certain cluster, it is preferable to add random noise to generate an image having unspecified defects in the cluster.
- the output means is determined to be a non-defective product by using the image of the inspection target, the information indicating whether or not the image of the inspection target is generated by the generation means, and the parameters as the inspection result.
- Information indicating whether or not the product is determined to be defective may be output.
- the user can make appropriate threshold adjustments and generate a missing defective product image.
- Another aspect of the present invention is a support method for supporting the parameter setting of parts inspection in a parts inspection apparatus, which includes a generation step of generating a defective image using a machine learning model for a stored non-defective image. , A setting step that accepts the setting of parameters for parts inspection from the user, an output step that inspects the stored non-defective product image and the generated defective product image using the above parameters, and outputs the inspection result. It is a support method including.
- the present invention can be regarded as a method including at least a part of the above processing, a program for realizing such a method, or a recording medium in which the program is recorded non-temporarily. It should be noted that each of the above means and treatments can be combined with each other as much as possible to form the present invention.
- FIG. 1 is a diagram showing a functional block of the parts inspection device according to the first embodiment.
- FIG. 2 is a diagram showing a hardware configuration of a component inspection device.
- FIG. 3 is a diagram showing the overall flow of the teaching process (threshold adjustment process) performed by the component inspection device.
- FIG. 4 is a diagram showing a flow of a defective product image generation process.
- FIG. 5 is an example of a screen for outputting the result of the model test and adjusting the threshold value used in the teaching process.
- FIG. 6 is an example of a designation screen for designating a defective product image generated in the defective product image generation process.
- FIG. 7 is an example of a designation screen for designating a defective product image generated in the defective product image generation process.
- FIG. 1 is a diagram showing a functional block of the parts inspection device according to the first embodiment.
- FIG. 2 is a diagram showing a hardware configuration of a component inspection device.
- FIG. 3 is a diagram showing the overall flow of the teaching process
- FIG. 8 is an example of a designation screen for designating a defective product image generated in the defective product image generation process.
- FIG. 9A is a diagram showing a functional block of the parts inspection device according to the second embodiment, and FIG. 9B is an example of a designation screen for designating a defective product image to be generated.
- 10A and 10B are diagrams illustrating a process of proposing a defect to be generated in the second embodiment.
- FIG. 1 shows a parts inspection device 1 to which the present invention is applied, and is characterized by assistive technology for appropriately performing parameter adjustment (tuning work) for parts inspection.
- the storage unit 40 stores non-defective product image data and defective product image data generated in the actual production line, which are taken by the imaging unit 10. With the progress of manufacturing technology in recent years, the defect occurrence rate has decreased, and it is not easy to collect necessary defective product images. Therefore, in the present invention, the defective product image generation unit 20 generates a defective product image based on the non-defective product image data 41 by using the machine learning technique.
- the user accepts a designation as to what kind of defect is to be generated from the user via the designation unit 21. By doing so, it is possible to generate a defective image in which a user-specified type of defect occurs, while being similar to an actual non-defective image.
- the user can perform appropriate tuning by adjusting the threshold value using the teaching unit 30 using the defective product image generated in this way and the actually captured non-defective product image and defective product image.
- FIG. 1 is a diagram showing a functional block of the component inspection device 1.
- FIG. 2 is a diagram showing a hardware configuration of the component inspection device 1.
- FIG. 3 is a diagram showing the overall flow of the teaching process (threshold adjustment process) performed by the component inspection device 1.
- FIG. 4 is a diagram showing a flow of a defective product image generation process.
- 5 to 8 are examples of display screens used in the teaching process, FIGS. 5 are mainly screens for model test results and threshold adjustment, and FIGS. 6 to 8 are defective product image generation processes. This is a specification screen for specifying a defective image to be generated.
- the parts inspection device 1 also has a function of actually performing a parts inspection based on the teaching result, but the functions related to the teaching process of the parts inspection device 1 will be mainly described below.
- the component inspection device 1 includes an imaging unit 10, a defective image generation unit 20, a teaching unit 30, and a storage unit 40.
- the imaging unit 10 photographs a component (for example, a component mounting printed circuit board) on the production line.
- the image selected by the user among the captured images is labeled as a non-defective image or a defective image, and is stored in the storage unit 40 as a model image.
- the defective product image generation unit 20 is a functional unit that generates a defective product image using machine learning technology.
- the defective product image generation unit 20 has a designation unit 21 for receiving a designation from the user for the type of defect of the defective product image to be generated, and a generation unit 22 for generating the designated defective product image.
- the designation unit 21 displays a screen (GUI) in which the user can specify the type of defect to be generated, and accepts input from the user.
- GUI screen
- the generation unit 22 applies a machine learning model to the non-defective product image stored in the storage unit 40 to generate a defective product image having the type of defect specified by the designated unit 21.
- the machine learning model used by the generation unit 22 is, for example, a generation model based on a hostile generation network (GAN).
- GAN a generator that generates an image similar to the correct image and a discriminator that discriminates between the correct image and the generated image are learned.
- the generator is trained to generate an image closer to the correct answer, and the discriminator is trained to more accurately discriminate between the correct image and the generated image.
- the generation unit 22 uses a generator to generate and output a defective image in which a specified type of defective feature amount is transferred to an actual non-defective image. In this way, the defective product image can be used to generate a defective product image having a specified type of defect for a component similar to the non-defective product image, that is, a defective product image that is likely to actually occur.
- the teaching unit 30 is a functional unit used for the user to set a parameter (threshold value) for component inspection.
- the teaching unit 30 includes an inspection unit 31, a result output unit 32, and a threshold value setting unit 33.
- the inspection unit 31 inspects (tests) the model image stored in the storage unit 40 using the currently set logic and threshold value.
- the test on the model image is also referred to as a model test.
- the result output unit 32 displays the inspection result by the inspection unit 31 to the user. The details will be described later, but the inspection result is a non-defective product using an image of the part to be inspected, information indicating whether or not the image to be inspected is generated by the defective product image generation unit 20, and the current threshold value. Includes information indicating whether it is determined to be a defective product or a defective product.
- the storage unit 40 stores the non-defective product image data 41, the defective product image data 42, and the inspection threshold value 43.
- the non-defective image data 41 is image data actually taken by the imaging unit 10.
- the defective product image data 42 includes the image data actually taken by the imaging unit 10 and the image data generated by the defective product image generation unit 20, and indicates whether the image data is actually taken or generated. Information is also associated and stored.
- the user specifies whether to register the image captured by the imaging unit 10 as a model image of a non-defective product image or a defective product image.
- the inspection threshold value 43 is a threshold value used in the component inspection logic, and can be changed by the user via the teaching unit 30.
- FIG. 2 is a diagram showing a hardware configuration of the parts inspection device 1.
- the parts inspection device 1 has the same configuration as a general computer (information processing device), and has a CPU (Central Processing Unit) 51, a ROM (Read Only Memory) 52, and a RAM (Random Access Memory). ) 53, storage 54, keyboard 55, mouse 56, monitor 57, and communication interface 58. Each configuration is communicably connected to each other via a bus 59.
- a bus 59 is a bus 59.
- the ROM 52 or the storage 54 stores a learning program for executing the learning process of the learning model.
- the CPU 51 is a central arithmetic processing unit that executes various programs and controls each configuration. That is, the CPU 51 reads the program from the ROM 52 or the storage 54, and executes the program using the RAM 53 as a work area. The CPU 51 controls each of the above configurations and performs various arithmetic processes according to the program recorded in the ROM 52 or the storage 54.
- the ROM 52 stores various programs and various data.
- the RAM 53 temporarily stores a program or data as a work area.
- the storage 54 is composed of an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory, and stores various programs including an operating system and various data.
- the keyboard 55 and the mouse 56 are examples of input devices and are used to perform various inputs.
- the monitor 57 is, for example, a liquid crystal display and displays a user interface.
- the monitor 57 may adopt a touch panel method and function as an input unit.
- the communication interface 58 is an interface for communicating with other devices, and standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark) are used.
- FIG. 3 is a flowchart showing the entire flow of the teaching process performed by the component inspection device 1.
- FIG. 4 is a flowchart showing the flow of defective product image generation processing.
- FIG. 5 is an example of a screen used in the teaching process.
- the screen 100 includes an enlarged image display area 110, a result output area 120, a product number designation area 130, a logic / threshold value setting area 140, a test execution button 150, and a defective product image generation button 160.
- the product number designation area 130 is a GUI for designating a component (product number) to be set as a threshold value, and the target component is designated by the user checking a check box.
- the logic threshold setting area 140 is a GUI for the user to select an inspection item and input a threshold value used in the inspection item.
- the result output area 120 indicates the image used for the test, the inspection result at the current setting, and whether the image is an actual non-defective product image, an actual defective product image, or a generated defective product image.
- the enlarged image display area 110 the image selected from the plurality of images to be inspected is enlarged and displayed, and the height profile in the X-axis and Y-axis directions at the cursor position is displayed.
- the test execution button 150 the inspection in the product number designation area 130 and the logic threshold setting area 140 is executed.
- the defective product image generation button 160 By pressing the defective product image generation button 160, a defective product image can be generated.
- the teaching unit 30 displays the screen 100.
- the processing order is only an example in the flowchart shown in FIG. 3, and it is not necessary to perform the processing according to this order, and the processing order may be changed as appropriate.
- step S1 the teaching unit 30 receives the designation of the part part number to be the target of the model test via the part number designation area 130.
- the product number "BBB" is selected in the product number designation area 130.
- step S2 the teaching unit 30 receives the designation of the inspection item of the model test via the logic threshold setting area 140.
- electrode wetting at the center of the end is selected as an inspection item.
- the current threshold value for the selected inspection item is displayed in the logic threshold setting area 140, and the value can be edited.
- the teaching unit 30 may display the model image of the specified product number and whether each model image is a good product or a defective product for the specified inspection item. good.
- step S3 when the user determines that the number of defective product images is insufficient when performing the model test, the user presses the defective product image generation button 160 to generate a defective product image and register it as a model image. ..
- step S3 is a process executed by the defective image generation unit 20.
- step S31 a screen for designating the type of defect to be generated in the designation unit 21 is displayed.
- 6 to 8 are examples of screens for designating the type of defect to be generated.
- defects the amount of solder, the floating of the electrode, and the non-wetting of the electrode will be described as targets. Since these defects are typical examples of defects for which it is difficult to set an appropriate threshold value in teaching, it is preferable to generate a defective product image for such defects, but generate a defective product image for other defects. It doesn't matter.
- 6 and 7 are screens in which the type of defect to be generated can be specified by designating an area in the feature space consisting of three feature quantities indicating defects.
- the amount of solder is divided into three levels
- the degree of floating of the electrode is divided into two levels
- the wet / non-wetness of the electrode is divided into two levels. It can be specified by the user.
- the amount of solder and the degree of floating of the electrodes may be divided into more or less levels.
- Wetting / non-wetting of electrodes is basically binary information, but it may be divided into 3 levels or more. Since it is impossible to combine some features, it is not possible to select them on the screen.
- the types of defects are represented by dividing the electrodes into wet and non-wet, with the horizontal axis representing the amount of solder and the vertical axis representing the degree of floating of the electrodes in the space.
- a total of 10 areas of ID11 to ID15 and ID21 to ID25 can be selected.
- an image showing an example of the defect corresponding to this feature space is shown. This makes it easy for the user to understand what kind of defects are generated. The user selects what kind of defect is to be generated on this specification screen.
- the frame 601 indicates that the defect is selected by the user.
- FIG. 7 is a screen for specifying the type of defect by specifying the area in the feature amount space as in FIG. 6, but the three axes are the amount of solder, the degree of floating, and wet / non-wet.
- This is a screen in which the user specifies an area in a three-dimensional space.
- an image showing an example of a defect is not displayed corresponding to each area, but it may be displayed.
- FIG. 8 is a diagram showing another example of the designated screen.
- the user can directly input the amount of solder, the degree of floating of the electrode, and the wetness / non-wetness of the electrode as values individually.
- a GUI for inputting the amount of solder and the degree of floating using a scroll bar is used, but other GUIs such as spin-up / down control and numerical input to a text box may be used.
- the wet / non-wetness of the electrode is binary information, it is convenient to specify it with a check box.
- step S32 the designation unit 21 receives from the user the type of defect to be generated. Specifically, the designation unit 21 acquires the defect type input when the NG generation button 603 is pressed in FIGS. 6 to 8 as the defect type to be generated.
- step S33 the generation unit 22 acquires an image of the target product number from the storage unit 40.
- the target product number is the product number specified in step S1.
- the generation unit 22 acquires one or more data of the target product number from the non-defective product image data 41 stored in the storage unit 40.
- a large number of images corresponding to the storage unit 40 may be randomly selected or selected according to a predetermined rule.
- step S34 the generation unit 22 generates a defective image from the image acquired in step S33.
- the generation unit 22 has a generator of a hostile generation network (GAN), inputs the acquired non-defective image to the generator, and noises the feature amount of the defect specified by the user.
- GAN hostile generation network
- the noise to be added is, for example, randomly determined from within the specified feature space. Therefore, even if the designated types of defects are the same, the actually generated defective image will fluctuate, and it is possible to generate defective images of the same type but with different details.
- step S35 the generation unit 22 presents the generated defective product image to the user. Specifically, on the screens of FIGS. 6 to 8, the defective image is displayed on the three-dimensional viewer units 602a and 602b. In the three-dimensional viewer units 602a and 602b, the user can arbitrarily specify the display position, enlargement ratio, and viewpoint direction of the generated image, and can confirm the generated defect.
- step S36 it is determined whether or not the user registers the generated defective product image as a model image.
- the user confirms the defective product image using the three-dimensional viewer units 602a and 602b, and if it is determined that the defective product image is suitable for use in teaching, the user presses the "Add to library" button 604. When the button 604 is pressed, the process proceeds to step S37.
- step S37 the generated defective product image is stored in the storage unit 40 as a model image.
- the defective product image is stored in the storage unit 40 so that it can be determined that the defective product image is a defective product image generated by the defective product image generation unit 20.
- step S38 the user determines whether or not to generate the next defective image, returns to step S32 if the generation is continued, and ends the process if the generation is finished.
- a pseudo defective product image having a defect specified by the user is generated by the machine learning-based generator and registered as a model image.
- the generated image does not necessarily have to be a defective image, and a non-defective image can be generated by the same processing.
- a non-defective image having desired characteristics is generated and registered as a model image. It doesn't matter.
- step S4 the user presses the test execution button 150 to execute the inspection on the model image using the current inspection content and the inspection threshold value.
- the inspection unit 31 executes an inspection of the inspection logic specified in step S2 for each of the model images stored in the storage unit 40.
- step S5 the result output unit 32 outputs the inspection result of step S4 to the result output area 120 of the screen 100.
- the model image in which the model test was performed the result of the model test, and information about the model image are displayed.
- the information about the model image is whether the model image is a good product image or a defective product image, and if the model image is a defective product image, it is a defective product image actually taken or a defective product image generator 20. Contains information indicating whether the image is a defective product generated by.
- model images 121a to 121e are displayed in the result output area 120, and the respective inspection results 122a to 122e and the information 123a to 123e related to the model image are displayed.
- the model images 121a and 121c are actual non-defective product images
- the model images 121b and 121d are actual defective product images
- the model image 121e is a defective product image generated by the defective product image generation unit 20.
- the model images 121a and 121b can be appropriately determined whether they are non-defective or defective
- the model image 121c is erroneously detected (overlooked) to determine that the non-defective product is defective, and the model image 121d. It can be seen that erroneous detection (missing) has occurred in which a defective product is judged to be a non-defective product.
- step S6 the user determines whether the inspection is performed correctly, that is, whether the threshold needs to be adjusted. If threshold adjustment is required, the user changes the threshold value in the logic threshold setting area 140.
- the threshold value setting unit 33 stores the threshold value changed by the user in the storage unit 40. After that, the process returns to step S4 and the model test is executed again.
- step S3 when the threshold value needs to be adjusted, only the threshold value is changed, but the defective product image generation process in step S3 is executed to add the defective product image. It doesn't matter.
- the user repeats the above process and adjusts the threshold value so that the parts can be inspected correctly.
- the above process can be performed for various part numbers and inspection logics.
- a pseudo defective image by generating a pseudo defective image using a machine learning model, it can be used for threshold adjustment processing (teaching work). It is difficult to collect defective product images, especially when a new production line is installed, but automatic generation makes it possible to compensate for the shortage of non-defective product images and adjust the threshold appropriately.
- an image having a specific defect may be required to confirm whether the threshold adjustment is properly performed, but since such a defective image can be specified by the user and generated, the image can be generated. From this point of view, appropriate threshold adjustment becomes easy.
- the generated defective product image is close to the actual non-defective product image and is similar to the defect that occurs in the actual site. It is possible to generate a defective image.
- the component inspection device In the first embodiment, the user needs to specify what kind of defective image is to be generated.
- the component inspection device according to the present embodiment further has a function of proposing what kind of defective product image should be generated.
- FIG. 9A is a diagram showing a functional block of the parts inspection device 2 according to the present embodiment.
- the defective product image generation unit 20 further includes the proposal unit 23, and the others are the same. Therefore, the proposal unit 23 will be mainly described below.
- FIG. 9B shows an example of a designation screen in which the user specifies the type of defective image to be generated.
- the designated screen includes a button 901 for executing a function of proposing the type of defect to be generated.
- the proposal unit 23 executes the generated image proposal process shown in FIG. 10A.
- step S101 the proposal unit 23 generates a defective product image using the generation unit 22 for each of the plurality of defective IDs.
- a plurality of defective image images for example, about 10 images, are generated for each defect.
- step S102 the proposal unit 23 extracts the feature amount by performing principal component analysis or the like on the generated defective product image, and maps it to the feature amount space.
- step S103 the proposal unit 23 performs a clustering process on the defective product image in the feature quantity space to classify the defective product image into a plurality of clusters.
- the clustering method is not particularly limited, and for example, k-means or the like may be used.
- step S104 the proposal unit 23 extracts the model image (library image) stored in the storage unit 40 as the feature amount and maps it on the feature amount space in the same manner as in step S102, and the model image is assigned to which cluster. Determine if it belongs.
- step S105 the proposal unit 23 identifies a cluster such that the library image does not exist in the area.
- step S106 the proposal unit 23 displays an alert on the GUI for the defective type corresponding to the cluster in which the model image does not exist in the area, and informs the user to generate the defective product image of this defective type. suggest.
- the proposal unit 23 may propose the defective type in which the number of existing model images is a predetermined number or less.
- FIG. 10B is a diagram illustrating the above process.
- the region in FIG. 10B shows a cluster (each defective region) obtained by mapping the defective product image generated in step S101 to a two-dimensional space and performing a clustering process.
- Ten clusters (areas) are obtained corresponding to ten defective types.
- the points in FIG. 10B are points where the model image is mapped on the two-dimensional space. From this result, it can be seen that the model image included in the cluster of ID12 does not exist.
- the proposal unit 23 issues an alert 902 to the ID 12 to indicate that the defect of the ID 12 is insufficient on the defect type designation screen (FIG. 9B) and therefore an image of the defect should be generated. indicate. If a plurality of types of defects are insufficient, an alert may be displayed for each defect.
- the defective product image is generated in the above explanation, a non-defective product image may be generated in the same manner. Good product images are available in large quantities, but good product images required for threshold adjustment may not be available. Therefore, if a non-defective image having desired characteristics is generated by the same method as described above, the threshold value can be adjusted more appropriately.
- the inspection target is the component mounting printed circuit board, but any component may be inspected.
- the types of defects, the amount of solder, the floating of the electrodes, and the non-wetting of the electrodes have been described as examples, but other types of defects may be inspection items.
- the defective product image is generated by the machine learning model based on the hostile generative network (GAN), but the defective product image is generated by using any machine learning model other than GAN, especially the deep generation model. It may be generated.
- Deep generative models include, for example, hostile generative networks (GANs), variational autoencoders (VAEs), and flow-based generative models.
- Support methods including.
- Imaging unit 20 Defective product image generation unit 21: Designation unit 22: Generation unit 23: Proposal unit 30: Teaching unit 31: Inspection unit 32: Result output unit 33: Threshold setting unit 40: Storage unit 41: Good product image data 42: Defective product image data 43: Inspection threshold
Abstract
Description
図1を参照して、本発明の適用例の一つについて説明する。図1は、本発明を適用した部品検査装置1を示し、部品検査のためのパラメータ調整(チューニング作業)を適切に行うための支援技術に特徴がある。記憶部40には、撮像部10によって撮影された、実際の製造ラインで生じた良品画像データおよび不良品画像データを記憶する。近年の製造技術の進歩に伴い不良発生率は低減しており、必要な不良品画像を収集することが容易ではない。そこで、本発明では、不良品画像生成部20が、機械学習技術を用いて、良品画像データ41に基づいて不良品画像を生成する。どのような種類の不良を含む画像を生成するかは、指定部21を介してユーザから指定を受け付ける。こうすることにより、実際の良品画像と類似しつつ、ユーザが指定した種類の不良が生じている不良品画像を生成することができる。
図1~図5を参照して、本発明の実施形態に係る部品検査装置1について詳細に説明する。図1は、部品検査装置1の機能ブロックを示す図である。図2は、部品検査装置1のハードウェア構成を示す図である。図3は、部品検査装置1が行うティーチング処理(閾値調整処理)の全体の流れを示す図である。図4は、不良品画像の生成処理の流れを示す図である。図5~図8はティーチング処理において用いられる表示画面の例であり、図5は主にモデルテストの結果および閾値調整のための画面であり、図6~図8は、不良品画像生成処理において生成する不良品画像を指定するための指定画面である。
図1に示すように、部品検査装置1は、撮像部10、不良品画像生成部20、ティーチング部30、記憶部40を含む。
図3は、部品検査装置1が行うティーチング処理の全体の流れを示すフローチャートである。図4は、不良品画像生成処理の流れを示すフローチャートである。図5は、ティーチング処理において用いられる画面の一例である。
によって、ユーザによって指定された不良を有する擬似的な不良品画像が生成されて、モデル画像として登録される。なお、生成する画像は必ずしも不良品画像とする必要はなく、同様の処理によって良品画像を生成することも可能であり、所望の特徴を有する良品画像を生成してモデル画像として登録するようにしても構わない。
第1の実施形態では、どのような不良品画像を生成するかをユーザが自ら指定する必要がある。本実施形態に係る部品検査装置は、どのような不良品画像を生成するべきかを提案する機能をさらに有する。
上記実施形態は、本発明の構成例を例示的に説明するものに過ぎない。本発明は上記の具体的な形態には限定されることはなく、その技術的思想の範囲内で種々の変形が可能である。
1.少なくとも良品画像(41)を記憶する記憶手段(40)と、
機械学習モデルを用いて不良品画像を生成する生成手段(22)と、
部品検査のためのパラメータをユーザが設定可能な設定手段(21,140)と、
前記パラメータを用いて、前記記憶手段に記憶されている良品画像と前記生成手段によって生成された不良品画像に対する検査を行い、検査結果を出力する出力手段(32,120)と、
を備える、部品検査装置(1)。
記憶されている良品画像に対して機械学習モデルを用いて不良品画像を生成する生成ステップ(S3)と、
部品検査のためのパラメータの設定をユーザから受け付ける設定ステップ(S7)と、
前記パラメータを用いて、記憶されている良品画像と生成された不良品画像に対する検査を行い(S4)、検査結果を出力する出力ステップ(S5)と、
を含む、支援方法。
10:撮像部
20:不良品画像生成部 21:指定部 22:生成部 23:提案部
30:ティーチング部 31:検査部 32:結果出力部 33:閾値設定部
40:記憶部 41:良品画像データ 42:不良品画像データ 43:検査閾値
Claims (13)
- 少なくとも良品画像を記憶する記憶手段と、
機械学習モデルを用いて不良品画像を生成する生成手段と、
部品検査のためのパラメータをユーザが設定可能な設定手段と、
前記パラメータを用いて、前記記憶手段に記憶されている良品画像と前記生成手段によって生成された不良品画像に対する検査を行い、検査結果を出力する出力手段と、
を備える、部品検査装置。 - 前記生成手段は、前記記憶手段に記憶された良品画像の少なくともいずれかに対して、機械学習モデルを適用して、不良品画像を生成する、
請求項1に記載の部品検査装置。 - 前記機械学習モデルは、深層生成モデルである、
請求項1または2に記載の部品検査装置。 - 前記生成手段で生成する不良品画像の不良の種類をユーザが指定可能な指定手段をさらに備え、
前記生成手段は、前記指定手段で指定された種類の不良を有する不良品画像を生成する、
請求項1から3のいずれか1項に記載の部品検査装置。 - 前記指定手段は、不良を表す複数の特徴量をユーザが指定可能な指定画面を表示する、
請求項4に記載の部品検査装置。 - 前記指定画面は、複数の特徴量からなる特徴空間内の領域をユーザが指定可能に構成される、
請求項5に記載の部品検査装置。 - 前記特徴空間内の領域の少なくともいずれかに、当該特徴空間に対応する不良を表す画像が表示される、
請求項6に記載の部品検査装置。 - 前記指定画面は、複数の特徴量の値をそれぞれ個別にユーザが指定可能に構成される、
請求項5に記載の部品検査装置。 - 生成すべき不良品画像の種類を提案する提案手段をさらに備える、
請求項1から8のいずれか1項に記載の部品検査装置。 - 前記提案手段は、
前記生成手段によって複数の種類の不良のそれぞれについて、不良品画像を生成し、
生成された不良品画像を複数のクラスタに分類するクラスタリング処理を施し、
前記記憶手段に記憶されている不良品画像のそれぞれがどのクラスタに属するかを判断し、
存在する不良品画像が所定数以下のクラスタに対応する不良を、生成すべき不良品画像の種類として提案する、
請求項9に記載の部品検査装置。 - 前記出力手段は、検査結果として、検査対象の画像と、当該検査対象の画像が前記生成手段によって生成されたものか否かを示す情報と、前記パラメータを用いて良品と判定されるか不良品と判定されるかを示す情報と、を出力する、
請求項1から10のいずれか1項に記載の部品検査装置。 - 部品検査装置において部品検査のパラメータ設定を支援する支援方法であって、
記憶されている良品画像に対して機械学習モデルを用いて不良品画像を生成する生成ステップと、
部品検査のためのパラメータの設定をユーザから受け付ける設定ステップと、
前記パラメータを用いて、記憶されている良品画像と生成された不良品画像に対する検査を行い、検査結果を出力する出力ステップと、
を含む、支援方法。 - コンピュータに、請求項12に記載の方法の各ステップを実行させるためのコンピュータプログラム。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/259,624 US20240070847A1 (en) | 2021-01-14 | 2021-03-09 | Component inspection device |
CN202180087845.7A CN116724333A (zh) | 2021-01-14 | 2021-03-09 | 部件检查装置 |
DE112021006792.3T DE112021006792T5 (de) | 2021-01-14 | 2021-03-09 | Bauteilprüfgerät |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2021004036A JP2022108855A (ja) | 2021-01-14 | 2021-01-14 | 部品検査装置 |
JP2021-004036 | 2021-01-14 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022153564A1 true WO2022153564A1 (ja) | 2022-07-21 |
Family
ID=82447089
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2021/009188 WO2022153564A1 (ja) | 2021-01-14 | 2021-03-09 | 部品検査装置 |
Country Status (5)
Country | Link |
---|---|
US (1) | US20240070847A1 (ja) |
JP (1) | JP2022108855A (ja) |
CN (1) | CN116724333A (ja) |
DE (1) | DE112021006792T5 (ja) |
WO (1) | WO2022153564A1 (ja) |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2011214903A (ja) * | 2010-03-31 | 2011-10-27 | Denso It Laboratory Inc | 外観検査装置、外観検査用識別器の生成装置及び外観検査用識別器生成方法ならびに外観検査用識別器生成用コンピュータプログラム |
JP2017211259A (ja) * | 2016-05-25 | 2017-11-30 | 株式会社シーイーシー | 検査装置、検査方法、及びプログラム |
JP2018077147A (ja) * | 2016-11-10 | 2018-05-17 | オムロン株式会社 | 検査システム、検査装置の制御方法及びプログラム |
WO2018173478A1 (ja) * | 2017-03-23 | 2018-09-27 | 日本電気株式会社 | 学習装置、学習方法および学習プログラム |
JP2018205163A (ja) * | 2017-06-06 | 2018-12-27 | 株式会社デンソー | 外観検査装置、変換データ生成装置、及びプログラム |
WO2019188040A1 (ja) * | 2018-03-29 | 2019-10-03 | 日本電気株式会社 | 画像処理装置、画像処理方法および画像処理プログラム |
WO2019186915A1 (ja) * | 2018-03-29 | 2019-10-03 | 三菱電機株式会社 | 異常検査装置および異常検査方法 |
JP2020052520A (ja) * | 2018-09-25 | 2020-04-02 | エヌ・ティ・ティ・コムウェア株式会社 | 判定装置、判定方法、およびプログラム |
-
2021
- 2021-01-14 JP JP2021004036A patent/JP2022108855A/ja active Pending
- 2021-03-09 DE DE112021006792.3T patent/DE112021006792T5/de active Pending
- 2021-03-09 WO PCT/JP2021/009188 patent/WO2022153564A1/ja active Application Filing
- 2021-03-09 CN CN202180087845.7A patent/CN116724333A/zh active Pending
- 2021-03-09 US US18/259,624 patent/US20240070847A1/en active Pending
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2011214903A (ja) * | 2010-03-31 | 2011-10-27 | Denso It Laboratory Inc | 外観検査装置、外観検査用識別器の生成装置及び外観検査用識別器生成方法ならびに外観検査用識別器生成用コンピュータプログラム |
JP2017211259A (ja) * | 2016-05-25 | 2017-11-30 | 株式会社シーイーシー | 検査装置、検査方法、及びプログラム |
JP2018077147A (ja) * | 2016-11-10 | 2018-05-17 | オムロン株式会社 | 検査システム、検査装置の制御方法及びプログラム |
WO2018173478A1 (ja) * | 2017-03-23 | 2018-09-27 | 日本電気株式会社 | 学習装置、学習方法および学習プログラム |
JP2018205163A (ja) * | 2017-06-06 | 2018-12-27 | 株式会社デンソー | 外観検査装置、変換データ生成装置、及びプログラム |
WO2019188040A1 (ja) * | 2018-03-29 | 2019-10-03 | 日本電気株式会社 | 画像処理装置、画像処理方法および画像処理プログラム |
WO2019186915A1 (ja) * | 2018-03-29 | 2019-10-03 | 三菱電機株式会社 | 異常検査装置および異常検査方法 |
JP2020052520A (ja) * | 2018-09-25 | 2020-04-02 | エヌ・ティ・ティ・コムウェア株式会社 | 判定装置、判定方法、およびプログラム |
Also Published As
Publication number | Publication date |
---|---|
US20240070847A1 (en) | 2024-02-29 |
CN116724333A (zh) | 2023-09-08 |
JP2022108855A (ja) | 2022-07-27 |
DE112021006792T5 (de) | 2023-11-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP5254612B2 (ja) | グラフィック再検査ユーザ設定インタフェース | |
US10260996B2 (en) | Board production monitoring device and board production monitoring method | |
TWI570654B (zh) | Use the test data for quality control | |
TWI536192B (zh) | 3d列印資料生成方法、3d圖檔列印方法及3d 列印資料處理系統 | |
JP2004012422A (ja) | パターン検査装置、パターン検査方法およびプログラム | |
JP5200970B2 (ja) | 品質管理システムおよび品質管理装置および品質管理プログラム | |
JP2009283584A (ja) | 表面欠陥データ表示管理装置および表面欠陥データ表示管理方法 | |
JP6122965B2 (ja) | 検査システム | |
JP6971112B2 (ja) | 教師データ作成支援装置、分類装置および教師データ作成支援方法 | |
JP2009272497A (ja) | レシピパラメータ管理装置およびレシピパラメータ管理方法 | |
TW201407395A (zh) | 產生用於製造工具之配方之方法及其系統 | |
WO2022153564A1 (ja) | 部品検査装置 | |
JP2009134623A (ja) | 工程改善支援装置、工程改善支援方法、工程改善支援用プログラム、および工程改善支援用プログラムを記録した記録媒体 | |
JP6305252B2 (ja) | 目視検査技能向上支援システム及びこれを用いた目視検査技能向上支援方法並びに目視検査技能向上支援システム用プログラム | |
JP2007264914A (ja) | データ解析方法 | |
JP7035857B2 (ja) | 検査方法、検査システム及びプログラム | |
US7451048B2 (en) | Method and apparatus for analyzing electromagnetic wave, and computer product | |
JP6166120B2 (ja) | データ処理装置、測定装置、選別装置、データ処理方法およびプログラム | |
WO2021039591A1 (ja) | 判定基準生成装置、判定装置、判定基準生成プログラム、および判定プログラム | |
JP2011035873A (ja) | 画像処理回路のシミュレーション装置、画像処理回路のシミュレーション方法、画像処理回路の設計方法、及び画像処理回路のシミュレーションプログラム | |
JP6679394B2 (ja) | 測定結果報知装置、測定システムおよび測定結果報知方法 | |
JP6739967B2 (ja) | 測定結果報知装置、測定システムおよび測定結果報知方法 | |
JP2009134518A (ja) | 試験プログラムの検証方法及びその検証システム | |
JP2014163710A (ja) | 基板処理装置 | |
US20230066672A1 (en) | Image processing method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21919457 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 202180087845.7 Country of ref document: CN |
|
WWE | Wipo information: entry into national phase |
Ref document number: 18259624 Country of ref document: US |
|
WWE | Wipo information: entry into national phase |
Ref document number: 112021006792 Country of ref document: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21919457 Country of ref document: EP Kind code of ref document: A1 |