Classifications

• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
  • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
  • G01N21/84—Systems specially adapted for particular applications
  • G01N21/88—Investigating the presence of flaws or contamination

Definitions

• the present invention relates to a technology for determining inspection conditions in an inspection system that inspects an object to be inspected.
• Patent Document 1 discloses a method for setting the lighting conditions in an inspection system, which includes illuminating the object with a light source capable of changing lighting parameters that define the lighting conditions when the object is imaged, imaging the object with a plurality of lighting parameters using an image sensor, acquiring captured images corresponding to the plurality of lighting parameters, and the object having known label data. Based on the image data set obtained by associating the captured images with the lighting parameters corresponding to the captured images, an estimated image of the object when the object is illuminated with variable lighting parameters is generated.
• the estimated image and the label data corresponding to the variable lighting parameters of the object are applied to learning of the machine learning model, and based on a comparison result between the estimation result of the machine learning model and the label data of the object, both the lighting parameters and the inspection algorithm parameters are simultaneously optimized, thereby setting both the lighting conditions and the inspection algorithm parameters of the machine learning model.
• Patent Document 1 makes it possible to simultaneously optimize both the lighting parameters and the parameters of the inspection algorithm, taking into account only the accuracy rate of the inspection.
• the present invention was developed in consideration of the above circumstances, and its purpose is to provide a technology that can appropriately determine the inspection conditions in an inspection system.
• an inspection condition determination system is an inspection condition determination system that includes a processor and determines inspection conditions for an inspection by an inspection system for a specified inspection target, and the processor receives input of a user request regarding at least two or more element evaluation indexes and the goals of the element evaluation indexes for the inspection by the inspection system, the element evaluation indexes being the inspection accuracy rate, false alarm rate, inspection time, inspection reproducibility, learning time, equipment cost, operating cost, equipment robustness, inspection versatility, lifespan, ease of assembly, required assembly precision, robustness against the environment, ease of maintenance, clarity of the judgment basis for the recognition process, and ease of recognition of the detection point, creates an evaluation index based on multiple element evaluation indexes for the inspection by the inspection system based on the user request, evaluates the inspection results under multiple inspection conditions using the evaluation indexes, and determines appropriate inspection conditions from among the multiple inspection conditions.
• the present invention makes it possible to appropriately determine the inspection conditions in an inspection system.
• FIG. 1 is a diagram showing the overall configuration of an inspection system according to an embodiment.
• FIG. 2 is a configuration diagram of a computer that determines inspection conditions according to an embodiment.
• FIG. 3 is a diagram illustrating inspection conditions according to an embodiment.
• FIG. 4 is a flowchart of a first inspection condition optimization process according to an embodiment.
• FIG. 5 is a flowchart of a second inspection condition optimization process according to an embodiment.
• FIG. 6 is a flowchart of an optical model calibration process according to an embodiment.
• FIG. 7 is a diagram showing a GUI screen according to an embodiment.
• FIG. 8 is a diagram for explaining inspection conditions when dividing an inspection area according to an embodiment.
• FIG. 9 is a diagram illustrating a change in an inspection image according to an embodiment.
• FIG. 10 is a flowchart of an inspection condition redetermining process according to an embodiment.
• FIG. 11 is a diagram illustrating a manufacturing process according to an embodiment.
• FIG. 1 is a diagram showing the overall configuration of the inspection system according to the first embodiment.
• the inspection system 100 includes a light 110, an image sensor 120, a handling mechanism 140, and a computer 150.
• the light 110 and the computer 150 are connected via a communication line 111
• the image sensor 120 and the computer 150 are connected via a communication line 121
• the handling mechanism 140 and the computer 150 are connected via a communication line 141.
• the lighting 110 emits light for inspection.
• the lighting 110 emits light onto the inspection object 130, defects present in the inspection object 130 can be made apparent.
• the imaging sensor 120 is a two-dimensional camera or the like that captures an image (digital image) of the inspection object 130.
• the imaging sensor 120 can capture an image of defects present in the inspection object 130.
• the handling mechanism 140 changes the relative positions of the lighting 110 and the imaging sensor 120, and the inspection object 130.
• the handling mechanism 140 can move at least one of the lighting 110 and the imaging sensor 120, and the inspection object 130.
• the computer 150 includes a memory 151, a processor 152, an input device 153, and a display device 154.
• Memory 151 is a storage device that stores various types of information.
• Memory 151 may be a non-volatile or volatile memory medium such as RAM (Random Access Memory) or ROM (Read Only Memory).
• Memory 151 may also be a rewritable storage medium such as a flash memory, a hard disk, or an SSD (Solid State Drive), or may be a USB (Universal Serial Bus) memory, a memory card, etc.
• RAM Random Access Memory
• ROM Read Only Memory
• Memory 151 may also be a rewritable storage medium such as a flash memory, a hard disk, or an SSD (Solid State Drive), or may be a USB (Universal Serial Bus) memory, a memory card, etc.
• USB Universal Serial Bus
• the memory 151 stores a recognition program 161 and a control program 162.
• the recognition program 161 is executed by the processor 152 to perform processing (recognition processing) to recognize defects from the image (digital data) captured by the imaging sensor 120.
• the recognition program 161 is capable of changing at least a portion of the recognition engine that executes the recognition processing and the parameters used in the recognition processing.
• the control program 162 is executed by the processor 152 to control the lighting 110, the imaging sensor 120, and the handling mechanism 140 in accordance with the inspection conditions.
• the processor 152 executes various processes according to the programs stored in the memory 151.
• the processor 152 may be, for example, a microprocessor, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a quantum processor, or any other semiconductor device capable of performing calculations.
• the input device 153 is, for example, a mouse, keyboard, etc., and accepts information input by the user.
• the display device 154 is, for example, a display, and displays and outputs a user interface including various types of information.
• the inspection system 100 at least some of the following inspection conditions can be changed: conditions related to illumination by the illumination 110 (illumination conditions), conditions related to imaging by the imaging sensor 120 (imaging conditions), and conditions related to operation by the handling mechanism 140 (handling conditions).
• the illumination conditions include, for example, at least one of the illumination type, illumination arrangement, illumination intensity, illumination time, and lighting cycle.
• the imaging conditions include at least one of the imaging sensor type, imaging sensor arrangement, aperture value, imaging time, and imaging cycle.
• the handling conditions include at least one of the handling mechanism type, position and orientation, and trajectory to reach the target position and orientation.
• FIG. 2 is a diagram showing the configuration of a computer that determines the inspection conditions according to the first embodiment.
• Computer 200 is an example of an inspection condition determination system, and includes memory 201, processor 202, input device 203, and display device 204. Computer 200 is connected to knowledge database 220 via a communication path. Computer 200 of this embodiment can acquire inspection images from computer 150, and can also transmit inspection conditions to computer 150.
• Memory 201 is a storage device that stores various types of information.
• Memory 201 may be a non-volatile or volatile memory medium such as RAM or ROM.
• Memory 201 may also be a rewritable storage medium such as a flash memory, a hard disk, or an SSD, or may be a USB memory, a memory card, etc.
• the memory 201 stores a recognition program 211, an inspection condition optimization program 212, an optical simulation program 213, an optical model calibration program 214, an inspection image change detection program 215, and a multiple machine inspection condition optimization program 216.
• the recognition program 211 is executed by the processor 202 to perform processing for recognizing defects from the inspection image.
• the inspection condition optimization program 212 is executed by the processor 202 to receive requests for inspection from users (customers) (customer requests: user requests) and perform processing (inspection condition optimization processing) to find optimal inspection conditions that satisfy the customer requests.
• the optical simulation program 213 is executed by the processor 202 to perform an inspection simulation using an optical model that can reproduce the inspection state of the inspection target in the inspection system 100.
• the optical model calibration program 214 is executed by the processor 202 to perform processing to calibrate the optical model (optical model calibration processing).
• the inspection image change detection program 215 is executed by the processor 202 to perform processing to detect changes in the inspection image.
• the multiple machine inspection condition optimization program 216 is executed by the processor 202 to perform processing to determine optimal inspection conditions in a manufacturing process using multiple inspection systems.
• the processor 202 executes various processes according to the programs stored in the memory 201.
• the processor 202 may be, for example, a microprocessor, a CPU, a GPU, an FPGA, a quantum processor, or any other semiconductor device capable of performing calculations.
• the input device 203 is, for example, a mouse, keyboard, etc., and accepts information input by the user.
• the display device 204 is, for example, a display, and displays and outputs a user interface including various information.
• the knowledge database 220 manages information about cases inspected by the inspection system 100, such as information about the image of the inspection target (inspection image) and the inspection conditions at that time. Note that while the knowledge database 220 is provided outside the computer 200, the present invention is not limited to this, and the knowledge database 220 may be provided within the computer 200.
• Customer requirements include multiple element evaluation indexes, including the inspection accuracy rate f1, false alarm rate f2, inspection time f3, inspection reproducibility f4, learning time f5, equipment cost f6, operating cost f7, equipment robustness f8, inspection versatility f9, life span f10, ease of assembly f11, required assembly precision f12, robustness against the environment f13, ease of maintenance f14, clarity of the judgment basis for the recognition process f15, and ease of recognition of the detection point f16.
• the inspection accuracy rate f1 is the rate at which good and bad items were correctly recognized relative to the number of items inspected.
• the inspection accuracy rate f1 can be expressed as the following formula (1).
• the false alarm rate f2 is the rate at which good products are mistakenly recognized as defective products relative to the total number of good products.
• the false alarm rate f2 can be expressed as the following formula (2).
• test accuracy rate f1 and the false alarm rate f2 are given as examples of indices relating to the accuracy of the test, but this is not limiting and other indices using more than one of TN, TP, FP, and FN may also be used.
• the inspection time f3 is the time related to the inspection, and may be, for example, the time required to inspect one inspection object, which is the sum of the movement time of the handling mechanism 140, the image capturing time of the image sensor 120, the data transfer time, and the time it takes for the recognition processing program to determine defects.
• Test reproducibility f4 is the reproducibility of a test when the same test subject is repeatedly tested.
• Learning time f5 is the time required for learning when a learning-type inspection algorithm is used as the recognition program 161, and is 0 when the recognition program 161 uses a non-learning-type inspection algorithm, for example, a method of detecting defects by comparing image features with a threshold value.
• Equipment cost f6 is the cost associated with constructing the inspection system 100.
• Operational cost f7 is the cost associated with operating the inspection system 100. Operational cost f7 includes electricity charges and maintenance costs for the inspection system 100.
• Equipment robustness f8 is an index that represents the resistance of the inspection system 100 to failure.
• Test versatility f9 is an index that indicates whether the test system 100 can be applied to various test objects.
• Lifespan f10 is the lifespan of the inspection system 100.
• Ease of assembly f11 is an index that represents the ease of assembly of the inspection system 100.
• the required assembly precision f12 is an index that represents the assembly precision required to achieve an inspection accuracy rate, etc.
• Environmental robustness f13 is an index that indicates whether the inspection results will not be distorted by environmental factors such as foreign matters, changes in temperature and humidity, and vibrations, and whether the inspection system 100 will not break down.
• Ease of maintenance f14 is an index that indicates how easy it is to clean and adjust the inspection system 100.
• the clarity of the judgment basis for the recognition process f15 is an index showing whether the recognition program 161 of the inspection system 100 is a recognition program that can explain the judgment basis for the recognition process. In general, recognition programs that use deep learning often have poor judgment basis and are less clear.
• the ease of recognizing the detected location f16 is an index that indicates whether or not it is possible to recognize which part of the image has a defect when the recognition program 161 detects a defect. Note that depending on the type of recognition engine of the recognition program 161, it may not be possible to recognize which part of the image has a defect, in which case the ease of recognition will be low.
• an index Ftarget containing multiple element evaluation indexes is generated as shown in formula (3) and used in the optimization process.
• wi is the weight corresponding to each element evaluation index gi. wi may be input via the GUI screen 900, and wi may also be reconfigured via the GUI screen 900.
• FIG. 3 is a diagram explaining the inspection conditions according to the first embodiment.
• the inspection conditions of the inspection system 100 include conditions related to lighting (illumination conditions), conditions related to imaging (imaging conditions), conditions related to handling (handling conditions), and conditions related to recognition (recognition conditions).
• Illumination conditions include lighting type, lighting arrangement, lighting intensity, lighting time, lighting cycle, etc.
• Imaging conditions include imaging sensor type, imaging sensor arrangement, aperture value, imaging time, imaging cycle, etc.
• Handling conditions include handling mechanism type, target position and orientation, route to the target position and orientation, etc.
• Recognition conditions include recognition engine type, recognition processing parameters, etc.
• each condition is represented by the symbols shown in FIG. 3.
• T indicates the type of object that is the subject of the condition.
• P indicates the three-dimensional position and orientation of the object.
• P corresponds to a vector (x, y, z, i, j, k) that combines (x, y, z) indicating the three-dimensional position and a unit vector (i, j, k) indicating the orientation.
• S indicates the period related to the object.
• H indicates the time related to the object.
• I indicates the illumination intensity (for example, watts)
• A indicates the aperture amount (aperture strength, for example, a value between 1 and 0) of the image sensor
• K indicates the trajectory of the handling mechanism 140 (for example, a set of vectors (x, y, z, i, j, k) of the three-dimensional coordinates and orientation of each position on the trajectory)
• M indicates the parameters (set of parameter values) of the recognition process.
• the subscripts (1) to (4) at the top right of the alphabet indicate values related to lighting, imaging, handling, and recognition, respectively.
• the subscript i at the bottom right of the alphabet indicates the number of arrangements of the same type of item.
• K has subscripts i and j at the bottom right, with i indicating the start point of the trajectory and j indicating the end point of the trajectory.
• T and M which relate to recognition, have subscripts i and l at the bottom right, with i indicating the location of the captured image and l indicating the number of the recognition process performed on that image.
• an appropriate combination of inspection conditions (T, P, S, H, I, A, K, M) is calculated and output based on the index Ftarget.
• the appropriate inspection conditions are determined by the index used, but for example, are inspection conditions where the value of the index Ftarget is minimum, maximum, extremely small, extremely large, a specified value, a gradual (stable) maximal value, or a gradual minimal value.
• inspection condition optimization process (first inspection condition optimization process) that optimizes the inspection conditions based on the actual inspection results by the inspection system 100.
• FIG. 4 is a flowchart of a first inspection condition optimization process according to one embodiment.
• the inspection condition optimization program 212 (more precisely, the processor 202 that executes the inspection condition optimization program 212) receives the customer indexes (f1 to f16) corresponding to the customer requirements, the target values of each element evaluation index, and the weight settings for each element evaluation index via the GUI screen 900 (S501).
• the inspection condition optimization program 212 creates an evaluation index (formula (3)) to be used in processing based on the comparison evaluation indexes (g1 to g16) corresponding to the customer indexes (f1 to f16) and the weight settings, and converts the target values of each element evaluation index into target values of the comparison evaluation index (comparison target values) (S502).
• the inspection condition optimization program 212 receives, for example, via the GUI screen 900, the consideration conditions to be used as constraints on the inspection conditions in the optimization process (S503).
• the consideration conditions include, for example, the type of lighting to be considered, the type of image sensor, the type of handling mechanism, the type of recognition engine, etc.
• the inspection condition optimization program 212 determines the inspection conditions to be considered according to the conditions under consideration, and sets the inspection system 100 according to the determined inspection conditions (target inspection conditions) (S504). Note that if the target inspection conditions include conditions that the inspection system 100 itself cannot set, the user of the inspection system 100 may set them.
• the inspection condition optimization program 212 causes the inspection system 100 to drive the handling mechanism 140 according to a predetermined trajectory (if the target inspection conditions include a trajectory condition, the trajectory corresponds to that condition) (S505), and causes the inspection system 100 to acquire an image of the inspection target 130 according to the target inspection conditions (S506).
• the inspection condition optimization program 212 determines whether or not all of the orbital movements have been completed and imaging by all of the imaging sensors 120 has been completed (S507), and if all of the orbital movements have been completed and imaging by all of the imaging sensors has not been completed (S507: NO), the process proceeds to step S505. This allows the orbital movements and imaging by the imaging sensors 120 to continue.
• the inspection condition optimization program 212 causes the recognition program 211 to evaluate the inspection performance using the captured images (S508), and calculates the evaluation index Ftarget based on the results of the performance evaluation, etc. (S509).
• the inspection condition optimization program 212 determines whether or not the evaluation indexes for the inspection conditions corresponding to all of the considered conditions have been calculated (S510), and if the evaluation indexes for the inspection conditions corresponding to all of the considered conditions have not been calculated (S510: NO), the inspection condition optimization program 212 proceeds to step S504 to execute the inspection conditions including the considered conditions for which the evaluation indexes have not been calculated.
• the inspection condition optimization program 212 determines optimal inspection conditions based on the values of the evaluation indexes for the multiple inspection conditions, and outputs the determined results (for example, displays them on the GUI screen 900) (S511). Note that in step S511, optimal inspection conditions are determined and output, but this is not limited thereto, and appropriate inspection conditions that satisfy a predetermined standard (for example, exceeding the target value of the evaluation index) may be determined and output.
• inspection condition optimization process (second inspection condition optimization process) that optimizes the inspection conditions based on the inspection results obtained by simulating the inspection in the inspection system 100 using an optical model.
• FIG. 5 is a flowchart of a second inspection condition optimization process according to one embodiment.
• the inspection condition optimization program 212 (more precisely, the processor 202 that executes the inspection condition optimization program 212) receives the customer indexes (f1 to f16) corresponding to the customer requirements, the target values of each element evaluation index, and the weight settings for each element evaluation index via the GUI screen 900 (S601).
• the inspection condition optimization program 212 creates an evaluation index (formula (3)) to be used for processing based on the comparison evaluation indexes (g1 to g16) corresponding to the customer indexes (f1 to f16) and the weight settings, and converts the target values of each element evaluation index into target values of the comparison evaluation index (comparison target values) (S602).
• the inspection condition optimization program 212 receives, for example, via the GUI screen 900, the consideration conditions to be used as constraints on the inspection conditions in the optimization process (S603).
• the consideration conditions include, for example, the type of lighting to be considered, the type of image sensor, the type of handling mechanism, the type of recognition engine, etc.
• the inspection condition optimization program 212 determines the inspection conditions to be considered according to the conditions to be considered, and the optical simulation program 213 creates an optical model that represents the inspection state of the inspection object 130 in the inspection system 100 based on the shape model 601 (three-dimensional shape model) of the inspection object 130, the surface texture 602 of the inspection object 130, and the inspection conditions (S604).
• the shape model 601 is a 3D-CAD model, which is design information for the product, or mesh or CAD data created from the results of measurement by a three-dimensional shape measuring device.
• the shape model 601 may be obtained, for example, from the memory 201 of the computer 200, or may be received from an external device.
• the surface texture 602 is information (surface texture information) on the light reflection characteristics determined by the material and minute irregularities not included in the shape model, which is expressed by BRDF (Bidirectional Reflectance Distribution Function) or the like, and may be, for example, input by a user or may be data measured by a surface texture measuring device.
• BRDF Bidirectional Reflectance Distribution Function
• the surface texture 602 may be obtained, for example, from the memory 201 of the computer 200, or may be received from an external device.
• the optical simulation program 213 moves the inspection object in the optical model according to a predetermined trajectory (if the target inspection conditions include a trajectory condition, the trajectory corresponds to that condition) (S605), and generates an image (estimated image) that is estimated to be captured of the inspection object 130 using the optical model (S606).
• the inspection condition optimization program 212 determines whether or not all of the orbital movements have been completed and the generation of estimated images estimated to be captured by all of the imaging sensors has been completed (S607), and if all of the orbital movements have been completed and the generation of estimated images by all of the imaging sensors has not been completed (S607: NO), the process proceeds to step S605. This allows the generation of estimated images that would result from the continued movement of the orbit.
• the inspection condition optimization program 212 causes the recognition program 211 to evaluate the inspection performance using the estimated images (S608), and calculates the evaluation index Ftarget based on the results of the performance evaluation, etc. (S609).
• the inspection condition optimization program 212 determines whether or not the evaluation indexes for the inspection conditions corresponding to all of the considered conditions have been calculated (S610), and if the evaluation indexes for the inspection conditions corresponding to all of the considered conditions have not been calculated (S610: NO), the inspection condition optimization program 212 proceeds to step S604 to execute the inspection conditions including the considered conditions for which the evaluation indexes have not been calculated.
• the inspection condition optimization program 212 determines optimal inspection conditions based on the values of the evaluation indexes for the multiple inspection conditions, and outputs the determined results (for example, displays them on the GUI screen 900) (S611). Note that in step S611, optimal inspection conditions are determined and output, but this is not limited thereto, and appropriate inspection conditions that satisfy a predetermined standard (for example, exceeding the target value of the evaluation index) may be determined and output.
• the inspection conditions are comprehensively changed to perform optimization, but the optimization method is not limited to this.
• an efficient search such as experimental design or Bayesian optimization may be performed.
• the Benders decomposition method may be used to search for optimal conditions by separating the conditions for the linear part of the device configuration (conditions represented by the symbol T in Figure 3) from the conditions for the nonlinear part (parameters).
• the lighting type, arrangement, and type of recognition engine may be limited based on similar cases in the knowledge database 220. In this case, the similar cases may be those found by the user or those found using a machine learning matching function.
• optical model calibration process for calibrating the optical model will be described.
• the optical model calibration process is executed, for example, before the second inspection condition optimization process shown in FIG. 5.
• FIG. 6 is a flowchart of the optical model calibration process according to one embodiment.
• the optical model calibration program 214 receives calibration condition settings from the user via the GUI screen, including the inspection conditions of the inspection system 100 to be used for calibration (e.g., the current inspection conditions of the inspection system 100) and the change conditions of the optical model (e.g., the change range of the reflectance of the inspection target (0.2 to 0.8, etc.)) (S701).
• the inspection conditions of the inspection system 100 to be used for calibration e.g., the current inspection conditions of the inspection system 100
• the change conditions of the optical model e.g., the change range of the reflectance of the inspection target (0.2 to 0.8, etc.)
• the optical model calibration program 214 creates an evaluation index for calibration (calibration evaluation index) (S702).
• the calibration evaluation index is a function that combines feature quantities to be compared in calibration, and may be, for example, the average value or sum of variances of the differences for each pixel between an estimated image (described below) and an acquired image.
• the calibration evaluation index may be defined by the user, or may be selected or generated based on examples.
• the optical model calibration program 214 executes a process (S711 to S714) for obtaining an estimated image using the optical model, and a process (S721 to S724) for actually obtaining an image using the inspection system 100.
• the optical model calibration program 214 determines the conditions of the optical model to be considered based on the calibration conditions (e.g., a predetermined value within the change range of the reflectance of the inspection object), and the optical simulation program 213 creates an optical model that represents the inspection state of the inspection object 130 in the inspection system 100 based on the setting conditions in the calibration conditions and the determined conditions of the optical model (S711).
• the calibration conditions e.g., a predetermined value within the change range of the reflectance of the inspection object
• the optical simulation program 213 moves the inspection object in the optical model according to a predetermined trajectory (if the target inspection conditions include a trajectory condition, the trajectory corresponds to that condition) (S712), and generates an image (estimated image) estimated to be captured of the inspection object 130 using the optical model (S713).
• the optical model calibration program 214 determines whether or not all movement of the orbit has been completed and generation of estimated images estimated to be captured by all imaging sensors has been completed (S714), and if all movement of the orbit has been completed and generation of estimated images by all imaging sensors has not been completed (S714: NO), the process proceeds to step S712. This allows generation of estimated images if movement of the orbit were continued.
• the optical model calibration program 214 advances the process to step S731.
• the optical model calibration program 214 first sets the inspection system 100 according to the inspection conditions (target inspection conditions) included in the calibration conditions (S721). Note that if the target inspection conditions include a condition that the inspection system 100 itself cannot set, the user of the inspection system 100 may set it.
• the optical model calibration program 214 causes the inspection system 100 to drive the handling mechanism according to a predetermined trajectory (if the target inspection conditions include a trajectory condition, the trajectory corresponds to that condition) (S722), and causes an image (actual image) of the inspection target 130 to be acquired according to the target inspection conditions (S723).
• the optical model calibration program 214 determines whether or not all movement of the orbit has been completed and imaging by all the imaging sensors has been completed (S724), and if all movement of the orbit has been completed and imaging by all the imaging sensors has not been completed (S724: NO), the process proceeds to step S722. This allows movement of the orbit and imaging by the imaging sensors to continue.
• the optical model calibration program 214 advances the process to step S731.
• step S731 the optical model calibration program 214 compares the features of the estimated image and the actual image, and calculates a calibration evaluation index for the optical model used.
• the optical model calibration program 214 determines whether changes have been made to all optical models that correspond to the change conditions included in the calibration conditions (S732).
• the optical model calibration program 214 changes the conditions for the optical models based on the change conditions (S733), and proceeds to step S711 to perform processing to obtain estimated images for other optical models.
• the optical model calibration program 214 identifies the parameters of the optical model that adequately represent the inspection of the actual inspection object based on the evaluation index (e.g., the reflectance of the inspection object), and outputs the parameters of the optical model as calibration data (S734).
• the parameters of the optical model that adequately represent the inspection are determined by the calibration evaluation index used, and are, for example, parameters that satisfy any of the following for which the value of the calibration evaluation index is minimum, maximum, extremely small, extremely large, a predetermined value, a gradual (stable) maximal value, or a gradual maximal value.
• the optical simulation program 213 When the second inspection condition optimization process shown in FIG. 5 is subsequently executed, the optical simulation program 213 generates an optical model based on the calibration data, and generates an optical model that matches the inspection conditions in the actual inspection system 100, making it possible to generate a highly accurate estimated image.
• FIG. 7 shows a GUI screen according to one embodiment.
• the GUI screen 900 includes an input area 910 and an output area 920.
• the input area 910 includes a customer requirement input area 911, a consideration condition input area 912, and a weight input area 913.
• the customer requirement input area 911 is an area for inputting customer requirements, and includes a symbol indicating the element evaluation index of the customer requirement, a customer requirement indicating the content of the element evaluation index, and a target value field for inputting the target value of the element evaluation index.
• the target value is set to unnecessary, it is possible to set it to an evaluation index that does not take into account the corresponding element evaluation index. In other words, it is possible to prevent the value of the corresponding element evaluation index from being taken into account in formula (3).
• the consideration condition input area 912 is an area for inputting conditions (consideration conditions) to be considered in the inspection conditions of the inspection system 100.
• the weight input area 913 is an area for inputting weight values for the values of each element evaluation index. In the weight input area 913, it is possible to input weight values and a setting as to whether the weight values are fixed or variable.
• the processing result list 921 includes entries of appropriate inspection conditions detected in the inspection condition optimization process. For example, when the weights wi of all element evaluation indices are input as fixed values, one entry is displayed, and when the weights wi of the element evaluation indices are input as variable values, an entry of appropriate inspection conditions corresponding to each set of weights of each element evaluation index is displayed.
• the entries in the processing result list 921 include fields for element evaluation index/weighting 921a and inspection conditions 921b.
• the element evaluation index/weighting 921a displays the value for each element evaluation index and the weighting for that element evaluation index.
• the inspection conditions 921b store the values of each inspection condition of the inspection system 100 (inspection condition information).
• the appropriate lighting, imaging sensor, and recognition program may differ for each part of the inspection object.
• the inspection object is a cut-out part of a casting
• the surface properties will be different between the part where the casting surface remains and the cut-out part, and the image characteristics will be significantly different. Therefore, changing the type of lighting, lighting angle, lighting intensity, or changing the parameters of the recognition program can lead to an improvement in the inspection success rate.
• the inspection area of the inspection object 130 may be divided into multiple parts for inspection. The case of dividing into multiple inspection areas for inspection is described below.
• FIG. 8 is a diagram explaining the inspection conditions when dividing the inspection area according to one embodiment. Note that FIG. 8 shows an example in which the inspection object 130 is divided into a flat surface 131 on the upper surface and a cylindrical surface 132 for inspection, and the flat surface 131 is imaged by the illumination 110a and the imaging sensor 120a, and the cylindrical surface 132 is imaged by the illumination 110b and the imaging sensor 120b for inspection.
• the inspection conditions of the inspection system 100 include conditions related to lighting (illumination conditions), conditions related to imaging (imaging conditions), conditions related to handling (handling conditions), and conditions related to recognition (recognition conditions).
• Illumination conditions include lighting type, lighting arrangement, lighting intensity, lighting time, lighting cycle, etc.
• Imaging conditions include imaging sensor type, imaging sensor arrangement, aperture value, imaging time, imaging cycle, etc.
• Handling conditions include handling mechanism type, target position and orientation, route to the target position and orientation, etc.
• Recognition conditions include recognition engine type, recognition processing parameters, etc.
• m is a number that represents the surface to be inspected (area to be inspected). In this example, if the surface to be inspected is a flat surface 131, m is 1, and if the surface to be inspected is a cylindrical surface 132, m is 2.
• the inspection condition optimization program 212 determines a division plan for dividing the inspection object 130 into multiple inspection areas (partial inspection areas) based on the processing conditions of the inspection object 130 (division plan determination step).
• the inspection condition optimization program 212 executes the inspection condition optimization process of FIG. 4 or FIG. 5 for each inspection area in the division plan, determines the optimal inspection conditions, and calculates the overall evaluation index value at the time of division (division plan evaluation index value) by summing the evaluation index values of each inspection area (division plan evaluation index calculation step).
• the inspection condition optimization program 212 performs a division plan determination step and a division plan evaluation index calculation step to determine the inspection conditions for multiple division plans for the inspection target 130 and calculate the respective division plan evaluation index values.
• the inspection condition optimization program 212 determines an appropriate (e.g., optimal) division plan and inspection conditions that satisfy predetermined conditions based on the division plan evaluation indexes of the multiple division plans, and outputs the determination result.
• This process makes it possible to determine an appropriate division plan for the inspection object 130 and appropriate inspection conditions for each inspection area in the division plan.
• FIG. 9 is a diagram illustrating changes in an inspection image according to one embodiment.
• FIG. 9(A) shows the state in which an image of an inspection object is captured
• FIG. 9(B) shows an inspection image immediately after the inspection system 100 starts operating
• FIG. 9(C) shows an inspection image after a certain amount of time has passed since the inspection system 100 started operating.
• the object to be inspected 130 is illuminated with light 110 from above and an image is taken by the image sensor 120.
• the inspection image of the inspection object 130 is a clear image of the inspection object 130, as shown in FIG. 9(B).
• the inspection image of the inspection object 130 may include the scratch 1031, as shown in FIG. 9C. Also, if the lens becomes dirty, the image may become dark overall. Even if the inspection object 130 is the same product, the color and brightness of the surface 1032 of the inspection object 130 may change over time due to changes in the materials used to manufacture the inspection object 130, etc.
• the inspection conditions in the inspection system 100 may no longer be optimal.
• the inspection system 100 performs a process to redetermine the inspection conditions in such cases (inspection condition redetermining process).
• FIG. 10 is a flowchart of the inspection condition redetermining process according to one embodiment.
• the inspection image change detection program 215 receives a tolerance for variation in the feature in the inspection image (S1101).
• the feature may be a statistical quantity such as the average brightness or brightness variance in the inspection image or a specified area of the inspection image.
• the tolerance may be received from the user when the inspection system 100 starts operating or after a specified number of inspections have been performed.
• the tolerance may be a default value.
• the inspection image change detection program 215 calculates image features based on the inspection image of the inspection object captured and stored by the inspection system 100 (S1102).
• the inspection image change detection program 215 determines whether the calculated feature amount exceeds the allowable value (S1103).
• the inspection image change detection program 215 issues an alert and executes the inspection condition optimization program 212 to execute the inspection condition optimization process of FIG. 4 or FIG. 5 (S1104), and the inspection image change detection program 215 outputs the result (S1105).
• some of the inspection conditions such as the illumination type, image sensor type, and handling mechanism may be fixed to their current state. In this way, the inspection conditions can be set so that the configuration already in use in the inspection system 100 can be used as is, and additional investment in the inspection system 100 can be prevented.
• the inspection image change detection program 215 outputs a result indicating that there is no need to execute the inspection condition optimization process (S1105).
• FIG. 11 is a diagram illustrating the manufacturing process according to one embodiment.
• This manufacturing process includes inspection process 1, processing process 1, inspection process 2, processing process 2, inspection process 3, processing process 3, and inspection process 4.
• inspection process 1 inspection system 100-1 inspects the received material
• inspection process 2 inspection system 100-2 inspects the processed product processed in processing process 1
• inspection process 3 inspection system 100-3 inspects the processed product processed in processing process 2
• inspection system 100-4 inspects the processed product processed in processing process 3.
• the information related to manufacturing in the processing processes is represented by Ak as the process capacity, Tk as the yield, Sk as the manufacturing speed, and Ck as the manufacturing cost. Note that k represents the number of the processing process.
• the inspection condition optimization process by the multiple machine inspection condition optimization program 216 in this example will be explained.
• the differences in processing between the multiple machine inspection condition optimization program 216 and the inspection condition optimization program 212 will be explained with reference to FIG. 3 (FIG. 4).
• the process capacity Ak, yield Tk, manufacturing cost Sk, and manufacturing cost Ck for each processing step are known or assumed and have been set or input.
• the evaluation index is the linear sum of the evaluation index of each processing step and the information on the manufacturing status of the processing step, but the present invention is not limited to this. For example, it may be the product or the sum of the reciprocals of each index.
• step S511 the multiple-machine inspection condition optimization program 216 determines optimal inspection conditions based on the values of the evaluation index (equation (4)) under multiple inspection conditions, and outputs the determined results.
• This process for optimizing conditions in the manufacturing process combines customer requirements for the inspection process and information about the processing process into a single evaluation index, making it possible to determine appropriate inspection conditions for the entire manufacturing process.
• the optimal inspection conditions are determined, but the present invention is not limited to this, and it is also possible to determine inspection conditions that satisfy predetermined conditions, rather than being limited to the optimal inspection conditions.
• the computer 200 is capable of executing the inspection condition optimization processes of FIG. 3 and FIG. 4, but it may be possible to execute only one of the inspection condition optimization processes.
• the computers 150 and 200 are separate computers, but for example, the functions of the computer 200 may be incorporated into the computer 150.
• the processing performed by the processor may be performed by a hardware circuit.
• the program in the above embodiment may be installed from a program source.
• the program source may be a program distribution server or a recording medium (e.g., a portable recording medium).

Landscapes

• Physics & Mathematics (AREA)
• Health & Medical Sciences (AREA)
• Life Sciences & Earth Sciences (AREA)
• Chemical & Material Sciences (AREA)
• Analytical Chemistry (AREA)
• Biochemistry (AREA)
• General Health & Medical Sciences (AREA)
• General Physics & Mathematics (AREA)
• Immunology (AREA)
• Pathology (AREA)
• Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)
• Image Analysis (AREA)

Priority Applications (1)

Applications Claiming Priority (2)

Publications (1)

Family

ID=90669118

Family Applications (1)

Country Status (3)

Families Citing this family (1)

Citations (10)

• 2022
  • 2022-10-11 JP JP2022163558A patent/JP2024056561A/ja active Pending
• 2023
  • 2023-09-20 CN CN202380065239.4A patent/CN119856045A/zh active Pending
  • 2023-09-20 WO PCT/JP2023/034056 patent/WO2024080087A1/ja not_active Ceased

Patent Citations (10)

Also Published As

Similar Documents

Legal Events