Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/0002—Inspection of images, e.g. flaw detection
  • G06T7/0004—Industrial image inspection
  • G06T7/001—Industrial image inspection using an image reference approach
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
  • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
  • G01N21/84—Systems specially adapted for particular applications
  • G01N21/88—Investigating the presence of flaws or contamination
  • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
  • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
  • G01N21/84—Systems specially adapted for particular applications
  • G01N21/88—Investigating the presence of flaws or contamination
  • G01N21/95—Investigating the presence of flaws or contamination characterised by the material or shape of the object to be examined
  • G01N21/9515—Objects of complex shape, e.g. examined with use of a surface follower device
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/23—Clustering techniques
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/0002—Inspection of images, e.g. flaw detection
  • G06T7/0004—Industrial image inspection
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/70—Determining position or orientation of objects or cameras
  • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
  • G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
  • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
  • G01N21/84—Systems specially adapted for particular applications
  • G01N21/88—Investigating the presence of flaws or contamination
  • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
  • G01N2021/8854—Grading and classifying of flaws
  • G01N2021/888—Marking defects
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
  • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
  • G01N21/84—Systems specially adapted for particular applications
  • G01N21/88—Investigating the presence of flaws or contamination
  • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
  • G01N2021/8887—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges based on image processing techniques
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10004—Still image; Photographic image
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20081—Training; Learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20092—Interactive image processing based on input by user
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20212—Image combination
  • G06T2207/20216—Image averaging
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30108—Industrial image inspection
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30108—Industrial image inspection
  • G06T2207/30164—Workpiece; Machine component

Definitions

• Inspection during production processes can be instrumental in ensuring the quality of the production. For example, inspection during production processes at manufacturing plants helps control the quality of products by identifying defects and then acting upon this identification, for example, by fixing the defect or discarding the defective part.
• the process of defect detection is essential for quality assurance (QA), gating and sorting on production lines, and is consequently useful in improving productivity, improving production processes and working procedures, reducing defect rates, and reducing re-work and waste.
• embodiments of the invention enable presenting a large amount of inspected items for user confirmation prior to beginning the inspection stage, thereby efficiently utilizing the user’s time and streamlining the inspection process.
• the inspection stage may commence with a much reduced risk of incorrect detection, thereby providing an improved inspection process.
• FIG. 2 is a schematic illustration of a system for visual inspection, according to an embodiment of the invention.
• FIG. 3B is a schematic illustration of a method for proceeding from the set-up stage to the inspection stage, according to embodiments of the invention.
• FIG. 4 is a schematic illustration of a method for a visual inspection line set-up process using images that are not necessarily defect-free, according to an embodiment of the invention
• FIG. 5 is a schematic illustration of a method for determining a cluster threshold, according to embodiments of the invention.
• the analysis of the set up images is used to determine a spatial range in which the defect free item shows no perspective distortion.
• the level of perspective distortion between items in different images can be analyzed, for example, by detecting regions in an item which do not have corresponding features between the set up images, by analyzing the intersection location and angles between the item’s borders or marked areas of interest on the item, etc.
• the borders of the spatial range may be calculated by comparing two (or more) set up images (in which items may be positioned and/or oriented differently) and determining which of the images show perspective distortion and which do not.
• a processor can detect a second item of the same type and perform inspection tasks, even if the second item was not previously learned by the processor. This allows the processor to detect when a new item (of the same type) is imaged, and then to analyze the new item, for example, to search for a defect on an inspected item, based on the analysis of set up images. Analyzing a new item typically includes applying inspection algorithms, e.g., defect detection algorithms, which, in one embodiment, include detecting the item in the image and comparing image data of the detected item to image data of reference images.
• inspection algorithms e.g., defect detection algorithms, which, in one embodiment, include detecting the item in the image and comparing image data of the detected item to image data of reference images.
• reference images may show a low level of registration quality or may not enable registration at all.
• reference images may show low probability of object detection or no object detection.
• reference images may show low correlation when compared to each other and/or objects or marked regions of interest within the images may show low correlation when compared to each other. These cases may indicate that the objects in these images are too different for comparison. However, since these images are obtained during the set- up stage, which is typically closely overseen by a user, these images indicate a real-life situation on the inspection line and similar images will probably be obtained during the inspection stage, which is less closely overseen by a user.
• Embodiments of the invention alleviate this risk by ensuring that all reference images are grouped to a cluster in which there are enough other similar reference images ensuring that comparison, registration and other analysis of the reference images in the cluster can be performed and ensuring that there will be a reference group for all inspection images, even if the objects in the inspection images have different visual appearances.
• Appearance (or visual appearance) of an object may include any feature visible to a camera.
• appearance may include spatial features of the object in the image (e.g., positioning and/or rotation of the object or parts of the object within the field of view (FOV) of the camera, shape and/or size of the object, visible patterns or markings on the object, etc.).
• FOV field of view
• the inspection stage may commence with a much reduced risk of incorrect detection, thereby providing an improved inspection process.
• an object 10 is detected in a number of set-up images 11, 12, 13 and 14 of an inspection line. In each set-up image the object 10 conforms to a criterion.
• a criterion may include, for example, a spatial feature of the object 10 in the image, e.g., a position or angle of placement of the object within the image (such as its position relative to the camera FOV, its rotation in three axes relative to the camera FOV, its shape or scale relative to the camera FOV, etc.).
• a criterion includes one or more visual features, such as, visible marks on the object. Other properties of the objects and/or images may be used as criteria.
• the images 11, 12, 13 and 14 are grouped into clusters (e.g., cluster A and cluster B) according to values of the criterion.
• the criterion for grouping the images is positioning of the object 10. Images 11 and 12, in which the object 10 is positioned similarly, i.e., can be defined by similar angle values, are both assigned to cluster A, whereas images 13 and 14, in which object 10 is positioned similarly to each other but differently from the positioning of object 10 in images 11 and 12, are assigned to cluster B.
• an inspection image is compared to a cluster of set up images based on the value of the criterion of the object in the inspection image.
• object 10 is detected in an inspection image 15.
• the positioning of the object 10 in inspection image 15 e.g., the angle of object 10 in relation to a point within the image
• the images in cluster B can be used as an appropriate reference for the object in inspection image 15, ensuring that there are reference images similar enough to inspection image 15 so that they can be used as a reference for detecting the object 10 and/or detecting defects on the object 10 in the inspection image 15.
• the inspection image 15 is compared to the images assigned to cluster B in order to detect a defect on object 10 based on the comparison.
• “same-type items” or“same-type objects” refers to items or objects which are of the same physical makeup and are similar to each other in shape and dimensions and possibly color and other physical features.
• items of a single production series, batch of same-type items or batch of items in the same stage in its production line may be“same-type items”. For example, if the inspected items are sanitary products, different sink bowls of the same batch are same-type items.
• a defect may include, for example, a visible flaw on the surface of the item, an undesirable size of the item or part of the item, an undesirable shape or color of the item or part of the item, an undesirable number of parts of the item, a wrong or missing assembly of interfaces of the item, a broken or burned part, and an incorrect alignment of the item or parts of the item, a wrong or defected barcode, and in general, any difference between the defect-free sample and the inspected item, which would be evident from the images to a user, namely, a human inspector.
• a defect may include flaws which are visible only in enlarged or high resolution images, e.g., images obtained by microscopes or other specialized cameras.
• An exemplary system which may be used for automated visual inspection of an item on an inspection line includes a processor 102 in communication with one or more camera(s) 103 and with a device, such as a user interface device 106 and/or other devices, such as storage device 108.
• Camera(s) 103 which are configured to obtain an image of an inspection line 105, are typically placed in relation to the inspection line 105 (e.g., a conveyer belt), such that items (e.g., item 104) placed on the inspection line 105 are within the FOV 103’ of the camera 103.
• Camera 103 may include a CCD or CMOS or other appropriate chip.
• the camera 103 may be a 2D or 3D camera.
• the camera 103 may include a standard camera provided, for example, with mobile devices such as smart-phones or tablets.
• the camera 103 is a specialized camera, e.g., a camera for obtaining high resolution images.
• the system may also include a light source, such as an LED or other appropriate light source, to illuminate the camera FOV 103’, e.g., to illuminate item 104 on the inspection line 105.
• a light source such as an LED or other appropriate light source
• Processor 102 receives image data (which may include data such as pixel values that represent the intensity of reflected light as well as partial or full images or videos) of objects on the inspection line 105 from the one or more camera(s) 103 and runs processes according to embodiments of the invention.
• image data which may include data such as pixel values that represent the intensity of reflected light as well as partial or full images or videos
• Processor 102 is typically in communication with a memory unit 112.
• Memory unit 112 may store at least part of the image data received from camera(s) 103.
• Memory unit 112 may include, for example, a random access memory (RAM), a dynamic RAM (DRAM), a flash memory, a volatile memory, a non-volatile memory, a cache memory, a buffer, a short term memory unit, a long term memory unit, or other suitable memory units or storage units.
• RAM random access memory
• DRAM dynamic RAM
• flash memory a volatile memory
• non-volatile memory a non-volatile memory
• cache memory a buffer
• a short term memory unit a long term memory unit
• other suitable memory units or storage units or storage units.
• processor 102 receives images (e.g., set-up images) of an object and groups the images into a cluster based on appearance of the object in each image.
• images e.g., set-up images
• processor 102 assigns images to a cluster and compared at least to the set up images grouped to the cluster. This ensures a group of images of objects of the same type which are comparable to the current inspection image without degraded results to the perspective distortion. Defects on the object in the in the successive image can then be detected based on the comparison.
• processor 102 causes the detected defects to be displayed (e.g., via user interface device 106), e.g., for a user’s approval.
• the clusters are stored in a reference image database (which may be maintained, for example, in storage device 108), which is later used for defect detection in inspection images.
• image data is maintained within the database such that reference images are linked to a cluster based on one or more criterion.
• set up images 11 and 12 may be stored in the reference image database with a link to the criterion defining cluster A (e.g., position, which can be expressed as a value of an angle or range of angles) and set-up images 13 and 14 may be stored in the reference image database with a link to the criterion defining cluster B.
• individual images of a particular cluster may be displayed. For example, a user may click on a displayed representative of a cluster (e.g., representative 16) to open a new window or screen which displays individual images that were assigned to that cluster.
• a user interface enables clusters to be approved or disapproved (e.g., deleted) as a whole, and/or individual images within a cluster to be approved and/or disapproved by a user.
• a cluster may be determined to be in an “incomplete” status, whereas a cluster including enough images is determined to be in “completed” status.
• a system for visual inspection includes a processor 102 in communication with a display (e.g., a monitor of user interface device 106), to receive an image of an object on an inspection line, detect the object in the image and assign the image to a cluster based on appearance of the object in the image (e.g., based on a spatial feature of the object in the image).
• Processor 102 may determine the status of the cluster and cause display of a representative of the cluster with an indication of the status of the cluster.
• the indication of the status of the cluster may include a visual characteristic that is different for each different status.
• the visual characteristic may be color such that a representative of a completed cluster may be displayed in a different color than a representative of an incomplete cluster.
• images are assigned to a cluster, during a set-up stage, until the cluster includes enough reference images and has achieved optimization of imaging parameters, after which the process proceeds to the inspection stage.
• a complete representation of the object is achieved when each new inspection image has a group of reference images in the cluster that can be used as a distortion-less reference to the inspection image and which are indicative of the tolerance and variations typical to the imaged item.
• a visual inspection method includes assigning images of an object to a cluster (312), typically based on appearance of the object in each image. For each cluster it is determined if a complete representation of the object has been achieved (314). If a complete representation of the object has been achieved, the cluster may proceed to the inspection stage (316). Namely, a successive image assigned to the cluster will be inspected for defects, e.g., by comparing the successive image to the images in the cluster.
• an image may be obtained which is assigned to a cluster for which a complete representation of the object has not yet been achieved and therefore cannot be inspected for defects.
• the image may be stored until enough images are assigned to the cluster to achieve complete representation of the object, while additional images that are assigned to completed clusters, may be inspected.
• the cluster is completed.
• the completed cluster and/or images from the completed cluster may be displayed (e.g., via user interface device 106) for user confirmation.
• images may be compared to the completed cluster retroactively, to detect defects in images that were obtained prior to the completion of the cluster. Thus, defects may be detected retroactively in images assigned to the cluster.
• determining if a complete representation of the object is achieved includes comparing images assigned to the cluster to each other. For example, determining if each image assigned to the cluster can be used as a distortion-less reference to all other images in the cluster, can be determined by comparing the images of the cluster to each other.
• clusters include a predetermined number of images.
• the predetermined number may be specific to characteristics of an object (e.g., clusters for a 2D object may require less reference images than clusters for a 3D object, which for example can be defined by using depth-from-focus techniques).
• determining if a complete representation of the object is achieved may include determining a number of images in the cluster. For example, if there is a predetermined number of images in a cluster it may be determined that a complete representation of the object has been achieved.
• Methods according to embodiments of the invention enable an improved visual inspection method in which some images may proceed to inspection, based on the cluster the images are assigned to, even before all the possible reference images for the object have been obtained, namely, before the set up stage is complete. This enables inspection to begin early and proceed with minimal interruption and with user involvement being concentrated to specific points during the process, thereby greatly streamlining the inspection process.
• the clusters may be displayed to a user for approval prior to updating the database of reference images.
• the first set of images includes only images of defect-free items, whereas the second set of images includes images that are not necessarily defect-free.
• images that are not necessarily defect-free can be used in a set-up stage, according to embodiments of the invention.
• the second set of images may be analyzed for defects, based on the visual appearance of the object determined from the first set of images.
• a visual appearance of an object is determined based on a first set of images of defect-free objects (402).
• the defects detected in step (406) are then displayed to a user for approval (410).
• the clusters can also be displayed to the user for approval.
• Clusters may be created, e.g., by processor 102, based on predefined criteria and/or based on criteria detected from the images.
• a cluster threshold is determined, which is the threshold images are checked against, to determine which cluster they will be assigned to.
• a cluster threshold may be determined based on the difference between the image and the confirmed set of images (508). Typically, the cluster threshold is determined based on the criterion and/or based on the value of the criterion.
• a cluster threshold may include a value or range of values of angles, a size and/or shape of the object, a number or range of numbers of visual marks, a size or range of sizes of visual marks, a location or range of locations of visual marks within the object, a color or range of colors of visual marks, and so on.
• a cluster threshold may include a range between two limits, e.g., a highest and lowest limit.
• methods according to embodiments of the invention may include the steps of grouping a pre- determined number of images into a cluster wherein the threshold for the cluster is determined based on the pre-determined number.
• same-type objects rotated in relation to the objects in the confirmed set of images, by an angle that is within a first range may define or may be assigned to a first cluster.
• Same-type objects having a pattern of red circles may define or may be assigned to a second cluster and same-type objects having a pattern of green circles may define or may be assigned to a third cluster, and so on.
• the method includes assigning a first image to a first cluster and assigning a second image to a second cluster if the difference between a value of the criterion in the first image and a value of the criterion in the second image is above a threshold.
• the criterion and/or value of the image is compared to the threshold of cluster A. If the criterion (and/or value) of the image is within (or above or below) the threshold of cluster A (610) then the image is assigned to cluster A (608). However, if the criterion (and/or value) of the image is the not within (or above or below) the threshold of cluster A (610), then the image is assigned to another cluster (e.g., cluster B) (612), for example, if the criterion and its’ value are compatible with the threshold of cluster B.
• cluster B e.g., cluster B
• a criterion and/or threshold of cluster may be pre-determined or dynamic, e.g., dependent on the specific type of object being inspected.
• a two dimensional object 702 having a pattern 712 on it is shown in two images, 71 and 72.
• the object 702 is rotated by 90° in image 72 compared with image 71 (as illustrated by the dashed arrow), however, the pattern 712 is visible in both images 71 and 72.
• images 71 and 72 may conceivably be assigned to a single cluster, which is defined by a broad range of visible patterns on the object.
• object 703 is a three dimensional object having pattern 713 on a first surface (a) of the object and pattern 714 on a second surface (b) of the object, as visible in image 73.
• the object 703 in image 74 is rotated along its longitudinal axis by 90° compared to object 703 in image 73 (as illustrated by the dashed arrow). In this case, the rotation causes the first surface (a) of object 703 to be occluded while the second surface (b) stays visible. Consequently, pattern 714 is visible but pattern 713 is not visible in image 74. Since different patterns are visible in images 73 and 74, these images would not be assigned to the same cluster.
• an outline may be created around an object (possibly an average or other representing object) and the outline is used as a cluster threshold. For example, all images in which objects fit within the outline, may be assigned to the same cluster.
• An outline representing a 2D object may loosely follow the contour of the object, whereas the outline representing a 3D object, will typically more closely follow the contour of the object.
• Embodiments of the invention enable to create, with minimal user involvement, a broad database of reference images to improve performance of inspection tasks.

Landscapes

• Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Quality & Reliability (AREA)
• Life Sciences & Earth Sciences (AREA)
• Data Mining & Analysis (AREA)
• Immunology (AREA)
• Pathology (AREA)
• General Health & Medical Sciences (AREA)
• Biochemistry (AREA)
• Analytical Chemistry (AREA)
• Chemical & Material Sciences (AREA)
• Health & Medical Sciences (AREA)
• General Engineering & Computer Science (AREA)
• Signal Processing (AREA)
• Bioinformatics & Cheminformatics (AREA)
• Evolutionary Biology (AREA)
• Evolutionary Computation (AREA)
• Bioinformatics & Computational Biology (AREA)
• Artificial Intelligence (AREA)
• Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)
• Image Processing (AREA)
• Image Analysis (AREA)

Priority Applications (8)

Applications Claiming Priority (4)

Publications (1)

Family

ID=65910786

Family Applications (1)

Country Status (10)

Cited By (2)

Families Citing this family (3)

Citations (6)

Family Cites Families (30)

• 2018
  • 2018-11-18 IL IL263097A patent/IL263097B2/en unknown
• 2019
  • 2019-11-18 WO PCT/IL2019/051260 patent/WO2020100146A1/en not_active Ceased
  • 2019-11-18 CA CA3117917A patent/CA3117917A1/en active Pending
  • 2019-11-18 KR KR1020217016265A patent/KR20210091189A/ko not_active Ceased
  • 2019-11-18 EP EP19885049.7A patent/EP3881059A4/en active Pending
  • 2019-11-18 CN CN201980074977.9A patent/CN113039432A/zh active Pending
  • 2019-11-18 JP JP2021527065A patent/JP2022507678A/ja active Pending
  • 2019-11-18 MX MX2021005739A patent/MX2021005739A/es unknown
  • 2019-11-18 US US17/294,752 patent/US12524866B2/en active Active
  • 2019-11-18 BR BR112021009487-3A patent/BR112021009487A2/pt not_active IP Right Cessation

Patent Citations (6)

Also Published As

Similar Documents

Legal Events