WO2024258811A1 - Methods and systems for detecting defects using topological persistence features - Google Patents
Methods and systems for detecting defects using topological persistence features Download PDFInfo
- Publication number
- WO2024258811A1 WO2024258811A1 PCT/US2024/033334 US2024033334W WO2024258811A1 WO 2024258811 A1 WO2024258811 A1 WO 2024258811A1 US 2024033334 W US2024033334 W US 2024033334W WO 2024258811 A1 WO2024258811 A1 WO 2024258811A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- topological
- feature
- persistence
- birth
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30164—Workpiece; Machine component
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/40—Analysis of texture
- G06T7/41—Analysis of texture based on statistical description of texture
Definitions
- defect detection systems are used to automatically detect defects in manufactured components that move through the assembly line. Some defect detection systems capture an image of a manufactured component while the manufactured component enters into, is processed through, or exits the assembly line and process the captured image to determine whether the image indicates a defect in the manufactured component.
- the present disclosure relates to techniques for monitoring manufacturing defects in manufactured components.
- the techniques provide a computerized method for monitoring for manufacturing defects.
- the method may include: receiving a two-dimensional (2D) image of a manufactured component; computing a respective set of values indicating a life for each topological feature of a plurality of topological features in the 2D image; determining a digital feature based on the computed sets of values for the plurality of topological features; and determining, based on the digital feature, whether the 2D image is indicative of a defect in the manufactured component.
- 2D two-dimensional
- each topological feature of the plurality of topological features for a 2D image may include a shape comprising an isolated block or a block in a first color completely enclosed by a region in a second color, wherein each pixel in the shape is below a filtration threshold.
- the set of values for each topological feature of the plurality of topological features in the 2D image may include a set of birth and death coordinates each respectively representing a birth and a death of the topological feature associated with a filtration threshold.
- the method in determining the digital feature, may determine a persistence statistics vector, a persistence curve vector, or a combination thereof, based on the computed sets of birth and death coordinates for the plurality of topological features.
- the method in determining whether the 2D image is indicative of a defect in the manufactured component, may use a trained statistical model and the digital feature as an input to the trained statistical model.
- the trained statistical model may be an XGBoost classifier.
- the method may further include segmenting the 2D image before computing the sets of values for the plurality of topological features.
- the method may further include performing a smoothing operation, e.g., a morphological closing operation, on the 2D image before computing the sets of values for the plurality of topological features.
- the method may further include determining a local binary pattern based on the 2D image, where using the trained statistical model and the digital feature comprises additionally using the local binary pattern as an input to the trained statistical model.
- receiving the 2D image of the manufactured component may include capturing the 2D image of the manufactured component while the manufactured component is moving along a conveyor belt.
- the manufactured component may be a syringe.
- Capturing the 2D image of the manufactured component may include capturing the 2D image from a top of the manufactured component.
- the defect may include a crack, a chip, or both.
- FIG. l is a diagram of an exemplary system for detecting defects in manufactured components while the manufactured components are moving along a conveyor belt, according to some embodiments.
- FIG. 2 is a diagram of an exemplary system for detecting whether a manufactured component in a 2D image has a defect, according to some embodiments.
- FIG. 3 is a flow chart showing an exemplary computerized method for detecting manufactured component defects, according to some embodiments.
- FIG. 4 is a flow chart showing an exemplary computerized method for determining a topological feature that is used for detecting whether a manufactured component has a defect, according to some embodiments.
- FIG. 5 is a flow chart showing an exemplary computerized method for determining a local binary pattern vector that is used for detecting whether a manufactured component has a defect, according to some embodiments.
- FIGS. 6A-6D show examples of 2D images containing a manufactured component that does not have any defect, according to some embodiments.
- FIGS. 6E-6F show examples of 2D images containing a manufactured component that has a defect, according to some embodiments.
- FIG. 7A shows an example of a 2D image containing a manufactured component that has a defect, according to some embodiments.
- FIG. 7B shows an example local binary pattern of the 2D image as shown in FIG. 7A, according to some embodiments.
- FIG. 7C shows an example histogram of the local binary pattern as shown in FIG. 7B, according to some embodiments.
- FIG. 8 shows sets of birth/death coordinates for an example 2D image containing a manufactured component without and with morphological closing performed, according to some embodiments.
- FIGS. 9A-9E show examples of topological features as the filtration threshold varies for a sample image, according to some embodiments.
- FIG. 10A shows an example raw image
- FIG. 10B shows an example of a segmented image from the raw image shown in FIG. 10A, according to some embodiments.
- FIGS. 11 A-l 1C show visualization of the spaces of features that represent the images absent defects and images having defects as detected by various configurations of the system, according to some embodiments.
- FIG. 12 shows an illustrative implementation of a computer system that may be used to perform any of the aspects of the techniques and embodiments disclosed herein, according to some embodiments.
- a manufactured component such as a syringe, a vial, or any other component
- This scanning may be done when the manufactured component enters an assembly line, while the component is being processed through the assembly line, and/or when the component exits the assembly line.
- defect detection may need to be performed in real time so that the detection can be completed while a manufactured component is still on, or before exiting, the assembly line. In such manner, a defective manufactured component can be timely removed from the assembly line (e.g., before moving to the next manufacturing stage). In some embodiments, even if the defect is not identified until after the manufactured component moves to the next production stage or exits the assembly line, it would still be desirable to detect defects quickly so as to minimize the number of faulty components produced and/or processed. For example, for high speed assembly lines, defect detection may need to occur on the order of milliseconds.
- a defect detection system may only have around 150 milliseconds (ms) to respond, i.e. to determine whether a manufactured component (e.g., a syringe that has entered into the assembly line) has a defect or is damaged. Further, the determination as to whether there is a defect in the manufactured component typically needs to be sufficiently accurate in order to minimize false positives (which can cause unnecessary disposal of manufactured components with no defects) or false negatives (which can cause defective manufactured components to enter into the next product stage).
- ms milliseconds
- topological data analysis can be used for defect detection to achieve both high accuracy and fast execution speed.
- topological features may contain geometrical, shape, and/or texture characteristics associated with a manufactured component, which can be used to detect defects in manufactured components.
- topological data analysis techniques may be suitable for extracting topological features from an image of a manufactured component, where computing the topological features can be performed with high efficiency.
- the inventors have developed new technologies for monitoring for manufacturing defects in manufactured components. Described herein are various techniques, including systems, computerized methods, and computer-executable instructions stored on non-transitory computer-readable media, that receive a two-dimensional (2D) image of a manufactured component.
- the system may capture the 2D image of the manufactured component when the component is moving along a conveyor belt.
- the 2D image of the manufactured component may be captured from the top of the manufactured component.
- the system may compute a set of values for each topological feature of a plurality of topological features in the 2D image, where the set of values indicate the life (e.g., start and end) of the topological feature.
- the plurality of topological features may contain geometrical, shape, and/or texture characteristics associated with a manufactured component.
- the topological features may be location and rotation invariant.
- a topological feature may be associated with a shape and may be of different levels.
- a O-th level topological feature may include an isolated block in a first color (e.g., a connected group of at least one black pixel), and a 1st level topological feature may include a block in a second color enclosed by a region in the first color (e.g., a block of white pixels enclosed by black pixels that are connected to each other).
- the system may determine a set of values indicating the life of the topological feature, e.g., start and end of the topological feature associated with a varying threshold.
- the set of values for a topological feature may include a set of birth and death coordinates, e.g., a pair of birth and death coordinates each indicating a value associated with a start and an end of the topological feature, respectively.
- the birth and death coordinate pair for a topological feature are determined by varying a filtration threshold and comparing the intensities of the 2D image with the filtration threshold to identify the set of birth and death coordinates.
- the system may determine a digital feature based on the computed set of birth and death coordinates for each topological feature of the plurality of topological features.
- the digital feature may include one or more persistence curve vectors based on the sets of birth and death coordinates for the plurality of topological features.
- the digital feature may include a persistence statistics vector, a persistence curve vector, or a combination thereof.
- a persistence statistics vector may include one or more of a mean, a standard, a skewness, a kurtosis, or an entropy associated with the birth and death coordinates in the sets of birth/death coordinates.
- a persistence curve vector may also be determined based on the birth and death coordinates in the sets of birth and death coordinates.
- the system may determine, based on the digital feature, whether the 2D image is indicative of a defect in the manufactured component.
- the system may use a trained statistical model and the digital feature as an input to the trained statistical model to detect the defect in the manufactured component.
- the statistical model may be any suitable model, such as a classifier, and may be implemented in any suitable framework, such as a gradient boosting machine learning model (XGBoost), a convolutional neural network (CNN), or any other suitable model.
- XGBoost gradient boosting machine learning model
- CNN convolutional neural network
- the output of the statistical model may indicate whether the manufactured component has a defect. Examples of detects that may be detected include a crack, a chip, and/or the like.
- additional features may also be provided to the statistical model to determine whether the 2D image is indicative of any defect in the manufactured component.
- the system may determine a local binary pattern vector based on the 2D image and provide the local binary pattern vector as input to the statistical model.
- the local binary pattern vector may be determined based on a local binary pattern image of the 2D image, where the local binary pattern image may contain grayscale and/or rotation invariant features.
- the local binary pattern vector may be determined based on the histogram of the local binary pattern image.
- the captured 2D image of the manufactured component may be pre-processed to improve the performance of the topological feature extraction and subsequent detection.
- the system may segment the 2D image (e.g., via cropping) before extracting the topological features.
- the system may also perform smoothing operation over the 2D image, e.g., by performing a morphological closing operation. It is appreciated that other smoothing techniques may also be used.
- the techniques described herein may provide advantages over conventional systems for detecting defects in manufactured components. For example, improved accuracy can be achieved over conventional systems with respect to measures, such as true positive, false negative, false positive, and true negative, area under receiver operating characteristic (AUROC), area under precision-recall curve (AUPRC), precision, recall, specificity and/or the harmonic mean of precision and recall (Fl).
- measures such as true positive, false negative, false positive, and true negative, area under receiver operating characteristic (AUROC), area under precision-recall curve (AUPRC), precision, recall, specificity and/or the harmonic mean of precision and recall (Fl).
- AUROC area under receiver operating characteristic
- AUPRC area under precision-recall curve
- Fl harmonic mean of precision and recall
- the techniques described herein can also achieve high speed that is sufficient to detect defects in real time.
- the processing speed for processing images collected under typical manufacturing operations show that 52ms can be achieved for processing each image, with slightly additional time for performing local binary pattern (33ms additionally) and morphological closing operation (0.6ms).
- FIG. 1 is a diagram of an exemplary system 100 for detecting defects in manufactured components while the manufactured components are moving along a conveyor belt, according to some embodiments.
- system 100 may include an imaging capturing device (e.g., a camera 112) configured to obtain an image/video of a manufactured component 102 in a production line while the manufactured component 102 is moving along a conveyor belt 104.
- an example of the manufactured component 102 is a container.
- the manufactured component can be any other device or component to be scanned for defect detection, such as a syringe, a glass vial, a bottle, ajar, and/or any suitable component.
- the manufactured component 102 is shown on a conveyor, the manufactured component may be moving on other platforms while being scanned/imaged, such as a turntable.
- System 100 may further include a server 110 having at least a processor, a memory, a storage medium, and/or other components.
- the at least one processor may be configured to execute programming instructions stored in the memory to process the captured image/video.
- the server 110 may analyze a 2D image in real time as each image is captured, and detect defect(s) in the image.
- the inspection process for the manufactured component can be completed in a short period of time, such as a fraction of a second, during which time the system may determine whether the manufactured component may likely have a defect. Responsive to determining that the manufactured component may likely have a defect, the system may perform one or more corrective measures, such as removing the manufactured component from the assembly line, before moving to the next production stage.
- the system may generate an alert indicative of the detected potential defect.
- the alert may include any suitable form and may be communicated to a user device in any suitable communication protocols to indicate to the user that a defect is detected.
- server 110 may send a notification, a warning, or a report to a user device 114, via a communication link (wired or wireless).
- the notification or the report may include information about the defect, and/or the type of defect.
- the operator may timely take corrective measures. Details of system 100 for detecting defects in manufactured components are further described with reference to FIGS. 2-10.
- FIG. 2 is a diagram of an exemplary system 200 for detecting whether a manufactured component in a 2D image is likely to have a defect, according to some embodiments.
- system 200 may be an example implementation of system 100 (FIG. 1).
- system 200 may be implemented in server 110 (FIG. 1).
- System 200 may receive 2D images of manufactured components, where the 2D images may be captured in the manner as described in embodiments of FIG. 1.
- the manufactured components may be syringes. Examples of detects that may be detected may include a crack, a chip, or both, or any other defects.
- FIGS. 6A-6F show example 2D images of a syringe that are captured from the top of the syringe. As shown, FIGS. 6A-6D are examples of 2D images containing a syringe that does not have any defect.
- FIGS. 6E-6F are examples of 2D images containing a syringe that has a defect.
- system 200 may include a topological feature extractor 206 configured to determine topological features from the 2D image.
- Topological feature extractor 206 may include a persistence diagram generator 206-1 and a digital feature extractor 206-2.
- the persistence diagram generator 206-1 may be configured to generate a persistence diagram of the 2D image.
- the persistence diagram may be generated using topological data analysis (TDA) as will be explained in detail in FIGS. 3-4.
- TDA topological data analysis
- the persistence diagram may include a plurality of sets of values, e.g., birth and death coordinates, where each set indicates a life of a topological feature of a plurality of topological features.
- the set of values for a topological feature may include a birth and death coordinate pair corresponding to the topological feature of a plurality of topological features.
- the birth and death coordinates in each pair may include two values (e.g., numerical values) indicative of the life (e.g., birth, death) of each topological feature of a plurality of topological features.
- the birth and death coordinates of a topological feature may respectively indicate the birth and death of the topological feature.
- the plurality of topological features may contain geometrical, shape, and/or texture characteristics associated with a manufactured component.
- the topological features may be location and rotation invariant.
- the digital feature extractor 206-2 may be configured to convert the persistence diagram (e.g., birth and death coordinates) to a digital feature for use with subsequent statistical model 208.
- the digital feature output from the digital feature extractor 206-2 may include one or more persistence curve vectors that are determined based on the sets of birth/death coordinates for the plurality of topological features.
- the digital feature may include a persistence statistics vector, a persistence curve vector, or a combination thereof.
- a persistence statistics vector may include one or more of mean, standard, skewness, kurtosis, or entropy associated with the birth and death coordinates pairs in the sets of birth/death coordinates.
- a persistence curve vector may also be determined based on the birth and death coordinates pairs in the sets of birth/death coordinates.
- system 200 may further include a trained statistical model 208 configured to determine, based on the digital feature from the topological feature extractor 206, whether the 2D image is indicative of a defect in the manufactured component.
- the statistical model may be any suitable model, such as a classifier, and may be implemented in any suitable framework, such as a gradient boosting machine learning model (XGBoost), a convolutional neural network (CNN), or any other suitable model.
- XGBoost gradient boosting machine learning model
- CNN convolutional neural network
- system 200 may include a local binary pattern (LBP) extractor 210 configured to provide additional features to the trained statistical model 208.
- LBP local binary pattern
- the LBP extractor 210 may determine a local binary pattern vector based on the 2D image and provide the local binary pattern vector as input to the statistical model 208.
- the local binary pattern vector may be determined based on local binary pattern image of the 2D image, where the local binary pattern image may be indicative of grayscale and/or rotation invariant features.
- the local binary pattern vector may be determined based on the histogram of the local binary pattern image.
- system 200 may optionally include one or more preprocessing units to pre-process the 2D images.
- system 200 may include an image segm enter 202 configured to segment (e.g., via cropping) the 2D image before extracting the topological features from the 2D image.
- system 200 may include a smooth operator 204 configured to further perform smoothing operation over the 2D image.
- the smooth operator 204 may be a morphological closing operator.
- the smooth operator 204 may be a Gaussian filter, a median filter, a bilateral filter, a morphological opening operator, or a combination thereof. It is appreciated that other suitable smoothing techniques may also be used.
- FIG. 3 is a flow chart showing an exemplary computerized method 300 for detecting manufactured component defects, according to some embodiments.
- method 300 may be implemented in system 100 (e.g., server 110, FIG. 1) or system 200 (FIG. 2).
- each act in method 300 may be implemented in a respective block in system 200.
- method 300 may start with receiving a 2D image of a manufactured component, at act 302, as described herein above.
- the 2D images may be captured by a camera, which is configured to capture a 2D image of a manufactured component while the manufactured component is moving along a moving platform, e.g., a conveyor.
- method 300 may further include determining topological features, at act 308.
- act 308 may be performed by the topological feature extractor 206 in FIG. 2. Act 308 is further described in detail with reference to an exemplary process in FIG. 4.
- FIG. 4 is a flow chart showing an exemplary computerized method 400 for determining a topological feature that is used for detecting whether a manufactured component is likely to have a defect, according to some embodiments.
- method 400 may be implemented in system 100, 200, such as topological feature extractor 206 (FIG. 2). As shown in FIG.
- method 400 starts with computing birth and death coordinates for each topological feature in the 2D image at act 402, where the birth and death coordinates may include a plurality of birth and death coordinates pairs, expressed as (b, d), for each topological feature.
- act 402 may be implemented in persistence diagram generator 206-1 (FIG. 2).
- the birth and death coordinates also make up the persistence diagram.
- a persistence diagram may include one or more sublevel persistence diagrams each corresponding to a topological feature.
- a persistence diagram may contain two sublevel persistence diagrams: a O-th level persistence diagram (Do) and 1st level persistence diagram (Di).
- the persistence diagram Do may contain a first set of birth death coordinates pairs (b,d) Do associated with a first type of topological feature (e.g., O-th level topological features), such as an isolated region in a first color (e.g., a black region having black pixels connected to each other).
- the persistence diagram Di may contain a second set of birth death coordinates pairs (b,d) Di associated a second type of topological feature (e.g., 1st level topological features), such as a block in a second color enclosed by a region in the first color (e.g., a block of white pixels completely enclosed by black pixels that are connected to each other).
- the first color and the second color e.g., black and white
- FIGS. 9A-9E Detailed operations for determining persistence diagrams Do and Di are further described with an example shown in FIGS. 9A-9E.
- FIGS. 9A-9E show examples of topological features as a filtration threshold varies for a sample image, according to some embodiments.
- FIG. 9A illustrates a sample grayscale image having 3x3 pixels with the intensity value of each pixel shown.
- FIGS. 9B-9E show binarized images when a respective filtration threshold is applied to the original image shown in FIG. 9A.
- a filtration threshold may be varied between the minimum and maximum intensity values of the image y(x), e.g., 1-10 in this example. When the filtration threshold increases by value from 1-10, the system tracks the changes in homology. Applying the filtration threshold to the image may result in a binarized image having two values: black and white, where black represents a pixel with a value at or below the threshold and white represents a pixel with a value above the threshold.
- topological feature 902 when a filtration threshold of value 1 is applied, a topological feature 902 appears (is born) from the binarized image. Topological feature 902 having an isolated region in black may thus be associated with a O-th level persistence diagram. As shown in FIG. 9C, when a filtration threshold of value 2 is applied, topological feature 904 is born from the binarized image. As shown, topological feature 904 is of the same type as the topological feature 902 and may also be associated with the O-th level persistence diagram (e.g., Do).
- O-th level persistence diagram e.g., Do
- topological features 902, 904 are merged.
- they are merged using an elder rule, under which the senior topological feature that was bom first survives whereas the younger one disappears (dies).
- topological feature 902 survives and topological feature 904 dies.
- the merged region of two topological features comprises pixels from both topological features being merged.
- topological features 902, 904 are merged as topological feature 902 having the black region with a white opening therein.
- a new type of topological feature 906 which corresponds to a white block completely enclosed by a black region (e.g., a white region completely enclosed by black pixels), is also bom.
- the topological feature 906 may be associated with a 1st level persistence diagram (e.g., Di .
- Di a 1st level persistence diagram
- FIG. 9E when a filtration threshold of value 10 is applied, the entire image becomes black, causing topological feature 906 to die, with topological feature 902 surviving.
- the birth coordinate for a particular topological feature is the filtration threshold level t at which that particular topological feature comes into existence as the filtration threshold level is increased from the minimum value of y(x) to the maximum value of y(x .
- the death coordinate for a particular topological feature is the filtration threshold level t at which that particular topological feature expires.
- a particular topological feature is considered to expire when it merges with another topological feature having an earlier birth coordinate, as discussed above.
- the birth and death coordinates pairs (1, ⁇ x), (2, 3) in Do respectively correspond to the topological features 902 and 904, whereas the birth and death coordinates pair (3, 10) corresponds to the topological feature 906.
- persistence diagrams may be determined based on Betti numbers associated with a plurality of filtration threshold levels in a grayscale image.
- a Betti number may be determined for each resulting binary image when applying a respective filtration threshold to the grayscale image. For example, at each filtration threshold value, a Betti number may represent the count of the number of topological spaces (shapes) based on the pixel connectivity in the binary image.
- a Betti number may include a O-th level number and a 1st level number, respectively, representing the number of isolated regions (e.g., connected black regions) and the number of voids (e.g., white regions that are completely enclosed by black pixels) at a given filtration threshold value.
- the Betti numbers are respectively (1, 0), (2,0), (1,1), and (1, 0), where the first number in each pair represents the O-th level Betti number, and the second number in each pair represents the 1 -st level Betti number.
- the birth and death coordinates pairs for persistence diagrams Do, Di may be associated with the change of Betti numbers as the filtration threshold changes.
- the Betti number changes from (1, 0) to (2, 0), representing a birth of a new topological feature (e.g., the O-th Betti number changes from 1 to 2).
- the Betti number changes from (2, 0) to (1, 1), representing a merge of topological features (e.g., the O-th Betti number changes from 2 to 1) and a birth of a new topological feature (e.g., the 1st Betti number changes from 0 to 1).
- the topological features may also be associated with Betti numbers or changes of Betti numbers associated with filtration threshold levels.
- method 400 may further determine a digital feature (e.g., at acts 404-410) based on the birth and death coordinates.
- acts 404-410 may be implemented in digital feature extractor 206-2 (FIG. 2). The digital feature is further described.
- method 400 may first determine midlife persistence values, at act 404. Method 400 may also determine lifespan persistence values, at act 406. The midlife persistence and lifespan persistence values may be determined for each set of birth and death coordinates obtained from act 402.
- act 404 may calculate a lifespan persistence (/.) for each respective topological feature of the plurality of topological features by subtracting the birth coordinate for that respective topological feature from the death coordinate for that respective topological feature.
- act 406 may calculate a midlife persistence (M) for each respective topological feature in the plurality of topological feature by calculating a mean average of the set of birth and death coordinates for that respective topological feature.
- method 400 may include determining a persistence statistics vector at act 408.
- act 408 may calculate a first persistence statistics vector based on one or more of a mean, a standard deviation, a skewness, a kurtosis, or an entropy of the calculated lifespan persistence (/.) across all topological features in the 2D image.
- act 408 may calculate a second persistence statistics vector based on one or more of a mean, a standard deviation, a skewness, a kurtosis, or an entropy of the calculated midlife persistence (M) across all topological features in the 2D image.
- M midlife persistence
- Persistence statistics may include summary statistics of M and L which are mean, standard deviation, skewness, kurtosis, and entropy for M and L.
- method 400 may include determining a persistence curve vector, at act 410.
- a persistence curve vector may be calculated based on the lifespan values of the sets of birth and death coordinates for the plurality of topological features.
- a persistence curve vector may be expressed as a product of two persistence curves, each based on a sum of lifespan values of the sets of birth and death coordinates corresponding to each of the persistence diagrams (e.g., Do, Di). In a nonlimiting example: > x].
- the second persistence curve in the persistence curve vector is the logarithm of the first persistence curve.
- the persistence curve vectors described above can have any suitable number of features (elements).
- the persistence statistics vector may include a ID vector having 10 features (e.g., one feature each for the mean, standard deviation, skewness, kurtosis, and entropy for both midlife persistence values and lifespan persistence values of the topological features in persistence diagram D).
- the persistence curve vector may be a 256-feature vector (this example assumes x, which represents intensity in the received 2D image of the manufactured component, may vary between 256 different levels from lowest to highest intensity.
- the persistence curve vector may be a 512 or 1024-feature vector, respectively).
- method 300 may further include using the topological feature to determine whether the 2D image is indicative of a defect in the manufactured component, at act 312.
- the topological feature used for act 312 may be the digital feature obtained from act 308 (details shown in FIG. 4).
- act 312 in determining whether the 2D image is indicative of a defect in the manufactured component, act 312 may use a trained statistical model. Any suitable statistical model may be used. For example, act 312 may use a gradient boosting machine learning model (XGBoost), a convolutional neural network (CNN), or any other suitable model. Such a model may be trained using digital features similar to those obtained from act 308 in combination with ground truth labelling data that indicates whether a particular digital feature is associated with an image of a defective or not-defective component.
- XGBoost gradient boosting machine learning model
- CNN convolutional neural network
- method 300 may optionally include one or more additional acts.
- method 300 may optionally include segmenting the 2D image, at act 304, before determining the topological feature (act 308).
- act 304 may be implemented in the image segmenter 202 (FIG. 2).
- segmenting the 2D image may include cropping.
- cropping may center the manufactured component in the image and/or trim off some background to reduce the computation required to process the image.
- FIG. 10B shows an example of a segmented image from the raw image shown in FIG. 10 A.
- method 300 may optionally include performing a smoothing operation at act 306.
- act 306 may be implemented in the smooth operator 204 (FIG. 2). Smoothing operation may suppress noise in the 2D image.
- the smooth operation may include a morphological closing operation.
- Any suitable size structuring element e.g., 3x3 may be used in the morphological closing operation. It is appreciated that other smoothing techniques may also be used.
- FIG. 8 shows the sets of birth/death coordinates for an example 2D image containing a manufactured component without and with morphological closing performed.
- FIG. 8(a) shows the original 2D image, wherein FIG. 8(d) shows the smoothed image of the image in FIG. 8(a) after performing the morphological closing operation.
- FIGS. 8(e) and 8(f) respectively show the O-th level persistence diagrams for the smoothed image in FIG. 8(d). As shown, relative fewer birth and death coordinates are extracted from the smoothed image.
- method 300 may optionally determine a local binary pattern, at act 310, and provide the local binary pattern as additional features to the statistical model (e.g., act 312, or 208 in FIG. 2).
- act 310 may be implemented in local binary pattern extractor 210 (FIG. 2). The details of extracting the local binary pattern are further described with reference to FIG. 5, which is a flow chart showing an exemplary computerized method 500 for determining a local binary pattern vector that is used for detecting whether a manufactured component has defect, according to some embodiments.
- method 500 may start with determining local binary pattern from the 2D image, at act 502.
- act 502 may be configured to extract uniform local binary pattern, which may be grayscale and rotation invariant.
- Method 500 may also determine a histogram of the local binary pattern, at act 504 and determine a local binary pattern vector based on the histogram, at act 506.
- FIG. 7A shows an example of 2D image containing a manufactured component that has defect, according to some embodiments.
- FIG. 7B shows an example local binary pattern of 2D image as shown in FIG. 7A, according to some embodiments.
- the local binary pattern from act 502 may be a grayscale image.
- the grayscale in the local binary pattern may range from 0-24. It is appreciated that any other suitable range may be possible.
- FIG. 7C shows an example histogram of a local binary pattern as shown in FIG. 7B, according to some embodiments.
- the size of the histogram corresponds to the grayscale range in the local binary pattern, e.g., 25 (0-24).
- act 506 may determine a local binary pattern vector based on the histogram, where the size of the vector is the size of the histogram, and the values of each element in the vector corresponds to the histogram value at each bin.
- the local binary pattern vector has a size of 25. It is appreciated that any other suitable size is possible for the local binary pattern vector.
- the digital feature provided to the trained statistical model may include a ID vector of any suitable size.
- the digital feature may concatenate the persistence curve vector from topological feature extractor 206 and/or the local binary pattern vector from local binary pattern extractor 210.
- the size of the persistence curve vector is 532
- the size of the local binary pattern vector is 25, resulting in the total size of the digital feature being 557.
- the 2D image is compressed in an efficient manner and converted to a ID vector of 557 elements.
- image segmenter 202, topological feature extractor 206, and trained statistical model 208 were implemented without using smooth operator 204 or local binary pattern extractor 210.
- image segmenter 202, smooth operator 204, topological feature extractor 206, and trained statistical model 208 were implemented without using local binary pattern extractor 210.
- all of the components e.g., image segmenter 202, smooth operator 204, topological feature extractor 206, local binary pattern extractor 210, and trained statistical model 208 were implemented. The experimental results for these configurations are further described below.
- Table 1 shows the validation result of the system for the different configurations as described above, using a validation data set. Validation can be used to fine-tune the model parameters.
- TN, FP, FN, and TP represent true negatives, false positives, false negatives, and true positives, respectively.
- Table 2 shows the testing result of the system for different configurations as described above, using a testing data set. Testing can be used to evaluate the performance of the model once the parameters are fine-tuned.
- Table 3 shows the results on holdout data sets (which are obtained from production line on different days and not used in the training).
- FIGS. 11 A-l 1C show visualization of the spaces of features that represent images absent defects (good images) and images having defects (defect images) as detected by various configurations of the system, in 3D space.
- the features such as described above with respect to the persistence curve vector, and/or local binary pattern vector, are projected into the 3D space using a suitable projection method. For example, FIGS.
- FIGS. 11 A and 1 IB show visualizations of the 3D spaces onto which 532 features (e.g., persistence curve vector) are projected, where defect images were respectively detected by the first and second configurations of the system described with respect to Tables 1-3.
- FIG. 11C shows visualization of the 3D space onto which 557 features (e.g., persistence curve vector and local binary pattern vector) are projected, where defect images were detected by the third configuration of the system.
- the dots in different shades/color represent images absent defects and having defects as detected by the system. As shown in FIGS. 11 A- 11C, separations between these two classes of images can be seen.
- the techniques as described can be performed reasonably efficiently in a modern computer, such as a CPU architecture with Intel x86-64 core having multiple subsystems.
- the various embodiments or components of the system may be configured such that the software can be optimized for execution on multiple instruction execution units (multi-threading), a multilevel cache hierarchy, multi-stage instruction pipelines, and/or branch prediction units.
- XGBoost may be implemented in a statistical model (e.g., statistical model 208 in FIG. 2) that implements a tree boosting algorithm.
- Such an algorithm may be configured to build multiple parallel, data-independent trees for computing partial predictions, which can be combined to generate the final prediction results.
- Each tree may have a simple binary tree structure with a limited number of levels (e.g., 10 levels), and can be executed in parallel in a multi -threaded CPU.
- the XGBoost algorithm may be configured to use exact greedy algorithm to find the optimal possible splits of all the features. The splits may indicate where the tree traversing procedure branches to the next tree leaf, where such tree branching strategies may be used to utilize the CPU’s branch prediction unit.
- XGBoost may be configured to map the feature data (e.g., topological features) to memory structures in order to increase the CPU cache utilization.
- the feature data may be stored in in- memory units (e.g., blocks).
- data may be represented in Compressed Column format. Blocks are then sorted, which turns a data finding procedure into a linear scan. Since CPU cache stores a complete cache line, a linear scan in such a scenario will yield an optimal cache hit rate.
- XGBoost may be configured to leverage a cache-aware prefetching algorithm. For example, an internal buffer may be allocated by each thread, to where XGBoost prefetches and performs accumulation over gradients.
- FIG. 12 shows an illustrative implementation of a computer system that may be used to perform any of the aspects of the techniques and embodiments disclosed herein, according to some embodiments.
- An illustrative implementation of a computer system 1000 that may be used to perform any of the aspects of the techniques and embodiments disclosed herein is shown in FIG. 12.
- the computer system 1000 may be installed in system 100 of FIG. 1, such as by server 110.
- the computer system 1000 may be configured to perform various methods and acts as described in FIGS. 3-5.
- the computer system 1000 may include one or more processors 1010 and one or more non -transitory computer-readable storage media (e.g., memory 1020 and one or more non-volatile storage media 1030) and a display 1040.
- non -transitory computer-readable storage media e.g., memory 1020 and one or more non-volatile storage media 1030
- the processor 1010 may control writing data to and reading data from the memory 1020 and the non-volatile storage device 1030 in any suitable manner, as the aspects of the invention described herein are not limited in this respect.
- the computer system 1000 may also be a complete system on module (SOM), such as NVIDIA’ s Jetson module, which includes CPU, GPU, memory, and other components in a system.
- SOM system on module
- the computer system 1000 may be located at any suitable site.
- the computer system 1000 e.g., server 110
- the system may not need to include a memory, but instead programming instructions are running on one or more virtual machines or one or more containers on a cloud.
- the various methods illustrated above may be implemented by a server on a cloud that includes multiple virtual machines, each virtual machine having an operating system, a virtual disk, virtual network and applications, and the programming instructions for detecting defects in manufactured components may be stored on one or more of those virtual machines on the cloud.
- the processor 1010 may execute one or more instructions stored in one or more computer-readable storage media (e.g., the memory 1020, storage media, etc.), which may serve as non-transitory computer- readable storage media storing instructions for execution by the processor 1010.
- computer-readable storage media e.g., the memory 1020, storage media, etc.
- code used to, for example, detect defects in manufactured components may be stored on one or more computer-readable storage media of computer system 1000.
- Processor 1010 may execute any such code to provide any techniques for detecting defects as described herein.
- Any other software, programs or instructions described herein may also be stored and executed by computer system 1000.
- computer code may be applied to any aspects of methods and techniques described herein. For example, computer code may be applied to interact with an operating system to detect defects through conventional operating system processes.
- the various methods or processes outlined herein may be coded as software that is executable on one or more processors that employ any one of a variety of operating systems or platforms. Additionally, such software may be written using any of numerous suitable programming languages and/or programming or scripting tools, and also may be compiled as executable machine language code or intermediate code that is executed on a virtual machine or a suitable framework.
- inventive concepts may be embodied as at least one non- transitory computer readable storage medium (e.g., a computer memory, one or more floppy discs, compact discs, optical discs, magnetic tapes, flash memories, circuit configurations in Field Programmable Gate Arrays or other semiconductor devices, etc.) encoded with one or more programs that, when executed on one or more computers or other processors, implement the various embodiments of the present invention.
- the non-transitory computer- readable medium or media may be transportable, such that the program or programs stored thereon may be loaded onto any computer resource to implement various aspects of the present invention as discussed above.
- program “software,” and/or “application” are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions that can be employed to program a computer or other processor to implement various aspects of embodiments as discussed above. Additionally, it should be appreciated that according to one aspect, one or more computer programs that when executed perform methods of the present invention need not reside on a single computer or processor, but may be distributed in a modular fashion among different computers or processors to implement various aspects of the present invention.
- Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices.
- program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types.
- functionality of the program modules may be combined or distributed as desired in various embodiments.
- data structures may be stored in non-transitory computer-readable storage media in any suitable form.
- Data structures may have fields that are related through location in the data structure. Such relationships may likewise be achieved by assigning storage for the fields with locations in a non-transitory computer-readable medium that convey relationship between the fields.
- any suitable mechanism may be used to establish relationships among information in fields of a data structure, including through the use of pointers, tags or other mechanisms that establish relationships among data elements.
- inventive concepts may be embodied as one or more methods, of which examples have been provided.
- the acts performed as part of a method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.
- the phrase “at least one,” in reference to a list of one or more elements should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements. This allows elements to optionally be present other than the elements specifically identified within the list of elements to which the phrase “at least one” refers, whether related or unrelated to those elements specifically identified.
- a reference to “A and/or B”, when used in conjunction with open-ended language such as “comprising” can refer, in one embodiment, to A only (optionally including elements other than B); in another embodiment, to B only (optionally including elements other than A); in yet another embodiment, to both A and B (optionally including other elements); etc.
- a computerized method for monitoring for manufacturing defects comprising: receiving a two-dimensional (2D) image of a manufactured component; computing a respective set of values indicating a life for each topological feature of a plurality of topological features in the 2D image; determining a digital feature based on the computed sets of values for the plurality of topological features; and determining, based on the digital feature, whether the 2D image is indicative of a defect in the manufactured component.
- 2D two-dimensional
- each topological feature of the plurality of topological features comprises a shape comprising an isolated block or a block in a first color completely enclosed by a region in a second color, wherein each pixel in the shape is below a filtration threshold.
- determining the digital feature comprises: determining a persistence statistics vector, a persistence curve vector, or a combination thereof, based on the computed sets of birth and death coordinates for the plurality of topological features.
- determining the persistence statistics vector comprises: determining a first persistence statistics vector containing a mean, a standard, a skewness, a kurtosis, and/or an entropy of the first set of values; and determining a second persistence statistics vector containing a mean, a standard, a skewness, a kurtosis, and/or an entropy of the second set of values.
- receiving the 2D image of the manufactured component comprises capturing the 2D image of the manufactured component while the manufactured component is moving along a conveyor belt.
- each topological feature of the plurality of topological features comprises a shape comprising an isolated block or a block in a first color completely enclosed by a region in a second color, wherein each pixel in the shape is below the filtration threshold; and the birth coordinate for each topological feature corresponds to a first level of the filtration threshold at which said topological feature first appears as the filtration threshold is increased.
- a system for detecting whether a manufactured component has a defect comprising: an image capturing device configured to capture an image of the manufactured component; and at least one processor to perform the method of any of aspects 1-20.
- a non-transitory computer-readable media storing instructions that, when executed by at least one processor, cause the at least one processor to perform the method of any of aspects 1-20.
Landscapes
- Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
Description
Claims
Priority Applications (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| AU2024304441A AU2024304441A1 (en) | 2023-06-12 | 2024-06-11 | Methods and systems for detecting defects using topological persistence features |
| EP24738157.7A EP4724980A1 (en) | 2023-06-12 | 2024-06-11 | Methods and systems for detecting defects using topological persistence features |
| CN202480038927.6A CN121399653A (en) | 2023-06-12 | 2024-06-11 | Method and system for detecting defects using topology persistence features |
Applications Claiming Priority (4)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US202363507491P | 2023-06-12 | 2023-06-12 | |
| US63/507,491 | 2023-06-12 | ||
| US202363584185P | 2023-09-21 | 2023-09-21 | |
| US63/584,185 | 2023-09-21 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2024258811A1 true WO2024258811A1 (en) | 2024-12-19 |
Family
ID=91782272
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/US2024/033334 Ceased WO2024258811A1 (en) | 2023-06-12 | 2024-06-11 | Methods and systems for detecting defects using topological persistence features |
Country Status (4)
| Country | Link |
|---|---|
| EP (1) | EP4724980A1 (en) |
| CN (1) | CN121399653A (en) |
| AU (1) | AU2024304441A1 (en) |
| WO (1) | WO2024258811A1 (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114972252A (en) * | 2022-05-26 | 2022-08-30 | 徐工汉云技术股份有限公司 | Metal surface defect detection method based on pixel-level positioning |
| US20220398715A1 (en) * | 2019-11-07 | 2022-12-15 | Amgen Inc. | Targeted application of deep learning to automated visual inspection equipment |
-
2024
- 2024-06-11 WO PCT/US2024/033334 patent/WO2024258811A1/en not_active Ceased
- 2024-06-11 CN CN202480038927.6A patent/CN121399653A/en active Pending
- 2024-06-11 AU AU2024304441A patent/AU2024304441A1/en active Pending
- 2024-06-11 EP EP24738157.7A patent/EP4724980A1/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220398715A1 (en) * | 2019-11-07 | 2022-12-15 | Amgen Inc. | Targeted application of deep learning to automated visual inspection equipment |
| CN114972252A (en) * | 2022-05-26 | 2022-08-30 | 徐工汉云技术股份有限公司 | Metal surface defect detection method based on pixel-level positioning |
Non-Patent Citations (3)
| Title |
|---|
| ANDREW M THOMAS ET AL: "Feature detection and hypothesis testing for extremely noisy nanoparticle images using topological data analysis", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 18 January 2023 (2023-01-18), XP091415206 * |
| CHUNG YU-MIN ET AL: "Topological approaches to skin disease image analysis", 2018 IEEE INTERNATIONAL CONFERENCE ON BIG DATA (BIG DATA), IEEE, 10 December 2018 (2018-12-10), pages 100 - 105, XP033508159, DOI: 10.1109/BIGDATA.2018.8622175 * |
| MAX M CHUMLEY ET AL: "Pattern Characterization Using Topological Data Analysis: Application to Piezo Vibration Striking Treatment", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 12 October 2022 (2022-10-12), XP091342295 * |
Also Published As
| Publication number | Publication date |
|---|---|
| EP4724980A1 (en) | 2026-04-15 |
| AU2024304441A1 (en) | 2025-12-11 |
| CN121399653A (en) | 2026-01-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113962274B (en) | Abnormity identification method and device, electronic equipment and storage medium | |
| CN109165538B (en) | Bar code detection method and device based on deep neural network | |
| US9965695B1 (en) | Document image binarization method based on content type separation | |
| CN1875378A (en) | Object detection in images | |
| CN111695373B (en) | Zebra stripes positioning method, system, medium and equipment | |
| CN115690102B (en) | Defect detection method, device, electronic device, storage medium, and program product | |
| CN113538500B (en) | Image segmentation method and device, electronic equipment and storage medium | |
| Azad et al. | New method for optimization of license plate recognition system with use of edge detection and connected component | |
| CN112052702A (en) | Method and device for identifying two-dimensional code | |
| CN113610772A (en) | Method, system and device for detecting bottle bottom code spraying defects of pop cans and storage medium | |
| CN112668379B (en) | Object positioning system and method | |
| CN121147129A (en) | A method and system for detecting end-face defects in steel coils based on CBAM-BiFPN and multi-loss optimization | |
| CN119671947A (en) | A highway engineering quality detection method and system based on image recognition | |
| CN119323567A (en) | Module lens dirt detection and glue line detection method | |
| Tribak et al. | QR code patterns localization based on Hu invariant moments | |
| CN114964628B (en) | Ammonia gas leakage shuffling self-attention light-weight infrared detection method and system | |
| WO2024258811A1 (en) | Methods and systems for detecting defects using topological persistence features | |
| CN117893934B (en) | An improved UNet3+ network drone image railway track detection method and device | |
| Liu et al. | Mobile-Deep Based PCB Image Segmentation Algorithm Research. | |
| CN116542998B (en) | Contour detection method, device, equipment and medium for photoetching film inductance | |
| KR20240161741A (en) | System and method for detecting defect in contact lens manufacturing process using artificial intelligence | |
| De Vitis et al. | Algorithms for the detection of blob defects in high speed glass tube production lines | |
| Han et al. | Single-scale workpiece defect detection based on deep learning | |
| CN121563979B (en) | A Semiconductor Wafer Visual Inspection Method Based on Interactive Feature Learning | |
| Khalid et al. | AI-Driven Vehicle Damage Detection: A Saliency-Based Segmentation Approach |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 24738157 Country of ref document: EP Kind code of ref document: A1 |
|
| WWE | Wipo information: entry into national phase |
Ref document number: AU2024304441 Country of ref document: AU |
|
| ENP | Entry into the national phase |
Ref document number: 2024304441 Country of ref document: AU Date of ref document: 20240611 Kind code of ref document: A |
|
| ENP | Entry into the national phase |
Ref document number: 2024738157 Country of ref document: EP Effective date: 20260112 |
|
| WWE | Wipo information: entry into national phase |
Ref document number: 2024738157 Country of ref document: EP |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| ENP | Entry into the national phase |
Ref document number: 2024738157 Country of ref document: EP Effective date: 20260112 |
|
| ENP | Entry into the national phase |
Ref document number: 2024738157 Country of ref document: EP Effective date: 20260112 |
|
| ENP | Entry into the national phase |
Ref document number: 2024738157 Country of ref document: EP Effective date: 20260112 |
|
| ENP | Entry into the national phase |
Ref document number: 2024738157 Country of ref document: EP Effective date: 20260112 |
|
| WWP | Wipo information: published in national office |
Ref document number: 2024738157 Country of ref document: EP |



