CN112884691B - Data enhancement device, data enhancement apparatus, and storage medium - Google Patents
Data enhancement device, data enhancement apparatus, and storage medium Download PDFInfo
- Publication number
- CN112884691B CN112884691B CN202110260102.7A CN202110260102A CN112884691B CN 112884691 B CN112884691 B CN 112884691B CN 202110260102 A CN202110260102 A CN 202110260102A CN 112884691 B CN112884691 B CN 112884691B
- Authority
- CN
- China
- Prior art keywords
- image
- fusion
- training
- images
- defect
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012549 training Methods 0.000 claims abstract description 182
- 230000007547 defect Effects 0.000 claims abstract description 144
- 238000001514 detection method Methods 0.000 claims abstract description 140
- 230000004927 fusion Effects 0.000 claims abstract description 96
- 238000000034 method Methods 0.000 claims abstract description 54
- 238000012545 processing Methods 0.000 claims description 43
- 230000009466 transformation Effects 0.000 claims description 42
- 238000012795 verification Methods 0.000 claims description 39
- 238000013519 translation Methods 0.000 claims description 16
- 238000010008 shearing Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 5
- 239000003292 glue Substances 0.000 claims description 2
- 238000002372 labelling Methods 0.000 claims description 2
- 230000003647 oxidation Effects 0.000 claims description 2
- 238000007254 oxidation reaction Methods 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 16
- 230000008569 process Effects 0.000 description 28
- 235000012431 wafers Nutrition 0.000 description 7
- 230000006870 function Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000003062 neural network model Methods 0.000 description 3
- 238000004091 panning Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000002950 deficient Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011426 transformation method Methods 0.000 description 1
- 230000037303 wrinkles Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
A data enhancement method, a data enhancement device, a data enhancement apparatus, and a non-volatile computer-readable storage medium. The data enhancement method comprises the steps of acquiring a first workpiece image with a defect; identifying and intercepting an image area where the defect is located in the first workpiece image to serve as a first fusion image; acquiring a second workpiece image without defects as a second fused image; and fusing the first fused image and the second fused image to obtain a training image. The number of training images corresponding to the defects of the type with lower occurrence probability can be increased, so that the number of the training images with the defects of different types is basically the same, the training effect is good when the target detection model is trained by the training images with more numbers, the generalization performance of the target detection model can be improved, and the detection effect of the target detection model on the defects is improved.
Description
Technical Field
The present application relates to the field of detection technologies, and in particular, to a data enhancement method, a data enhancement device, a data enhancement apparatus, and a non-volatile computer readable storage medium.
Background
At present, defects of a workpiece are generally detected through a template detection algorithm, but the template detection algorithm has the problems of over-detection and low recognition accuracy, and compared with the neural network model with higher detection accuracy, the neural network model is gradually favored, but the occurrence probability of different types of defects is extremely unbalanced when the neural network model is trained, so that the model generalization performance is relatively poor when a deep learning method is used for wafer defect recognition or detection, and the final detection accuracy is directly influenced.
Disclosure of Invention
The application provides a data enhancement method, a data enhancement device, a data enhancement apparatus and a non-volatile computer readable storage medium.
The data enhancement method of the embodiment of the application comprises the steps of obtaining a first workpiece image with defects; identifying and intercepting an image area where the defect is located in the first workpiece image to serve as a first fusion image; acquiring a second workpiece image without defects as a second fused image; and fusing the first fused image and the second fused image to obtain a training image.
The data enhancement device of the embodiment of the application comprises a first acquisition module, an identification module, a second acquisition module and a fusion module. The first acquisition module is used for acquiring a first workpiece image with defects; the identification module is used for identifying and intercepting an image area where the defect is located in the first workpiece image to serve as a first fusion image; the second acquisition module is used for acquiring a second workpiece image without defects to serve as a second fusion image; and the fusion module fuses the first fusion image and the second fusion image to acquire a training image.
The data enhancement device of the embodiment of the application comprises a processor. The processor is configured to: acquiring a first workpiece image with a defect; identifying and intercepting an image area where the defect is located in the first workpiece image to serve as a first fusion image; acquiring a second workpiece image without defects as a second fused image; and fusing the first fused image and the second fused image to obtain a training image.
A non-transitory computer readable storage medium containing a computer program that, when executed by one or more processors, causes the processors to perform the data enhancement method. The data enhancement method comprises the steps of acquiring a first workpiece image with defects; identifying and intercepting an image area where the defect is located in the first workpiece image to serve as a first fusion image; acquiring a second workpiece image without defects as a second fused image; and fusing the first fused image and the second fused image to obtain a training image.
According to the data enhancement method, the data enhancement device, the data enhancement equipment and the non-volatile computer readable storage medium, the image area where the defect is located in the first workpiece image with the defect and the second workpiece image without the defect are fused to generate more workpiece images with the defect, so that a sufficient number of training images are obtained, the number of training images corresponding to the type of defect with low occurrence probability can be increased, the number of training images with different types of defects is basically the same, the training effect is good when the target detection model is trained through the training images with the large number, the generalization performance of the target detection model can be improved, and the defect detection effect of the target detection model is improved.
Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.
Drawings
In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only some embodiments of the application, and that other drawings can be obtained from them without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a data enhancement method of some embodiments of the present application;
FIG. 2 is a block diagram of a data enhancement device according to some embodiments of the present application;
FIG. 3 is a schematic plan view of a data enhancement device and a detection device of some embodiments of the present application;
FIGS. 4-9 are flow diagrams of data enhancement methods according to certain embodiments of the present application;
FIGS. 10-14 are schematic illustrations of data enhancement methods according to certain embodiments of the present application; and
FIG. 15 is a schematic diagram of a connection of a processor and a computer-readable storage medium according to some embodiments of the application.
Detailed Description
Embodiments of the present application are further described below with reference to the accompanying drawings. The same or similar reference numbers in the drawings refer to the same or similar elements or elements having the same or similar functions throughout. In addition, the embodiments of the present application described below with reference to the drawings are exemplary only for explaining the embodiments of the present application and are not to be construed as limiting the present application.
Referring to fig. 1 to 3, the data enhancement method according to the embodiment of the application includes the following steps:
011: acquiring a first workpiece image with a defect;
012: identifying and intercepting an image area where the defect is located in the first workpiece image to serve as a first fusion image;
013: acquiring a second workpiece image without defects as a second fused image; and
014: The first fused image and the second fused image are fused to obtain a training image.
The data enhancement device 10 according to the embodiment of the present application includes a first acquisition module 11, an identification module 12, a second acquisition module 13, and a fusion module 14. The first acquisition module 11 is used for acquiring a first workpiece image with defects; the identification module 12 is used for identifying and intercepting an image area where the defect is located in the first workpiece image to be used as a first fusion image; the second acquisition module 13 is configured to acquire a second workpiece image without defects as a second fused image; the fusion module 14 is configured to fuse the first fused image and the second fused image to obtain a training image. That is, step 011 may be implemented by the first acquisition module 11, step 012 may be performed by the identification module 12, step 013 may be performed by the second acquisition module 13, and step 014 may be performed by the fusion module 14.
The data enhancement device 100 of an embodiment of the present application includes a processor 20. The processor 20 is configured to: acquiring a first workpiece image with a defect; identifying and intercepting an image area where the defect is located in the first workpiece image to serve as a first fusion image; acquiring a second workpiece image without defects as a second fused image; and fusing the first fused image and the second fused image to obtain a training image. That is, steps 011, 012, 013, and 014 may be performed by the processor 20.
Specifically, the data enhancement device 100 is connected to the detection device 200, the sensor 210 of the detection device 200 captures the workpiece 300 to obtain image data, the processor 20 reads the image data from the sensor 210 to obtain a workpiece image, for example, when detecting the workpiece 300, the workpiece 300 is placed on the motion platform 220, and the sensor 210 acquires information of the workpiece 300 to generate the workpiece image.
The motion platform 220 may be used to carry the workpiece 300, and the motion platform 220 moves to drive the workpiece 300 to move, so that the sensor 210 collects information of the workpiece 300.
For example, the motion stage 220 includes an XY motion stage 221 and a Z motion stage 222, and the sensor 210 is disposed on the motion stage 220, specifically: the sensor 210 is disposed on the Z-motion stage 222, wherein the XY-motion stage 221 is configured to control the workpiece 300 to move along a horizontal plane, and change the relative position of the workpiece 300 and the sensor 210 in the horizontal plane, and the Z-motion stage 222 is configured to control the sensor 210 to move along a direction perpendicular to the horizontal plane, so that the XY-motion stage 221 and the Z-motion stage 222 cooperate to change the three-dimensional position of the sensor 210 relative to the workpiece 300 (i.e., the relative position in the horizontal plane and the relative position in the direction perpendicular to the horizontal plane).
It will be appreciated that the motion stage 220 is not limited to the above configuration, and may be capable of changing the three-dimensional position of the sensor 210 relative to the workpiece 300.
The sensor 210 may be one or more, and the plurality of sensors 210 may be different types of sensors 210, such as the sensors 210 may include a visible light camera, a depth camera, and the like. In this embodiment, the sensor 210 is a visible light camera.
When capturing an image of the workpiece, the processor 20 may adjust the distance between the sensor 210 and the workpiece 300 according to the field of view of the sensor 210 so that the workpiece 300 is located in the field of view, thereby capturing an image of the entire workpiece 300 once; or the sensor 210 can be made to cover only a partial area of the workpiece 300 in each photographing, and the movement platform 220 is moved to photograph different areas of the workpiece 300, so as to obtain a plurality of workpiece images. In this embodiment, the sensor 210 captures only a partial region of the workpiece 300 at a time to obtain a plurality of workpiece images.
In the present embodiment, the workpiece 300 is a wafer, and defects of the wafer generally include foreign matters, residual glue, oxidation, bubbles, wrinkles, cracks, and the like.
After the plurality of workpiece images are acquired, the first workpiece image with the defects and the second workpiece image without the defects can be accurately determined through a detection model or manual assistance. Then, the image area with the defect in the first workpiece image is identified and cut out to be used as a first fusion image. The first workpiece image contains various defects, and the first fusion image which is identified and cut out is the defect image which is different in type or type but has different characteristics (such as size, color and the like).
The defects can be identified manually, and for common types of defects, detection personnel can generally and accurately identify the defects, and after the defects are identified, the defects can be intercepted by drawing edge lines on the edges of the defects, so that a first fusion image is obtained; or the defect identification can also be realized through the processor 20, and the processor 20 identifies the position and the type of the defect based on a preset detection model, so that the image area where the defect is located is cut out according to the position of the defect, and a first fusion image is obtained.
In order to improve the training effect, when the second workpiece image is acquired, the wafer image or the second workpiece image of a plurality of wafers with different wafer background patterns can be selected, so that a plurality of second workpiece images with different image backgrounds are acquired, the diversity of the second workpiece image is improved, the training effect is improved, the influence of the trained target detection model on the image background is reduced, the influence of the complex image background on the defect detection precision is reduced, and the target detection model can accurately detect the defects even under different image backgrounds.
And then fusing the first fused image with a second workpiece image serving as a second fused image to obtain a plurality of training images with defects, so that the training images are enhanced in a plurality of aspects such as quantity, types and the like, and the detection effect of the training images on the target detection model is improved. The target detection model may be a second-order detection algorithm (such as fast R-CNN and its variants), a first-order detection algorithm (such as YOLOV and its variants), an anchor-free detection algorithm (such as CENTERNET and its variants), etc., which are not limited herein.
According to the data enhancement method, the data enhancement device 10 and the data enhancement equipment 100, the image area where the defect is located in the first workpiece image with the defect and the second workpiece image without the defect are fused to generate more workpiece images with the defect, so that a sufficient number of training images are obtained, the number of training images corresponding to the type of defect with low occurrence probability can be increased, the number of training images with different types of defects is basically the same, the training effect is good when the target detection model is trained through the training images with the large number, the generalization performance of the target detection model can be improved, and the defect detection effect of the target detection model is improved.
Referring to fig. 2,3 and 4, in some embodiments, the detection algorithm further includes, prior to step 013:
015: before fusing the first fused image and the second fused image, performing transformation processing on the first fused image, wherein the transformation processing comprises at least one of mirroring, translation, rotation, shearing and deformation.
In some embodiments, the data enhancement device 10 further includes a transformation module 15. The transformation module 15 is configured to perform a transformation process on the first fused image before fusing the first fused image and the second fused image, where the transformation process includes at least one of mirroring, translation, rotation, shearing, and deformation. That is, step 015 may be performed by the transformation module 15.
In some embodiments, the processor 20 is further configured to perform a transformation process on the first fused image prior to fusing the first fused image and the second fused image, the transformation process including at least one of mirroring, translation, rotation, shearing, and morphing. That is, step 015 may be performed by the processor 20.
In particular, it can be appreciated that the number of the first fused images is limited, so as to improve the diversity of the first fused images, the first fused images can be subjected to transformation before being fused with the second fused images, and different first fused images are generated by performing operations such as mirroring, translation, rotation, shearing, deformation and the like on defects of the first fused images, so that the diversity of training images obtained by fusion is improved.
For example, the processor 20 mirrors, translates, rotates, shears, or deforms the first fused image; of course, the processor 20 may also perform the translation process and the rotation process on the first fused image at the same time; or simultaneously carrying out translation processing, rotation processing and mirror image processing; or simultaneously carrying out translation treatment, rotation treatment, mirror image treatment and shearing treatment; or the translation process, the rotation process and the mirror image process are performed simultaneously, and the translation process, the rotation process and the mirror image process are performed for a plurality of times respectively at different distances, different angles and different symmetry axes, and the like, which are not listed here.
In performing the transformation, the processor 20 may randomly generate a transformation method to transform the first fused image; or the processor 20 may record the transformation processing manner that has been executed for each first fused image, and when each transformation processing is performed on the first fused image, first determine a transformation processing manner that is different from the previous transformation processing manner, and then perform the transformation processing on the first fused image, so that the first fused image after each transformation processing is different from the first fused image generated before, thereby further improving the diversity of the first fused image and improving the training effect.
Referring to fig. 2,3 and 5, in some embodiments, the first fused image is a plurality, and step 014 includes:
0141: selecting a target fusion image from the plurality of first fusion images; and
0142: And fusing the target fusion image with the second fusion image to obtain a training image.
In some embodiments, the fusion module 14 is further configured to select a target fusion image from the plurality of first fusion images; and fusing the target fusion image with the second fusion image to obtain a training image. That is, steps 0141 and 0142 may be performed by fusion module 14.
In some embodiments, the processor 20 is further configured to select a target fusion image from the plurality of first fusion images; and fusing the target fusion image with the second fusion image to obtain a training image. That is, step 0141 and step 0142 may be performed by processor 20.
Specifically, since the number of defective wafers is generally smaller, the number of first fused images is generally smaller, and in order to ensure the training effect, the first fused images are selected to include at least images of all types of defects. For example, selecting a typical first workpiece image corresponding to the type from the first workpiece images corresponding to each defect to obtain a first fused image corresponding to the type; or selecting a plurality of typical first workpiece images (such as two images, three images and the like) corresponding to the type from the first workpiece images corresponding to each defect so as to obtain a plurality of first fusion images corresponding to the type.
When the first fusion image and the second fusion image are fused, firstly, carrying out transformation processing on the first fusion image, then selecting one or more first fusion images from the transformed first fusion images to serve as target fusion images, and then fusing the target fusion images and the second fusion images; or one or more first fusion images are selected randomly from the first fusion images to serve as target fusion images, then transformation processing is carried out on the target fusion images, and finally the target fusion images are fused with the second fusion images. So that each second fused image is fused with a different first fused image. For example, if the second fused image is 1000, 1000 different training images may be generated. Thus, the diversity of the generated training images can be improved.
Referring to fig. 2, 3 and 6, in some embodiments, the data enhancement method further includes:
016: and taking all training images as a training set, and inputting the training images into the target detection model for training so as to enable the target detection model to converge.
In some embodiments, the data enhancement device further includes a training module 16. The training module 16 is configured to input all training images as a training set to the target detection model for training, so as to make the target detection model converge. That is, step 016 may be implemented by training module 16.
In some embodiments, the processor 20 is further configured to input all training images as a training set to the target detection model for training, so that the target detection model converges. That is, step 016 may be implemented by the processor 20.
Specifically, after the training images are obtained, all the training images can be input into the target detection model for training until the target detection model converges, so that the training of the target detection model is realized. The convergence of the target detection model refers to that when the target detection model trained by the training image can accurately detect the defect of the workpiece 300, the target detection model can be considered to converge, for example, the detection accuracy of the defect of the workpiece 300 reaches a predetermined accuracy (for example, 90%, 95%, 98% and the like).
Referring to fig. 2, 3 and 7, in some embodiments, step 016 comprises:
0161: marking the type and the position of the defect corresponding to the first fusion image in the training image to generate a verification image;
0162: inputting the training set to the target detection model to output a detection result;
0163: determining a loss value according to the verification image and the detection result; and
0164: And adjusting the target detection model according to the loss value so as to enable the target detection model to be converged.
In some embodiments, the training module 16 is further configured to annotate the type and location of the defect in the training image that corresponds to the first fused image to generate the verification image; inputting the training set to the target detection model to output a detection result; determining a loss value according to the verification image and the detection result; and adjusting the target detection model according to the loss value so as to enable the target detection model to be converged. That is, steps 0161 to 0164 may be performed by the training module 16.
In some embodiments, the processor 20 is further configured to annotate the type and location of the defect in the training image corresponding to the first fused image to generate a verification image; inputting the training set to the target detection model to output a detection result; determining a loss value according to the verification image and the detection result; and adjusting the target detection model according to the loss value so as to enable the target detection model to be converged. That is, steps 0161 to 0164 may be performed by the processor 20.
Specifically, after the training image is obtained, defects in the training image may be labeled in advance. For example, the processor 20 may obtain the position of the first fused image in the second fused image and the type of defect corresponding to the first fused image during fusion; at this point, the processor 20 may automatically label the defects in the training image, such as by marking the type and location of the defects in the training image. After labeling all the training images, a verification image corresponding to each training image can be generated, and the processor 20 generates a training set according to the training images and the verification images, for example, the training set is a set of all the training images and all the verification images.
During training, a training image is firstly input into a target detection model, then the target detection model outputs a detection result, the detection result comprises the type and the position of the defect of the training image, then the detection result is compared with a corresponding verification image, for example, whether the type of the defect of the comparison detection result is the same as the type of the defect of the corresponding verification image or not is judged, and the deviation of the position is confirmed, so that a loss value is confirmed.
The processor 20 adjusts the object detection model based on the loss values such that the object detection model converges. For example, the type detection parameters of the defects are adjusted according to whether the types of the defects of the detection result and the corresponding verification image are the same, the position detection parameters of the defects are adjusted according to the defect position deviation of the detection result and the corresponding verification image, and the training of the target detection model is realized through the training set comprising the training image and the verification image, so that the target detection model is converged, and the detection effect of the target detection model is ensured.
Referring to fig. 2,3 and 8, in some embodiments, step 0163 comprises:
01631: comparing the type of the defect in the detection result with the type of the defect of the corresponding verification image to determine a type loss value;
01632: comparing the position of the defect in the detection result with the position of the defect of the verification image corresponding to the verification set to determine a position loss value;
01633: a penalty value is determined based on the type penalty value and the location penalty value.
In some embodiments, the second obtaining module 13 is further configured to compare the type of the defect in the detection result with the type of the defect of the corresponding verification image, so as to determine a type loss value; comparing the position of the defect in the detection result with the position of the defect of the verification image corresponding to the verification set to determine a position loss value; a penalty value is determined based on the type penalty value and the location penalty value. That is, steps 1631 to 01633 may be performed by the second acquisition module 13.
In some embodiments, the processor 20 is further configured to compare the type of defect in the detection result with the type of defect of the corresponding verification image to determine a type loss value; comparing the position of the defect in the detection result with the position of the defect of the verification image corresponding to the verification set to determine a position loss value; a penalty value is determined based on the type penalty value and the location penalty value. That is, step 01631 and step 01633 can be performed by the processor 20.
Specifically, when determining the loss value, the type of the defect in the detection result and the type of the defect corresponding to the corresponding verification image may be compared to determine the type loss value, if the type of the defect and the type of the defect are the same, the loss value is determined to be 0, and if the type of the defect and the type of the defect are different, the loss value is determined to be 1.
The location of the defect in the detection result and the location of the defect in the corresponding verification image may then be compared to determine a location loss value. If the positions of the defects in the detection result are marked by the first defect frame, verifying that the corresponding defects in the image are marked by the second defect frame, and calculating the difference value of the position coordinates of the first defect frame and the second defect frame (such as the difference value of the position coordinates of the centers of the first defect frame and the second defect frame), and determining the position loss value according to the difference value, wherein the larger the difference value is, the larger the position loss value is.
Since the importance of the determination of the defect type is high, when determining the loss value according to the type loss value and the position loss value, a larger weight may be given to the type loss value, for example, loss value=a×type loss value+b×position loss value, where a is greater than b. Thereby ensuring that the processor 20 detects the type of defect with accuracy after adjusting the target detection model according to the loss value.
Referring to fig. 2, 3 and 9, in some embodiments 016 further comprises:
0165: when the loss value is smaller than a preset threshold value, determining that the target detection model converges;
0166: and when the loss value is greater than a preset threshold value, carrying out transformation processing on the training set, and training the target detection model again according to the transformed training set until the target detection model converges.
In some embodiments, the training module 16 is further configured to determine that the target detection model converges when the loss value is less than a preset threshold; and when the loss value is greater than a preset threshold value, carrying out transformation processing on the training set, and training the target detection model again according to the transformed training set until the target detection model converges. That is, step 0165 and step 0166 may be performed by the second acquisition module 13.
In some embodiments, the processor 20 is further configured to determine that the target detection model converges when the loss value is less than a preset threshold; and when the loss value is greater than a preset threshold value, carrying out transformation processing on the training set, and training the target detection model again according to the transformed training set until the target detection model converges. That is, step 0165 and step 0166 may be performed by the processor 20.
Specifically, after the target detection model is adjusted according to the loss values, whether the target detection model converges is determined, after the training set is input to the target detection model, the target detection model outputs the loss values, if one loss value is output in each training image, the average value of the loss values corresponding to all the training images is calculated as the final output loss value, at this time, the processor 20 judges whether the loss value is greater than a preset threshold, if the loss value is less than or equal to the preset threshold, the detected loss is smaller, the detection accuracy reaches the requirement, and it can be determined that the target detection model converges.
If the loss value is larger than the preset threshold value, the loss of detection is too large, the detection accuracy still does not meet the requirement, and at the moment, the target detection model can be determined to be not converged, and training needs to be continued. At this time, the training set may be subjected to a transformation process, which is specifically as follows:
referring to fig. 10, for example, the processor 20 performs a mirroring process on each training image P1 to obtain a mirror image P2 of each training image P1, and uses the mirror image P2 as a new training image P1. The mirror image P2 after the mirror image processing and the training image P1 are mirror-symmetrical, and the symmetry axis may be arbitrary, for example, mirror-image processing is performed with any side of the training image P1 as the symmetry axis (mirror-image processing is performed with the rightmost side of the training image P1 as the symmetry axis in fig. 10), or mirror-image processing is performed with a diagonal line of the training image P1 or a line of midpoints of any two sides as the symmetry axis, so as to obtain a plurality of new training images through the mirror-image processing.
Referring to fig. 11, for another example, the processor 20 performs a panning process on each training image P1 to obtain a panning image P3 of each training image P1, and uses the panning image P3 as a new training image P1. Specifically, a predetermined image area (i.e., an area occupied by the training image P1) is determined by the training image P1, then the training image P1 is translated, such as left-shifted, right-shifted, left-shifted and the like (right-shifted in fig. 11), then an image of the predetermined image area (i.e., a translated image P3) is taken as a new training image P1, and the position of the translated defect in the image is changed, so as to obtain a plurality of new training images P1.
Referring to fig. 12, for another example, the processor 20 performs a rotation process on each training image P1 to obtain a rotation image P4 of each training image P1, and uses the rotation image P4 as a new training image P1. Specifically, a predetermined image area is determined by using the training image P1, then the training image P1 is rotated, for example, by 10 degrees, 30 degrees, 60 degrees, 90 degrees, 140 degrees, etc. (fig. 12 is rotated by 30 degrees counterclockwise), then the image (and the rotated image P4) of the predetermined image area is used as a new training image P1, and the position of the rotated defect in the image is changed, so as to obtain a plurality of new training images P1.
Referring to fig. 13, for another example, the processor 20 performs a cropping process on each training image P1 to obtain a cropped image P5 of each training image, and uses the cropped image P5 as a new training image P1. Specifically, a predetermined image area is determined by using the training image P1, then the training image P1 is cut, for example, 1/4, 1/3, 1/2, etc. of the cut training image P1 (fig. 13 is 1/2 of the cut training image), and then an image of the predetermined image area (i.e., the cut image P5) is used as a new training image P1, so as to obtain a plurality of new training images P1.
Referring to fig. 14, for another example, the processor 20 performs a morphing process on each training image P1 to obtain a morphed image P6 of each training image P1, and uses the morphed image P6 as a new training image P1. Specifically, a predetermined image area is determined by using a training image P1, then the training image P1 is deformed, for example, the training image P1 is compressed in a transverse direction, so that the original rectangular training image P1 is changed into a rectangle with a notch, then an image of the predetermined image area (i.e., a deformed image P6) is used as a new training image P1, and the position and the shape of the deformed defect in the image are changed, so that a plurality of new training images P1 are obtained.
Of course, the processor 20 may also perform the translation process and the rotation process on the training image at the same time; or simultaneously carrying out translation processing, rotation processing and mirror image processing; or simultaneously carrying out translation treatment, rotation treatment, mirror image treatment and shearing treatment; or the translation process, the rotation process and the mirror image process are performed simultaneously, and the translation process, the rotation process and the mirror image process are performed for a plurality of times respectively at different distances, different angles and different symmetry axes, and the like, which are not listed here.
Similarly, when the training images are subjected to transformation processing, transformation processing can be synchronously performed on verification images corresponding to the training images, and the transformation processing modes of the training images corresponding to the verification images are the same, so that after transformation processing of a training set, the training images and the verification images still correspond, and the subsequent training effect on the target detection model is ensured.
Then, the processor 20 performs a second training on the target detection model according to the training set after the transformation processing, after the training, determines whether the target detection model converges according to the loss value again, and if not, performs the transformation processing on the training set again, and performs a third training according to the training set after the transformation processing, so that the cycle is performed until the trained target detection model converges.
In other embodiments, in order to ensure accuracy of judgment of convergence of the target detection model, after training the target detection model according to the training set, a preset verification set is input to enable the target detection model to output a loss value, and images in the verification set are different from training images in the training set, so that the verification set can accurately verify whether the target detection model is converged, if the number of the training images is multiple, one part of the training images is used as training images of the training set, and the other part of the training images is used as images of the verification set; or the verification set is obtained according to the transformation processing of the training set.
Referring to fig. 2 and 3, in some embodiments, the processor 20 is further configured to detect an image of the workpiece 300 according to the converged object detection model to determine the type, location, and confidence of the defect; and outputting the type, the position and the confidence of the defect when the confidence is larger than a confidence threshold corresponding to the type of the defect.
Specifically, after the training of the target detection model is completed, the sensor 210 acquires an image of the workpiece 300 to be measured, and the processor 20 detects the image of the workpiece 300 according to the target detection model to determine the type, position, and confidence of the defect. When the confidence coefficient is larger than the confidence coefficient threshold value corresponding to the type of the current defect, the current defect detection accuracy can be determined, and accordingly the type, the position and the confidence coefficient of the current defect are output to serve as detection results.
Wherein the confidence coefficient threshold value corresponds to the type of the defect, and the defects of different types correspond to different confidence coefficient threshold values, so that the detection accuracy of the defects of different types is improved in a targeted manner, the target detection model is an end-to-end model, only one model and one target function are used for the end-to-end model, compared with the training effect which is difficult to reach the optimum and is caused by the possible fine difference of the training targets of the multi-module model, errors among different modules can affect each other, the implementation and maintenance of the end-to-end model are simpler, the trained model can reach the optimum effect, the detection effect is better, and the engineering complexity is lower.
Referring to FIG. 15, one or more non-transitory computer-readable storage media 300 embodying a computer program 302 that, when executed by one or more processors 20, enables the processors 20 to perform the calibration method of any of the embodiments described above, are provided in embodiments of the present application.
For example, referring to fig. 1-3, when the computer program 302 is executed by one or more processors 20, the processor 20 is caused to perform the steps of:
011: acquiring a first workpiece image with a defect;
012: identifying and intercepting an image area where the defect is located in the first workpiece image to serve as a first fusion image;
013: acquiring a second workpiece image without defects as a second fused image; and
014: The first fused image and the second fused image are fused to obtain a training image.
As another example, referring to fig. 2,3 and 4, when the computer program 302 is executed by one or more processors 20, the processor 20 may further perform the steps of:
015: before fusing the first fused image and the second fused image, performing transformation processing on the first fused image, wherein the transformation processing comprises at least one of mirroring, translation, rotation, shearing and deformation.
In the description of the present specification, reference is made to the terms "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples," etc., meaning that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the various embodiments or examples described in this specification and the features of the various embodiments or examples may be combined and combined by those skilled in the art without contradiction.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and further implementations are included within the scope of the preferred embodiment of the present application in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.
While embodiments of the present application have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the application, and that variations, modifications, alternatives and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the application.
Claims (11)
1. A method of data enhancement, comprising:
acquiring a first workpiece image with a defect;
Identifying and intercepting an image area where the defect is located in the first workpiece image to serve as a first fusion image;
Acquiring a second workpiece image without defects as a second fused image; and
Fusing the first fused image and the second fused image to obtain a training image;
The first fusion image is a plurality of, the fusion of the first fusion image and the second fusion image to obtain a training image includes:
Selecting a target fusion image from a plurality of first fusion images; and
Fusing the target fusion image with the second fusion image to obtain the training image;
The selecting the target fusion image from the plurality of first fusion images comprises the following steps:
Performing transformation processing on the plurality of first fusion images, and selecting one or more of the transformed first fusion images as the target fusion image; or alternatively
And randomly selecting one or more of the first fusion images, and performing transformation processing to serve as the target fusion image.
2. The data enhancement method according to claim 1, further comprising:
Before fusing the first fused image and the second fused image, performing transformation processing on the first fused image, wherein the transformation processing comprises at least one of mirroring, translation, rotation, shearing and deformation.
3. The data enhancement method according to any one of claims 1-2, further comprising:
and taking all the training images as a training set, and inputting the training images into a target detection model for training so as to enable the target detection model to converge.
4. The data enhancement method according to claim 3, wherein said inputting all of said training images as a training set to a target detection model for training so as to converge said target detection model comprises:
labeling the type and the position of the defect corresponding to the first fusion image in the training image to generate a verification image;
inputting the training set to the target detection model to output a detection result;
determining a loss value according to the verification image and the detection result; and
And adjusting the target detection model according to the loss value so as to enable the target detection model to converge.
5. The data enhancement method according to claim 4, wherein said determining a loss value from said verification image and said detection result comprises:
Comparing the type of the defect in the detection result with the type of the defect of the corresponding verification image to determine a type loss value;
Comparing the position of the defect in the detection result with the position of the defect of the verification image corresponding to the verification set to determine a position loss value;
And determining the loss value according to the type loss value and the position loss value.
6. The data enhancement method according to claim 4 or 5, wherein said inputting all of said training images as a training set to said object detection model for training so as to converge said object detection model, further comprises:
When the loss value is smaller than a preset threshold value, determining that the target detection model converges;
And when the loss value is larger than the preset threshold value, carrying out transformation processing on the training set, and training the target detection model again according to the transformed training set until the target detection model converges.
7. The data enhancement method according to claim 1, wherein the second workpiece images are plural, and image backgrounds of the plural second workpiece images are different from each other.
8. The data enhancement method of claim 1, wherein the workpiece comprises a wafer and the defect comprises at least one of a foreign object, a residual glue, and an oxidation.
9. A data enhancement device, comprising:
The first acquisition module is used for acquiring a first workpiece image with defects;
the identification module is used for identifying and intercepting an image area where the defect is located in the first workpiece image to be used as a first fusion image;
The second acquisition module is used for acquiring a second workpiece image without defects to be used as a second fusion image;
the fusion module is used for fusing the first fusion image and the second fusion image to obtain a training image;
The fusion module is used for selecting a target fusion image from the plurality of first fusion images; fusing the target fusion image with the second fusion image to obtain the training image;
The fusion module is also used for: performing transformation processing on the plurality of first fusion images, and selecting one or more of the transformed first fusion images as the target fusion image; or randomly selecting one or more of the first fusion images, and performing transformation processing to serve as the target fusion image.
10. A data enhancement device comprising a processor configured to:
acquiring a first workpiece image with a defect;
Identifying and intercepting an image area where the defect is located in the first workpiece image to serve as a first fusion image;
Acquiring a second workpiece image without defects as a second fused image; and
Fusing the first fused image and the second fused image to obtain a training image;
The processor is used for selecting a target fusion image from the plurality of first fusion images; fusing the target fusion image with the second fusion image to obtain the training image;
The processor is further used for carrying out transformation processing on the plurality of first fusion images, and selecting one or more of the transformed first fusion images as the target fusion image; or randomly selecting one or more of the first fusion images, and performing transformation processing to serve as the target fusion image.
11. A non-transitory computer readable storage medium containing a computer program which, when executed by a processor, causes the processor to perform the data enhancement method of any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110260102.7A CN112884691B (en) | 2021-03-10 | 2021-03-10 | Data enhancement device, data enhancement apparatus, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110260102.7A CN112884691B (en) | 2021-03-10 | 2021-03-10 | Data enhancement device, data enhancement apparatus, and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112884691A CN112884691A (en) | 2021-06-01 |
| CN112884691B true CN112884691B (en) | 2024-09-10 |
Family
ID=76054058
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110260102.7A Active CN112884691B (en) | 2021-03-10 | 2021-03-10 | Data enhancement device, data enhancement apparatus, and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112884691B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109583489A (en) * | 2018-11-22 | 2019-04-05 | 中国科学院自动化研究所 | Defect classifying identification method, device, computer equipment and storage medium |
| CN111640091A (en) * | 2020-05-14 | 2020-09-08 | 阿丘机器人科技(苏州)有限公司 | Method for detecting product defects and computer storage medium |
| CN111709948A (en) * | 2020-08-19 | 2020-09-25 | 深兰人工智能芯片研究院（江苏）有限公司 | Method and device for detecting defects of container |
| CN111814867A (en) * | 2020-07-03 | 2020-10-23 | 浙江大华技术股份有限公司 | Defect detection model training method, defect detection method and related device |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6804381B2 (en) * | 2000-04-18 | 2004-10-12 | The University Of Hong Kong | Method of and device for inspecting images to detect defects |
| CN108875821A (en) * | 2018-06-08 | 2018-11-23 | Oppo广东移动通信有限公司 | The training method and device of disaggregated model, mobile terminal, readable storage medium storing program for executing |
| CN109919888B (en) * | 2019-02-26 | 2023-09-19 | 腾讯科技（深圳）有限公司 | Image fusion method, model training method and related device |
-
2021
- 2021-03-10 CN CN202110260102.7A patent/CN112884691B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109583489A (en) * | 2018-11-22 | 2019-04-05 | 中国科学院自动化研究所 | Defect classifying identification method, device, computer equipment and storage medium |
| CN111640091A (en) * | 2020-05-14 | 2020-09-08 | 阿丘机器人科技(苏州)有限公司 | Method for detecting product defects and computer storage medium |
| CN111814867A (en) * | 2020-07-03 | 2020-10-23 | 浙江大华技术股份有限公司 | Defect detection model training method, defect detection method and related device |
| CN111709948A (en) * | 2020-08-19 | 2020-09-25 | 深兰人工智能芯片研究院（江苏）有限公司 | Method and device for detecting defects of container |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112884691A (en) | 2021-06-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112161619B (en) | Pose detection method, three-dimensional scanning path planning method and detection system | |
| CN109934847B (en) | Method and device for estimating posture of weak texture three-dimensional object | |
| CN111982921B (en) | Method and device for detecting hole defects, conveying platform and storage medium | |
| CN112884743B (en) | Detection method and device, detection equipment and storage medium | |
| CN111814850A (en) | Defect detection model training method, defect detection method and related device | |
| CN110298891A (en) | The method and device that Camera extrinsic precision is assessed automatically | |
| CN115713476A (en) | Visual detection method and device based on laser welding and readable storage medium | |
| CN109389105A (en) | A kind of iris detection and viewpoint classification method based on multitask | |
| CN117274258B (en) | Method, system, equipment and storage medium for detecting defects of main board image | |
| CN114549513B (en) | Part identification method, part identification device, quality inspection method, electronic equipment and storage medium | |
| CN117788400A (en) | Binocular vision-based photovoltaic module defect detection method and system | |
| CN112884691B (en) | Data enhancement device, data enhancement apparatus, and storage medium | |
| CN113066069B (en) | Adjustment method and device, adjustment device and storage medium | |
| CN114594102B (en) | Machine vision-based data line interface automatic detection method | |
| CN112884744A (en) | Detection method and device, detection equipment and storage medium | |
| CN112926438B (en) | Detection method and device, detection equipment and storage medium | |
| CN111815560B (en) | Photovoltaic power station fault detection method and device, portable detection equipment and storage medium | |
| CN112926439B (en) | Detection method and device, detection equipment and storage medium | |
| CN112926437B (en) | Detection method and device, detection equipment and storage medium | |
| CN109449096B (en) | Method for identifying and detecting wafer chip | |
| CN112950563A (en) | Detection method and device, detection equipment and storage medium | |
| JP7317913B2 (en) | Information processing device, information processing method and information processing program | |
| CN111832439B (en) | Multi-face rapid identification method and processing terminal | |
| CN113192021A (en) | Detection method and device, detection equipment and storage medium | |
| CN118135179B (en) | Cascade identification method and device for cutting plate, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |