CN113724218B - Method, device and storage medium for identifying chip welding defect by image - Google Patents
Method, device and storage medium for identifying chip welding defect by image Download PDFInfo
- Publication number
- CN113724218B CN113724218B CN202110992821.8A CN202110992821A CN113724218B CN 113724218 B CN113724218 B CN 113724218B CN 202110992821 A CN202110992821 A CN 202110992821A CN 113724218 B CN113724218 B CN 113724218B
- Authority
- CN
- China
- Prior art keywords
- area
- determining
- defect
- shape
- extension
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000007547 defect Effects 0.000 title claims abstract description 156
- 238000003466 welding Methods 0.000 title claims abstract description 90
- 238000000034 method Methods 0.000 title claims abstract description 41
- 238000003860 storage Methods 0.000 title claims abstract description 9
- 238000013528 artificial neural network Methods 0.000 claims abstract description 69
- 238000001514 detection method Methods 0.000 claims abstract description 65
- 230000011218 segmentation Effects 0.000 claims abstract description 15
- 230000002950 deficient Effects 0.000 claims description 38
- 238000004891 communication Methods 0.000 claims description 22
- 238000012549 training Methods 0.000 claims description 14
- XBWAZCLHZCFCGK-UHFFFAOYSA-N 7-chloro-1-methyl-5-phenyl-3,4-dihydro-2h-1,4-benzodiazepin-1-ium;chloride Chemical compound [Cl-].C12=CC(Cl)=CC=C2[NH+](C)CCN=C1C1=CC=CC=C1 XBWAZCLHZCFCGK-UHFFFAOYSA-N 0.000 claims description 5
- 238000012545 processing Methods 0.000 claims description 4
- 238000002372 labelling Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 6
- 238000005491 wire drawing Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 230000004927 fusion Effects 0.000 description 4
- 229910000679 solder Inorganic materials 0.000 description 3
- 238000011179 visual inspection Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012858 packaging process Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/042—Knowledge-based neural networks; Logical representations of neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/187—Segmentation; Edge detection involving region growing; involving region merging; involving connected component labelling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30148—Semiconductor; IC; Wafer
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The application relates to the technical field of image recognition, in particular to a method, a device and a storage medium for recognizing chip welding defects through images. The method comprises the following steps: acquiring a target image, wherein the target image comprises a welding area after the welding of the chip pins is finished; inputting the target image into a neural network algorithm; determining the shape and/or position of a defect area in the target image based on a neural network algorithm; determining a defect class based on the shape and/or location of the defect region; the neural network algorithm comprises a computer vision algorithm, a target detection algorithm and a semantic segmentation algorithm combined with an expert knowledge base; the computer vision algorithm is used for determining the shape and/or position of the defect area with skewed pins, the semantic segmentation algorithm is used for determining the shape and/or position of the defect area with low occurrence frequency in combination with the expert knowledge base, and the target detection algorithm is used for determining the shape and/or position of the defect area with too short extension area.
Description
Technical Field
The application relates to the technical field of image recognition, in particular to a method, a device and a storage medium for recognizing chip welding defects through images.
Background
In the processing and manufacturing of electronic parts, the quality of the product often needs to be monitored so as to prevent workpieces which are not in accordance with the specification of the product or the quality of the workpiece is not in accordance with the standard from flowing out. The quality of die bonding is considered an important point for defect detection, as it often directly affects whether the product can function properly.
Currently, chip defect detection is mainly performed by special visual inspection personnel, however, manual visual inspection can lead to detection time being unable to be ensured and judgment deviation exists among different visual inspection personnel. If the chip defects are to be automatically detected by a computer vision method, the following problems exist: the types of the chip defects are often more and the positions of the chip defects are not fixed, so that the chip defects are difficult to judge through the traditional computer vision algorithm; the occurrence frequency of various defects is small, so that partial defect sample pictures are small, and the training of a deep learning algorithm is not facilitated.
Disclosure of Invention
The application provides a method, a device and a storage medium for identifying chip welding defects, which can solve the problem that a deep learning algorithm is difficult to train due to the lack of a chip welding defect picture sample in the related technology, and the chip welding defects cannot be diagnosed through an image identification automatic technology.
To solve the technical problem described in the background art, a first aspect of the present application provides a method for identifying a chip bonding defect by using an image, the method comprising the following steps:
Acquiring a target image, wherein the target image comprises a welding area after the welding of the chip pins is finished;
Inputting the target image into a neural network algorithm;
Determining the shape and/or position of a defect region in the target image based on the neural network algorithm;
determining a defect category based on the defect region shape and/or location;
The neural network algorithm comprises a computer vision algorithm, a target detection algorithm and a semantic segmentation algorithm combined with an expert knowledge base; the computer vision algorithm is used for determining the shape and/or the position of the pin skew defect area, the combined expert knowledge base semantic segmentation algorithm is used for determining the defect area with low occurrence frequency, and the target detection algorithm is used for determining the defect area with too short extension area.
Optionally, after the step of obtaining the target image, which includes the bonding area after the die pin bonding is completed, before the step of inputting the target image into the neural network algorithm is performed, the steps are further performed:
acquiring a target image, and determining that the target image comprises a plurality of welding areas;
Dividing the target image based on an image dividing process to form a plurality of sub-target images;
Such that one of the weld zones is included in each of the sub-target images.
Optionally, the step of determining the shape and/or the position of the defect region located in the target image based on the neural network algorithm includes:
determining the welding zone based on the neural network algorithm;
determining all defective pixels located at the weld zone locations based on the neural network algorithm; the set of all defective pixels is the defective area;
The shape and/or position of the defective area is determined based on the position of each defective pixel and the commonly constituted shape.
Optionally, the neural network algorithm includes at least the computer vision algorithm;
the step of determining the welding area based on the neural network algorithm comprises the following steps:
based on a computer vision algorithm, performing binarization processing on the target image to form a binarized image;
identifying all connected areas of the binarized image;
Determining a maximum communication area in the communication areas;
And determining each corner point of the welding area according to the maximum communication area.
Optionally, the determining, based on the neural network algorithm, all defective pixels located at the weld zone location; a step of collecting all defective pixels as the defective area, comprising:
Determining pixel points which are transversely beyond the corner points in the maximum communication area as defective pixels based on each corner point of the welding area;
the set of all the defective pixels forms the defective area.
Optionally, before the step of inputting the target image into the neural network algorithm, the method further includes:
Acquiring a plurality of sample images, wherein each sample image comprises a welding area after the welding of a chip pin is finished;
Labeling the defect areas in each sample image based on an expert knowledge base, and determining the types of the defect areas;
training the neural network algorithm by using the sample image marked with the defect area and determining the defect type to form the neural network algorithm combined with the expert knowledge base.
Optionally, the neural network algorithm combined with the expert knowledge base at least comprises a semantic segmentation algorithm combined with the expert knowledge base.
Optionally, the step of determining the shape and/or the position of the defect area located at the welding area based on the neural network algorithm includes:
determining the relative position relation between a welding spot area and a chip pin area in the target image based on a neural network algorithm;
Determining the shape of an extension area based on the relative position relationship, wherein the extension area is an area of the chip pin area extending beyond the welding spot area;
And determining whether the extension area is a defect area or not based on the extension length of the extension area shape, wherein the extension length is the length of the extension area in the extending direction of the chip pin area, and the shape and/or the position of the extension area is the shape and/or the position of the defect area.
Optionally, the step of determining the relative positional relationship between the pad area and the chip pin area in the target image based on the neural network algorithm includes:
generating a first detection frame and a second detection frame in the target image based on a neural network algorithm, wherein the first detection frame is used for determining the shape and/or the position of the welding spot area, and the second detection frame is used for determining the shape and/or the position of the chip pin area;
and enabling the relative position relation between the first detection frame and the second detection frame to be used as the relative position relation between the welding spot area and the chip pin area in the target image.
Optionally, the step of determining an extension area shape based on the relative positional relationship, where the extension area is an area where the chip pin area extends beyond the solder joint area, includes:
Determining an extension detection frame from which the second detection frame extends out of the first detection frame based on a relative positional relationship between the first detection frame and the second detection frame;
the extension detection frame is used for determining the shape and/or the position of the extension area.
Optionally, the step of determining whether the extension area is a defect area based on an extension length of the extension area shape, wherein the extension length is a length of the extension area in the extending direction of the chip pin area, and the shape and/or position of the extension area is the shape and/or position of the defect area includes:
determining the length of the extension detection frame as the extension length of the extension area shape;
and determining whether the extension area is a defect area based on the length of the extension detection frame.
Optionally, the method further comprises the step of determining whether the extension region is a defective region after the step of determining the extension length of the extension region shape:
and when the extension length is smaller than the extension length threshold value, determining that the defect type of the defect area is that the extension area is too short.
A second aspect of the present application provides an apparatus for image recognition of a die bonding defect, the apparatus comprising a processor and a memory, the memory having stored therein at least one instruction or program loaded and executed by the processor to implement the method for image recognition of a die bonding defect according to the first aspect of the present application.
A third aspect of the present application provides a computer readable storage medium having stored therein at least one instruction or program loaded and executed by a processor to implement a method of image recognition of die bonding defects according to the first aspect of the present application.
The technical scheme of the application at least comprises the following advantages: the target image comprising the welding area after the welding of the chip pins is finished is acquired, the target image is input into a neural network algorithm, the shape and/or the position of a defect area positioned at the position of the welding area are determined based on the neural network algorithm, the defect type can be accurately determined based on the shape and/or the position of the defect area, and the problem that the chip welding defect cannot be diagnosed through an image identification automatic identification technology is avoided.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present application, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a method for identifying a chip bonding defect according to an embodiment of the present application;
FIG. 1a illustrates a schematic view of a target image in accordance with one embodiment;
FIG. 1b shows a schematic diagram of a target image with pin skew defects;
FIGS. 1c to 1f are schematic views sequentially showing target images of the defects of incomplete welding spots, foreign matter defects of welding spots, trailing defects of wire drawing of welding spots and welding spot fusion welding defects;
FIG. 1g shows a schematic view of a target image with extended area overcomplete notches;
FIG. 2 is a schematic diagram showing the determination of the maximum connected region of the target image shown in FIG. 1 b;
FIG. 3 shows a schematic diagram of a target image for generating a detection frame;
fig. 4 is a block diagram showing a structure of an apparatus for recognizing a chip bonding defect according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made more apparent and fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the application are shown. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
In the description of the present application, it should be noted that the directions or positional relationships indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings, are merely for convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
In the description of the present application, it should be noted that, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be either fixedly connected, detachably connected, or integrally connected, for example; can be mechanically or electrically connected; the two components can be directly connected or indirectly connected through an intermediate medium, or can be communicated inside the two components, or can be connected wirelessly or in a wired way. The specific meaning of the above terms in the present application will be understood in specific cases by those of ordinary skill in the art.
In addition, the technical features of the different embodiments of the present application described below may be combined with each other as long as they do not collide with each other.
In the chip packaging process, pins (pins) of a semiconductor chip are required to be soldered in pads (pads) of a carrier through solder joints (points). The quality of die bonding is considered an important point for defect detection, as it often directly affects whether the product can function properly.
Fig. 1 is a flowchart of a method for identifying a chip bonding defect by using an image, which is provided in an embodiment of the present application, and is used for diagnosing whether a bonding defect occurs after the bonding of an identification pin (pin) is completed, as can be seen from fig. 1, the method includes the following steps that are sequentially performed:
step S1: and obtaining a target image, wherein the target image comprises a welding area after the welding of the chip pins is finished.
Fig. 1a shows a schematic view of a target image in an embodiment, and as can be seen in fig. 1a, the target image 100 includes a bonding area 110 after die-pin bonding is completed, i.e., a die-pin area 130 and a bonding-pad area 120 are also located in the bonding area 110.
Step S2: so that the target image is input to a neural network algorithm.
The neural network algorithm comprises a computer vision algorithm, a target detection algorithm and a semantic segmentation algorithm combined with an expert knowledge base.
Step S3: based on the neural network algorithm, a shape and/or location of a defect region located at the weld zone location is determined.
The computer vision algorithm is used for determining the shape and/or the position of a defect area with skewed pins, the combined expert knowledge base semantic segmentation algorithm is used for determining the shape and/or the position of the defect area with low occurrence frequency, and the target detection algorithm is used for determining the shape and/or the position of the defect area with too short extension area.
Step S4: based on the defect region shape and/or location, a defect class is determined.
The neural network algorithm is based on the neural network algorithm, and can determine defect types, for example, a computer vision algorithm can identify pin skew defects, an expert knowledge base semantic segmentation algorithm can identify defects with low occurrence frequency, and a target detection algorithm can identify defects with too short extension area.
The pin skew defect is that the pin area of the chip is laterally offset from the welding area after the welding is completed. Referring to fig. 1b, which shows a schematic view of a target image with pin skew defects, it can be seen from fig. 1b that the lateral direction is the x-direction in the target image 100, and the die pin area 130 and the pad area 120 are each offset from the bonding area 110 in the x-direction, i.e., the right portions of the die pin area 130 and the pad area 120 exceed the right boundary of the bonding area 110.
Among them, defects with low occurrence frequency include, for example, a weld spot incomplete defect, a weld spot foreign matter defect, a weld spot wire drawing defect, a weld zone fusion welding defect, and the like. Fig. 1c to 1f are schematic views sequentially showing target images in which there are a weld spot incomplete defect, a weld spot foreign matter defect, a weld spot wire drawing tail defect, and a weld zone fusion welding defect.
Referring to fig. 1c, there is a missing region 141 in the pad area 120, so that there is a pad incomplete defect in the pad area 110 of the target image shown in fig. 1 c.
Referring to fig. 1d, the spot welding area 120 thereof has a spot welding foreign matter area 142, so that the spot welding foreign matter defect exists in the spot welding area 110 of the target image shown in fig. 1 d.
Referring to fig. 1e, the pad area 120 thereof has a streaking area 143, so that the pad streaking defect exists in the pad area 110 of the target image shown in fig. 1 e.
Referring to fig. 1f, the weld zone 110 has a weld zone 144 such that the weld zone 110 of the target image of fig. 1f has a weld zone weld defect.
The defect that the extension area is too short is a defect that the welding spot area is too short when the chip pin area extends longitudinally. Fig. 1g shows a schematic view of a target image with an extended area over-run defect, and it can be seen from fig. 1g that the die pin area 130 is too short in the longitudinal direction (y-direction shown in fig. 1 g) relative to the extended area 145 of the bond pad area 120.
According to the embodiment, the target image comprising the welding area after the welding of the chip pins is completed is acquired, so that the target image is input into a neural network algorithm, the shape and/or the position of the defect area positioned at the position of the welding area are determined based on the neural network algorithm, the defect type can be accurately determined based on the shape and/or the position of the defect area, and the problem that the chip welding defect cannot be diagnosed through an image identification automatic technology is avoided.
Alternatively, in order to facilitate analysis of weld defect problems in weld zones one by one, there may be only one weld zone in one target image, while there may be multiple weld zones in the other target image, for which image segmentation is required so that multiple sub-target images are formed after segmentation, one weld zone in one sub-target image. Namely, in step S1: acquiring a target image, wherein the target image comprises a welding area after welding of chip pins is completed, and step S2 is carried out: before the step of inputting the target image into the neural network algorithm, the method further comprises the following steps:
Step S11: and acquiring a target image, and determining that the target image comprises a plurality of welding areas.
Step S12: and dividing the target image based on an image dividing process to form a plurality of sub-target images.
Step S13: such that one of the weld zones is included in each of the sub-target images.
The sub-target image is then input into a neural network algorithm to identify the defective area opinion defect type.
In order to improve the accuracy of the determined shape and/or position of the defect area, step S3: based on the neural network algorithm, determining the shape and/or position of the defect region in the target image, including steps S311 to S313 sequentially performed:
step S311: the weld zone is determined based on the neural network algorithm.
Alternatively, the neural network algorithm includes at least a computer vision algorithm from which the weld zone may be determined. That is, the target image may be first binarized based on a computer vision algorithm to form a binarized image. All connected regions of the binarized image are identified again. And finally, determining the maximum communication area in the communication areas, and determining the welding area based on the maximum communication area.
The binary image includes black pixels and white pixels, that is, the gray value of any pixel point in the binary image is only two possible, which is 0 or 255. The binarization processing is to sequentially judge whether the gray value of each pixel point in the target image is larger than a gray threshold value, if so, the gray value of the pixel point is changed to 255, otherwise, the gray value of the pixel point is changed to 0. The pixel having a gray value of 255 is determined as the pixel of the target image object area, and the pixel having a gray value of 0 is determined as the pixel of the target image background area.
The communication area is a pixel point set in which all the pixel points are mutually communicated, namely any two pixel points in the communication area are mutually communicated. The connected regions in the binarized image represent the object regions in the corresponding target image.
In the process of identifying the connected region of the binary image, all black pixels in the binary image can be determined first, and then a set that all black pixels are connected to each other is determined as the connected region of the binary image.
Since the area occupied by the welding area in the target image is maximum, particularly for the target image including only one welding area, after the maximum communication area with the largest area is determined, each corner point of the welding area in the target image can be determined according to the maximum communication area, and the welding area can be determined according to the corner point. The communication area with the largest pixel points is the largest communication area, namely the largest communication area occupies the largest pixel points.
Fig. 2 shows a schematic diagram of determining the maximum connected region of the target image shown in fig. 1 b. As shown in fig. 2, for the maximum connected region 150 of the target image 100, four corner points A, B, C, D of the welding area can be determined according to the maximum connected region 150, and the welding area of the target image 100 can be determined according to the four corner points A, B, C, D. The corner A, B, D can be calculated and determined based on a corner detection algorithm, and after any three corners of the welding area are determined, the last corner C can be uniquely determined.
Step S312: determining all defective pixels located at the weld zone locations based on the neural network algorithm; the set of all defective pixels is the defective area.
After determining the respective corner points of the weld zone, the steps may be performed as described above: and determining the pixel points which are transversely beyond the corner points in the maximum communication area as defective pixels. The set of all defective pixels forms the defective area.
For example, after each corner point of the maximum communication area and the welding area is determined, whether the chip pins have pin skew defects is judged according to whether pixel points transversely exceeding the corner points exist in the maximum communication area.
And determining the pixel points with positions transversely exceeding the corner points as pin skew defect pixels, wherein the set of all the pin skew defect pixels is formed into a pin skew defect area.
As shown in fig. 2, the x-direction in fig. 2 is a lateral direction, and the right portion of the maximum communication area 150 laterally exceeds the corner points B and C of the bonding area, and the portion is a pin skew defect area 151, thereby determining that a pin skew defect exists in the target area shown in fig. 1B.
Step S313: the shape and/or position of the defective area is determined based on the position of each defective pixel and the commonly constituted shape.
In the step S3, whether there is a defect that the extension area is too short between the die pad and the solder joint is detected: determining the shape and/or position of a defect area in the target image based on the neural network algorithm, wherein the neural network algorithm at the step at least comprises a target detection algorithm. The step S3 thus includes the following steps S321 to S324, which are performed in order:
Step S321: and determining the relative position relation between the welding spot area and the chip pin area in the target image based on a neural network algorithm.
The first detection frame and the second detection frame can be generated in the target image, the shape and/or the position of the welding spot area in the target image are determined through the first detection frame, the shape and/or the position of the chip pin area in the target image are determined through the second detection frame, and the relative position relationship between the first detection frame and the second detection frame can be used as the relative position relationship between the welding spot area and the chip pin area.
Referring to fig. 3, which illustrates a schematic view of a target image for generating a detection frame, it can be seen from fig. 3 that a first detection frame 161 is used to determine the shape and/or position of a pad area, and a second detection frame 162 is used to determine the shape and/or position of a chip pin area.
Step S322: and determining an extension area based on the relative position relation, wherein the extension area is an area of the chip pin area extending beyond the welding spot area.
Since the first detection frame determines the welding spot area in the target image, the second detection frame determines the chip pin area in the target image, so that the area of the chip pin area extending beyond the welding spot area can be determined by extending the second detection frame out of the extending detection frame of the first detection frame, namely, the shape and/or the position of the extending area can be determined by extending the detection frame.
With continued reference to fig. 3, it can be seen that the second detection frame 162 extends upwardly in the y-direction (longitudinally) beyond the first detection frame 161 to form an extended detection frame 163, which extended detection frame 163 is used to determine the shape and/or location of the extended region.
Step S323: based on the extension length of the extension region, it is determined whether the extension region is a defective region.
The length of the extension detection frame can be determined to be the extension length of the extension area shape; and determining whether the extension area is a defect area based on the length of the extension detection frame.
Step S324: and when the extension length is smaller than the extension length threshold value, determining the defect type of the defect area as extension area overshortage.
Generally, in step S2: before the target image is input into the neural network algorithm, machine learning training is further carried out on the neural network algorithm, so that the trained neural network algorithm can identify a defect area, and defect types can be determined based on the shape and/or the position of the defect area.
But for defects with low occurrence frequency, the sample images are less, such as incomplete defects of welding spots, foreign matter defects of welding spots, wiredrawing defects of welding spots, welding zone fusion welding defects and the like. Therefore, the reliability of the classification depth algorithm trained according to the smaller sample size is lower, so the embodiment also provides a training method for training the neural network based on the small sample, and the training method combines with an expert knowledge base, can improve the reliability of the trained neural network training under the condition that the sample images for training are smaller, and comprises the following steps:
step S21: and acquiring a plurality of sample images, wherein each sample image comprises a welding area after the welding of the chip pins is finished.
Wherein the number of acquired sample images is much smaller than the number of samples required for machine learning in the prior art.
Step S22: and labeling the defect areas in each sample image based on an expert knowledge base.
Step S23: training the neural network algorithm by using the sample image marked with the defect area and determining the defect type to form the neural network algorithm combined with the expert knowledge base.
Because the number of the obtained sample images is small, the traditional neural network algorithm can cause over fitting and poor algorithm performance, so that an expert knowledge base is introduced, the defect areas in the sample images are finely marked through the expert knowledge base, and the types of the defect areas are determined. And training the neural network algorithm by using a sample image marked with the defect area and determining the defect type to form the neural network algorithm combined with the expert knowledge base. The number of training samples in time is smaller, but the expert knowledge base is combined, so that the trained neural network algorithm can improve the recognition rate of some defects with low occurrence frequency.
For example, the training method described in steps S21 to S23 may be used to train the semantic segmentation algorithm in the neural network algorithm, so that a semantic segmentation algorithm combined with the expert knowledge base is formed after training, and the semantic segmentation algorithm combined with the expert knowledge base can identify defects with low occurrence frequency and classify the target image according to the defect types. For example, defects with low occurrence frequency such as incomplete welding spots, foreign matter defects of welding spots, wire drawing defects of welding spots, welding region welding defects and the like can be identified.
Fig. 4 is a block diagram of an apparatus for identifying a die bonding defect according to an embodiment of the present application, and as can be seen from fig. 4, the apparatus includes a processor 410 and a memory 420, where at least one instruction or program is stored in the memory 420, and the instruction or program is loaded and executed by the processor 410 to implement a method for identifying a die bonding defect according to the first aspect of the present application. Wherein processor 410 and memory 420 interact with each other via bus 430.
The present application also provides a computer readable storage medium having stored therein at least one instruction or program loaded and executed by a processor to implement a method of identifying die bonding defects according to the first aspect of the present application.
It is apparent that the above examples are given by way of illustration only and are not limiting of the embodiments. Other variations or modifications of the above teachings will be apparent to those of ordinary skill in the art. It is not necessary here nor is it exhaustive of all embodiments. While still being apparent from variations or modifications that may be made by those skilled in the art are within the scope of the application.
Claims (14)
1. A method of image recognition of a die bonding defect, the method comprising the steps of:
Acquiring a target image, wherein the target image comprises a welding area after the welding of the chip pins is finished;
Inputting the target image into a neural network algorithm;
Determining the shape and/or position of a defect region in the target image based on the neural network algorithm;
determining a defect category based on the defect region shape and/or location;
The neural network algorithm comprises a computer vision algorithm, a target detection algorithm and a semantic segmentation algorithm combined with an expert knowledge base; the computer vision algorithm is used for determining the shape and/or the position of a defect area with skewed pins, the combined expert knowledge base semantic segmentation algorithm is used for determining the shape and/or the position of the defect area with low occurrence frequency, and the target detection algorithm is used for determining the shape and/or the position of the defect area with too short extension area.
2. The method of claim 1, wherein after the step of obtaining a target image including a bonding pad after die pin bonding is completed, the step of inputting the target image into a neural network algorithm is further performed before:
acquiring a target image, and determining that the target image comprises a plurality of welding areas;
Dividing the target image based on an image dividing process to form a plurality of sub-target images;
Such that one of the weld zones is included in each of the sub-target images.
3. The method of claim 1, wherein the step of determining the shape and/or location of the defective area in the target image based on the neural network algorithm comprises:
determining the welding zone based on the neural network algorithm;
determining all defective pixels located at the weld zone locations based on the neural network algorithm; the set of all defective pixels is the defective area;
The shape and/or position of the defective area is determined based on the position of each defective pixel and the commonly constituted shape.
4. The method of identifying die bonding defects according to claim 3, wherein the neural network algorithm comprises at least the computer vision algorithm;
the step of determining the welding area based on the neural network algorithm comprises the following steps:
based on a computer vision algorithm, performing binarization processing on the target image to form a binarized image;
identifying all connected areas of the binarized image;
Determining a maximum communication area in the communication areas;
And determining each corner point of the welding area according to the maximum communication area.
5. The method of identifying die bonding defects according to claim 4, wherein the determining all defective pixels located at the bond pad locations is based on the neural network algorithm; a step of collecting all defective pixels as the defective area, comprising:
Determining pixel points which are transversely beyond the corner points in the maximum communication area as defective pixels based on each corner point of the welding area;
the set of all the defective pixels forms the defective area.
6. The method of claim 1, further comprising, prior to performing the step of inputting the target image into a neural network algorithm:
Acquiring a plurality of sample images, wherein each sample image comprises a welding area after the welding of a chip pin is finished;
Labeling the defect areas in each sample image based on an expert knowledge base, and determining the types of the defect areas;
training the neural network algorithm by using the sample image marked with the defect area and determining the defect type to form the neural network algorithm combined with the expert knowledge base.
7. The method of claim 6, wherein the neural network algorithm combined with expert knowledge base includes at least a semantic segmentation algorithm combined with expert knowledge base.
8. The method of image recognition of die bonding defects according to claim 1, wherein the step of determining the shape and/or location of the defective area at the location of the bonding pad based on the neural network algorithm comprises:
determining the relative position relation between a welding spot area and a chip pin area in the target image based on a neural network algorithm;
Determining the shape of an extension area based on the relative position relationship, wherein the extension area is an area of the chip pin area extending beyond the welding spot area;
And determining whether the extension area is a defect area or not based on the extension length of the extension area shape, wherein the extension length is the length of the extension area in the extending direction of the chip pin area, and the shape and/or the position of the extension area is the shape and/or the position of the defect area.
9. The method of image recognition of die bonding defects according to claim 8, wherein the step of determining the relative positional relationship of the pad area and the die pin area in the target image based on a neural network algorithm comprises:
generating a first detection frame and a second detection frame in the target image based on a neural network algorithm, wherein the first detection frame is used for determining the shape and/or the position of the welding spot area, and the second detection frame is used for determining the shape and/or the position of the chip pin area;
and enabling the relative position relation between the first detection frame and the second detection frame to be used as the relative position relation between the welding spot area and the chip pin area in the target image.
10. The method of image recognition of a die bonding defect according to claim 9, wherein the step of determining an extended area shape, which is an area where the die lead area extends beyond the pad area, based on the relative positional relationship, includes:
Determining an extension detection frame from which the second detection frame extends out of the first detection frame based on a relative positional relationship between the first detection frame and the second detection frame;
the extension detection frame is used for determining the shape and/or the position of the extension area.
11. The method of recognizing a die bonding defect according to claim 10, wherein the step of determining whether the extension region is a defective region based on an extension length of the extension region shape, the extension length being a length of the extension region in an extension direction of the die lead region, the extension region shape and/or position being a shape and/or position of the defective region, comprises:
determining the length of the extension detection frame as the extension length of the extension area shape;
and determining whether the extension area is a defect area based on the length of the extension detection frame.
12. The method of image recognition of a die bonding defect according to claim 8, further comprising, after the step of determining whether the extended area is a defective area, an extended length of the extended area shape:
and when the extension length is smaller than the extension length threshold value, determining that the defect type of the defect area is that the extension area is too short.
13. An apparatus for image recognition of die bonding defects, characterized in that it comprises a processor and a memory in which at least one instruction or program is stored, which is loaded and executed by the processor to implement the method for image recognition of die bonding defects according to any one of claims 1 to 12.
14. A computer-readable storage medium, characterized in that at least one instruction or program is stored in the storage medium, which is loaded and executed by a processor to implement the method of identifying die bonding defects according to any of claims 1 to 12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110992821.8A CN113724218B (en) | 2021-08-27 | 2021-08-27 | Method, device and storage medium for identifying chip welding defect by image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110992821.8A CN113724218B (en) | 2021-08-27 | 2021-08-27 | Method, device and storage medium for identifying chip welding defect by image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113724218A CN113724218A (en) | 2021-11-30 |
| CN113724218B true CN113724218B (en) | 2024-04-30 |
Family
ID=78678470
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110992821.8A Active CN113724218B (en) | 2021-08-27 | 2021-08-27 | Method, device and storage medium for identifying chip welding defect by image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113724218B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114742832B (en) * | 2022-06-13 | 2022-09-09 | 惠州威尔高电子有限公司 | Welding defect detection method for MiniLED thin plate |
| CN115166469B (en) * | 2022-09-07 | 2022-12-27 | 江苏东海半导体股份有限公司 | Method and device for testing Mosfet device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020048119A1 (en) * | 2018-09-04 | 2020-03-12 | Boe Technology Group Co., Ltd. | Method and apparatus for training a convolutional neural network to detect defects |
| JP2020144688A (en) * | 2019-03-07 | 2020-09-10 | 株式会社Ihi | Learning method, learning apparatus, and weld defect detecting apparatus with same learning apparatus |
| CN111862067A (en) * | 2020-07-28 | 2020-10-30 | 中山佳维电子有限公司 | Welding defect detection method and device, electronic equipment and storage medium |
| CN113160204A (en) * | 2021-04-30 | 2021-07-23 | 聚时科技(上海)有限公司 | Semantic segmentation network training method for generating defect area based on target detection information |
-
2021
- 2021-08-27 CN CN202110992821.8A patent/CN113724218B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020048119A1 (en) * | 2018-09-04 | 2020-03-12 | Boe Technology Group Co., Ltd. | Method and apparatus for training a convolutional neural network to detect defects |
| JP2020144688A (en) * | 2019-03-07 | 2020-09-10 | 株式会社Ihi | Learning method, learning apparatus, and weld defect detecting apparatus with same learning apparatus |
| CN111862067A (en) * | 2020-07-28 | 2020-10-30 | 中山佳维电子有限公司 | Welding defect detection method and device, electronic equipment and storage medium |
| CN113160204A (en) * | 2021-04-30 | 2021-07-23 | 聚时科技(上海)有限公司 | Semantic segmentation network training method for generating defect area based on target detection information |
Non-Patent Citations (2)
| Title |
|---|
| 基于机器视觉的SOP芯片引脚缺陷检测研究;李绎铃;;科技资讯(10);全文 * |
| 基于深度学习的电路板焊接异常检测算法研究;秦颖;李鹏;李居尚;;电子器件(02);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113724218A (en) | 2021-11-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113724218B (en) | Method, device and storage medium for identifying chip welding defect by image | |
| US6799130B2 (en) | Inspection method and its apparatus, inspection system | |
| JP3051279B2 (en) | Bump appearance inspection method and bump appearance inspection device | |
| CN111127571B (en) | Small sample defect classification method and device | |
| CN112577969B (en) | Defect detection method and defect detection system based on machine vision | |
| JP5010207B2 (en) | Pattern inspection apparatus and semiconductor inspection system | |
| TWI669519B (en) | Board defect filtering method and device thereof and computer-readabel recording medium | |
| US7634131B2 (en) | Image recognition apparatus and image recognition method, and teaching apparatus and teaching method of the image recognition apparatus | |
| US5642158A (en) | Method and apparatus to detect capillary indentations | |
| CN115147370A (en) | Battery top cover welding defect detection method and device, medium and electronic equipment | |
| CN113030121A (en) | Automatic optical detection method, system and equipment for circuit board components | |
| CN113506243A (en) | PCB welding defect detection method and device and storage medium | |
| CN100538347C (en) | The automatic optical detecting system of membrane type or belt printed circuit board and disposal route | |
| CN113111903A (en) | Intelligent production line monitoring system and monitoring method | |
| JP2000009447A (en) | Apparatus and method for detecting fault of tape carrier | |
| JP5498109B2 (en) | Defect detection apparatus and defect detection method | |
| JP2021022637A (en) | Inspection equipment of bonding wire, inspection method of bonding wire, and inspection program of bonding wire | |
| JP4828741B2 (en) | Probe mark measuring method and probe mark measuring apparatus | |
| JP4453503B2 (en) | Substrate inspection device, substrate inspection method, inspection logic generation device and inspection logic generation method for substrate inspection device | |
| CN115100166A (en) | Welding spot defect detection method and device | |
| JP5672918B2 (en) | Continuous part inspection method, continuous part inspection program, and continuous part inspection apparatus | |
| JP3200953B2 (en) | Inspection method of soldering condition of square chip | |
| JPH08234226A (en) | Image processor for production of liquid crystal substrate | |
| JP2000121495A (en) | Screen inspection method | |
| JP5096940B2 (en) | Inspection method and apparatus for printed wiring board |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |