CN112161997B - Online precise visual measurement method and system for three-dimensional geometric dimension of semiconductor chip pin - Google Patents
Online precise visual measurement method and system for three-dimensional geometric dimension of semiconductor chip pin Download PDFInfo
- Publication number
- CN112161997B CN112161997B CN202011037463.7A CN202011037463A CN112161997B CN 112161997 B CN112161997 B CN 112161997B CN 202011037463 A CN202011037463 A CN 202011037463A CN 112161997 B CN112161997 B CN 112161997B
- Authority
- CN
- China
- Prior art keywords
- chip
- image
- eye image
- point
- cameras
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 239000004065 semiconductor Substances 0.000 title claims abstract description 24
- 238000000691 measurement method Methods 0.000 title claims abstract description 7
- 230000000007 visual effect Effects 0.000 title claims description 6
- 238000000034 method Methods 0.000 claims abstract description 20
- 238000001514 detection method Methods 0.000 claims abstract description 16
- 239000011159 matrix material Substances 0.000 claims description 21
- 238000005259 measurement Methods 0.000 claims description 20
- 238000000605 extraction Methods 0.000 claims description 13
- 238000004422 calculation algorithm Methods 0.000 claims description 8
- 238000004806 packaging method and process Methods 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 7
- 238000004519 manufacturing process Methods 0.000 claims description 6
- 230000003287 optical effect Effects 0.000 claims description 6
- 238000013519 translation Methods 0.000 claims description 4
- 230000005540 biological transmission Effects 0.000 claims description 2
- 238000007781 pre-processing Methods 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 description 9
- 239000013598 vector Substances 0.000 description 7
- 238000012858 packaging process Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 238000003384 imaging method Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000009954 braiding Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
- G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
- G01N21/84—Systems specially adapted for particular applications
- G01N21/88—Investigating the presence of flaws or contamination
- G01N21/95—Investigating the presence of flaws or contamination characterised by the material or shape of the object to be examined
- G01N21/956—Inspecting patterns on the surface of objects
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/002—Measuring arrangements characterised by the use of optical techniques for measuring two or more coordinates
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/02—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/02—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness
- G01B11/06—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness for measuring thickness ; e.g. of sheet material
- G01B11/0608—Height gauges
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/14—Measuring arrangements characterised by the use of optical techniques for measuring distance or clearance between spaced objects or spaced apertures
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/26—Measuring arrangements characterised by the use of optical techniques for measuring angles or tapers; for testing the alignment of axes
- G01B11/27—Measuring arrangements characterised by the use of optical techniques for measuring angles or tapers; for testing the alignment of axes for testing the alignment of axes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20228—Disparity calculation for image-based rendering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30148—Semiconductor; IC; Wafer
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30204—Marker
- G06T2207/30208—Marker matrix
Abstract
The invention discloses an on-line precise vision measurement method and a system for three-dimensional geometric dimensions of a pin of a semiconductor chip, wherein the method comprises the following steps: acquiring a semiconductor chip image in the braid in a fixed area; extracting and analyzing the features of the image shot in the stereo vision measuring system, and extracting two key corner features on the outer side of each pin on the chip and the feature points on the surface of the chip; performing feature matching on the pin feature points of the image and the surface feature points of the chip plastic package body to realize three-dimensional reconstruction of the pin feature points and the surface feature points of the plastic package body in a camera coordinate system; whether the chip pins are qualified or not can be detected by calculating the three-dimensional geometric dimension between the corner points of each pin and the three-dimensional characteristic points of the surface of the plastic package. The method of the invention not only can calculate the width distance of the chip pins, the spacing distance between adjacent pins and the coplanarity of the pins, but also can realize the detection of whether the pins warp and other common faults.
Description
Technical Field
The invention belongs to the technical field of chip packaging detection, and particularly relates to an on-line precise vision measurement method and system for three-dimensional geometric dimensions of a pin of a semiconductor chip based on a binocular stereo vision technology.
Background
The semiconductor chip manufacturing process includes the steps of distributing the chip and other elements on a frame or a substrate by using a film technology and a fine processing technology, pasting, fixing and connecting, leading out a wiring terminal, and embedding and fixing the wiring terminal through a plastic insulating medium to form an integral three-dimensional structure. With the rapid development of semiconductor chip detection technology, the current market has higher and higher requirements on the quality of semiconductor chip products.
The application and popularization of the computer vision technology provide an idea for the current chip packaging and quality intelligent detection technology. The measurement of the critical geometric dimensions of the semiconductor chip pins, including the critical dimensions such as the width distance of the pins, the spacing distance between adjacent pins, and the height of the pins (whether there is a failure such as tilting or collapsing), is a problem that needs to be solved urgently in the current chip packaging process.
In a common solution, key feature points of a pin are usually extracted, pixel coordinate point information of the feature points is obtained, and then a conversion ratio of pixels is used to obtain an actual geometric size. This method requires accurate knowledge of the actual physical distance represented by each pixel of the camera, which has a good effect on solving between planar objects in the field of view of the camera. In the actual packaging process of the chip, the chip in the braid always has some inclination, so that the solution of the method has certain errors. In addition, the method cannot measure three-dimensional information of the depth direction of the target in the field of view of the camera, which is an important parameter index in the chip packaging process.
Disclosure of Invention
The technical purpose is as follows: aiming at the defects of the prior art, the method and the system for online and high-precision visual measurement of the key three-dimensional geometric dimension of the pin of the semiconductor chip are provided, and the online detection of the quality of the chip product is realized by detecting the three-dimensional information such as the width, the height, the coplanarity and the like of the pin of the chip in the chip packaging process in real time.
The technical scheme is as follows: in order to achieve the technical purpose, the invention adopts the following technical scheme:
an on-line precision vision measuring method for the three-dimensional geometric dimension of a pin of a semiconductor chip is characterized in that the chip is provided with a plastic package surface and a chip pin, and comprises the following steps:
s1, initial image acquisition: shooting the same chip transmitted to the measuring position by two cameras at the same time to respectively obtain a left-eye initial image and a right-eye initial image; the two cameras can shoot a complete image of the chip positioned on the measuring position;
s2, feature point extraction: respectively preprocessing the left-eye initial image and the right-eye initial image, and capturing the interested regions of the images to obtain a corresponding left-eye image and a corresponding right-eye image; extracting coordinate information of chip pins and plastic package surface characteristic points from the left eye image and the right eye image, wherein the plastic package surface characteristic points comprise characters;
s3, feature point matching and three-dimensional coordinate obtaining: acquiring two coordinates of the same characteristic point P on a left-eye image and a right-eye image at the same moment, solving a parallax relation between the two coordinates, and solving a three-dimensional coordinate of the characteristic point P in a camera coordinate system according to the coordinates and the parallax relation between the coordinates; and calculating the width of the chip pins and the distance between the adjacent pins by calculating the three-dimensional geometric dimension between corner points of each pin.
Preferably, the method further comprises the steps of:
s4, judging whether the pins of the chip are qualified: and performing plane fitting on the three-dimensional characteristic points on the surface of the plastic package, respectively solving the three-dimensional geometric distance from the corner point of the pin of the chip to the reference plane by taking the three-dimensional plane of the plastic package surface as the reference plane, judging whether the calculated three-dimensional geometric distance is within a preset range, and further judging whether the pin of the chip is qualified.
Preferably, the two cameras are disposed at the same vertical position, and the step S3 includes:
s3.1, obtaining the coordinates of the same characteristic point P on the left-eye image and the right-eye image, wherein the coordinates are respectively as follows:
P left ＝(X left ，Y left )，P right ＝(X right ，Y right )；
s3.2, when the images of the two cameras are on the same plane, the Y coordinates of the image coordinates of the feature points P are the same, and the geometric relationship is obtained as follows:
wherein, B is the base line distance, namely the distance of the connecting line of the projection centers of the two cameras;
s3.3, calculating the three-dimensional coordinate of the feature point P in the camera coordinate system as follows:
wherein f is the focal length of the camera, and the Disparity is X left -X right (ii) a Disparity represents parallax.
Preferably, the step S4 includes:
s4.1, in the binocular stereo vision system, the space point to be measured is marked as a point P, and the optical centers of the two cameras are respectively marked as C l And C r Connecting line of optical centers of two cameras
The intersection point of the baseline and the left eye image and the right eye image is a pole of the image, and the connecting line of the pole in the left eye image and the projection point corresponds to the polar line of the projection point in the right eye image and is recorded as the polar line of the projection point in the right eye image
The line connecting the pole in the right eye image and the projection point corresponds to the polar line of the projection point in the left eye image and is recorded as
S4.2, in the binocular stereo vision system, the projection point P of any space point P in the left eye image and the right eye image l And p r The following relationship is satisfied: left eye image projection point p l Corresponding point p on the right eye image r Polar lines in the right-eye image plane
Upper, right eye image projection point p r Corresponding point p on the left eye image l Polar line determined in left eye image plane
The above step (1);
according to the relation, the internal and external parameters between cameras in the stereoscopic vision measuring system are obtained;
s4.3, according to the perspective projection relation of the camera, the equation of the left and right epipolar lines in the corresponding image can be obtained:
wherein F is a basis matrix, A l And A r The three-dimensional coordinate system is characterized by comprising internal parameter matrixes of a left camera and a right camera respectively, R is a rotation matrix in external parameters of two camera coordinate systems, and S is an antisymmetric matrix of a translation matrix T in the external parameters of the two camera coordinate systems;
and S4.4, fitting the plastic package surface equation by using a least square method to obtain a chip plastic package surface equation under a camera coordinate system.
Preferably, in step S2, the positions of the corner points of the chip pins in the image are determined by using a corner point detection method based on a gradient correlation matrix.
Preferably, SURF algorithm is adopted to extract the characteristic points of the plastic package surface of the chip.
Preferably, if different types of chips are loaded in the braid, the steps S1-S3 are sequentially executed to complete the detection of each chip pin; and if the same type of chip is loaded in the braid, determining the region of interest of the image by using the prior information.
An on-line precision vision measurement system for three-dimensional geometric dimensions of a semiconductor chip pin is characterized in that: the measuring system comprises a measuring position, two cameras and an image processing module, wherein the measuring system is built on a chip packaging production line, the braid for carrying the chip to be detected is arranged on the chip packaging production line, the measuring system comprises the measuring position, the two cameras and the image processing module, when the braid is moved to the measuring position one chip at a time during transmission, the two cameras can both shoot the image of the chip positioned on the measuring position, and the image processing module is used for executing the method.
Preferably, the cameras arranged in the measuring system comprise a left camera and a right camera, and each camera is provided with a matched telephoto lens.
Has the advantages that: due to the adoption of the technical scheme, the invention has the following technical effects:
the invention discloses a binocular stereoscopic vision technology-based semiconductor chip pin key three-dimensional size precise online measurement method. The method of the invention not only can calculate the width distance of the chip pins, the spacing distance between adjacent pins and the coplanarity of the pins, but also can realize the detection of whether the pins warp and other common faults.
Drawings
FIG. 1 is a flow chart of the method for on-line precise visual measurement of the three-dimensional geometric dimension of a semiconductor chip pin according to the present invention;
FIG. 2 is a schematic structural diagram of a measurement system to which the three methods of the present invention are applied;
FIG. 3 is a schematic diagram of a head-up binocular stereo imaging;
FIG. 4 is a polar line geometric constraint relationship of a binocular stereo vision measurement system;
fig. 5 is a schematic diagram of a plastic package surface plane equation, plastic package surface character feature points and three-dimensional position structures of pin corner points under a camera coordinate system.
Detailed Description
The invention utilizes the stereoscopic vision measurement technology and combines the characteristics of the chip packaging process to research an on-line measurement method capable of realizing the three-dimensional size of the key part of the chip pin, and completes the on-line measurement of the key geometric size and the on-line detection of the chip packaging quality in the chip packaging process. The method comprises the following steps: firstly, extracting and analyzing the features of an image shot in a stereo vision measuring system, and extracting two key corner features on the outer side of each pin on a chip and the feature points on the surface of the chip. And then, performing feature matching on the pin feature points of the two images in the measuring system and the surface feature points of the chip plastic package body by using the calibrated binocular stereo vision measuring system to realize three-dimensional reconstruction of the pin feature points and the surface feature points of the plastic package body in a camera coordinate system. The width of the chip pins and the distance between the adjacent pins can be realized by calculating the three-dimensional geometric dimension between the corner points of each pin. Meanwhile, the three-dimensional characteristic points on the surface of the plastic package are subjected to plane fitting, so that the three-dimensional distance between each pin and the surface of the plastic package can be realized, and further, whether the chip pin has faults such as pin warping or not can be detected.
Fig. 2 shows a schematic structural diagram of a measuring system, which includes a left camera 1 and a right camera 2, each camera is provided with a matched telephoto lens 3, a braid 4 carrying a semiconductor chip 5 to be detected is carried by the camera, and an arrow 7 shows a moving direction of the braid. The flow chart of the invention is shown in fig. 1, and comprises the following steps:
(1) a binocular stereoscopic vision measuring system is built for a chip packaging production line, so that when a braid carrying a chip device passes through a precision mechanical conveying device, the position of one chip is moved at each time, and the purpose is to enable two CCD vision sensors in the measuring system to respectively obtain a chip clear image at a fixed position of a mechanical device when the two CCD vision sensors shoot at each time.
(2) The braid chip passes through a precision mechanical conveying device, and the position and the direction of the chip in the braid are basically fixed. The precise mechanical conveying device moves for one braiding chip interval every time, the visual measurement sensor acquires a chip image of a fixed area every time, and chip pins are basically located at the image fixing positions. Therefore, aiming at the chips with the same type carried by the braid, the region of interest (ROI) of the image can be determined by utilizing the prior information, and the chip pin corner points and the plastic package surface feature points are quickly extracted in the ROI of the image.
(3) Binocular stereoscopic vision measurement principle:
the binocular stereo vision three-dimensional measurement is based on the parallax principle, as shown in fig. 3, a simple head-up binocular stereo imaging principle diagram is shown, 9 and 10 are image planes of a left camera and a right camera respectively, 8 is a target point P to be measured, 11 and 12 are feature points of the target point on the left image plane and the right image plane respectively, and 13 is the distance between the projection centers of the two cameras, namely, the base line distance is B.
Two cameras view the same characteristic point P of a space object at the same time, and images of the point P are acquired on the left eye and the right eye respectively, and the image coordinates of the point P are as follows:
P left ＝(X left ，Y left )，P right ＝(X right ，Y right )
assuming that the images of the two cameras are on the same plane, the Y coordinates of the image coordinates of the feature points P are the same, i.e., Y right ＝Y right Let Y be the focal length of the camera, and f, then we get from the geometric relationship:
then the parallax is Disparity X left -X right . Therefore, the three-dimensional coordinates of the feature point P in the camera coordinate system can be calculated as follows:
therefore, any point on the left camera image plane can determine the three-dimensional coordinates of the point as long as the corresponding matching point (the two points are the points of the same point in space on the left and right camera image planes) can be found on the right camera image plane. The method is a point-to-point operation, and all points on an image plane can participate in the operation as long as corresponding matching points exist, so that corresponding three-dimensional coordinates of the points are obtained.
(4) Fast extraction of chip characteristic points:
according to the binocular stereo vision measurement principle, if three-dimensional of the related information of the pins of the semiconductor chip is realized, the extraction of the image characteristic points is very critical. According to the measurement characteristics of the pins of the semiconductor chip, the invention analyzes feature point extraction from two aspects:
extracting angular points of pins of the A chip: due to the action of the braid and external light, edges and corners at two ends of a chip pin angular point sometimes present an elliptical state, so that the pin angular point characteristics are not obvious, and a chip pin target point cannot be accurately extracted by using a traditional angular point extraction algorithm, so that the three-dimensional reconstruction precision of a subsequent target point is influenced.
The invention adopts an angular point detection method (Zhang X, Wang H, Smith AW B, et al. corner detection based on gradient correlation matrix of planar currents [ J ]. Pattern Recognition, 2010, 43 (4): 1207-.
Firstly, extracting the image edge of the ROI by using a Canny edge extraction algorithm, and then smoothing the edge contour by using a Gaussian function, so that the algorithm has stronger robustness to image noise. Because the pin and the braid background color are obviously contrasted, the pin edge is easily extracted.
Next, assume that the above-mentioned extracted one-pin plane curve equation is
C(t)＝(x(t)，y(t)) (3)
Where t is the curve equation parameter, x (t) and y (t) are coordinate functions.
From equation (3), the gradient vector of any point on the curve can be found:
C ′(t) ＝(x ′(t) ，y′(t))＝(dx，dy) (4)
where dx, dy are the gradients of the curve c (t) in the x and y directions, respectively.
For a curve centered at any P point on the curveLine segment, gradient vector around this point noted as (dx) i ，dy i ). To point (dx) i ，dy i ) The perpendicular distance between the point and the characteristic line, i.e. the projection of the point on the unit normal vector n of the characteristic line, is represented by d i (n) represents the distance, i.e.
When the sum of squared distances of all gradient vector distribution points satisfies the minimum, the unit normal vector n satisfies the following minimum:
wherein k is the serial number of the kth edge point in the support area in the curve segment,
is the autocorrelation matrix of the gradient. The eigenvector corresponding to the minimum eigenvalue of the M matrix is the most matched gradient unit normalized vector, and the minimum eigenvalue reflects the distribution of the characteristics, namely the dispersion degree of the vectors in the gradient coordinates. The discrete form of the M matrix may be defined as:
wherein (Δ x) ki ，Δy ki ) W is the radius of the support region for the ith order difference in the support region.
The response degree of the corner points can be determined by analyzing the determinant of the M matrix, and the points with local maximum values larger than a preset threshold value in the response degree are screened out to be the solved corner points. According to the characteristics of the chip pins, the scale sigma of the Gaussian function is set to be 2.0, the radius W of the contour support area is set to be 1, the preset threshold value is [0.004, 0.008], and the pin corner effect obtained by using the parameters is good.
The algorithm has moderate calculation complexity, high detection speed and good detection effect, and is suitable for extracting the pin corner points of the chip.
B, extracting characteristic points of the plastic package surface of the chip:
the printed characters on the surface of the chip plastic package have strong angular point information, and the character characteristic points on the surface of the plastic package can be subjected to three-dimensional reconstruction by using the extracted character characteristic points on the surface of the plastic package, so that the three-dimensional characteristic information on the surface of the plastic package is fitted. And (3) taking the three-dimensional plane of the plastic package surface as a reference surface, and respectively calculating the three-dimensional geometrical distance from the three-dimensional point information of the corner point of the chip pin to the plane by using a computer binocular stereo vision measurement principle, so that whether the chip pin has foot collapse and foot tilting faults, and the measurement of key geometrical information such as coplanarity of each pin can be judged.
Characters on the surface of each plastic package are clear and obvious, and feature information is convenient to extract. In consideration of the characteristic extraction efficiency, the invention adopts SURF algorithm (H.Bay, T.Tuytelaars, L.V.Gool, SURF: Speed Up Robust Features, ECCV, 2006) to extract the character characteristic points on the surface of the plastic package, has stable characteristic extraction, stronger robustness on information such as rotation and scale, and fast extraction Speed, and is suitable for extracting the character characteristic on the surface of the plastic package.
(5) Feature point matching and three-dimensional reconstruction:
in the binocular stereo vision system, as shown in fig. 4, it is assumed that 14 is the spatial point P to be measured, 19 and 20 represent the optical centers C of the left and right cameras, respectively l And C r Connecting line of optical centers of two cameras
As a baseline, the intersections 17 and 18 of the baseline with the two image planes are called poles of the image, and the line 15 connecting the pole 17 in the left eye image with the projection point 21 is called the epipolar line corresponding to the projection point 22 in the right eye image (denoted as epipolar line)
) (ii) a Similarly, the line 16 connecting the pole 18 and the projection point 22 in the right eye image is called the epipolar line corresponding to the projection point 21 in the left eye image (denoted as epipolar line)
). That is, in the binocular stereoscopic vision system, the projection point P of an arbitrary spatial point P in the left and right images l And p r The following relationship must be satisfied: left eye image projection point p l Corresponding point p on the right eye image r Polar lines in the right-eye image plane
Upper, right eye image projection point p r Corresponding point p on the left eye image l Polar line determined in left eye image plane
The above. This relationship is known as the epipolar constraint between image planes in a stereoscopic system. After the internal and external parameters between cameras in the stereo vision measuring system are obtained, the equation of the left and right epipolar lines in the corresponding images can be obtained according to the perspective projection relation of the cameras:
wherein F is a basis matrix, A l And A r The three-dimensional coordinate system is characterized by comprising internal parameter matrixes of a left camera and a right camera respectively, R is a rotation matrix in external parameters of two camera coordinate systems, and S is an antisymmetric matrix of a translation matrix T in the external parameters of the two camera coordinate systems.
The three-dimensional reconstruction of the image characteristic points in the binocular stereo vision measurement system is established on the basis of the parallax of the corresponding points, so that the matching of the corresponding edge points in the image is a key link for realizing the thickness measurement of the skateboard. And epipolar constraint gives an important relation of corresponding point matching search, and the search of corresponding matching points is compressed to a straight line from the whole image, so that the search range of target points is greatly reduced.
The extracted pin corner points and the plastic package surface characteristic points are registered by polar line constraint on the characteristic points on the two images shot by the binocular stereoscopic vision camera, so that the matching of target points can be accurately realized. The epipolar geometric constraint has certain significance on the rapid matching of the pin feature points of the semiconductor image, and is a key technology for realizing the three-dimensional reconstruction of the pin feature stereoscopic vision of the semiconductor chip. And the matched image characteristic point pairs are subjected to three-dimensional extraction of target characteristic points under a camera coordinate system by using a binocular vision measurement principle, so that the precise measurement of the key geometric dimension of the chip pin can be realized.
Examples
The vision measuring camera in FIG. 2 is Basler acA2500 camera, whose lens is RICOHFL-CC2518-5MX, and the left camera internal reference calibration result
Calibration result of internal parameters of right camera
Rotation matrix between left and right cameras
Translation matrix T ═ 95.5963-1.4835-142.8034]The extraction results of the corner points of the semiconductor chip pins (4 pins on the left and right, 2 corner points on each pin) in the left and right cameras are as follows:
TABLE 1
The extraction results of the 4 character SURF characteristic points corresponding to the plastic package surface of the semiconductor chip in the left camera and the right camera are respectively as follows:
TABLE 2
The algorithm and the process provided by the invention can be used for respectively calculating the three-dimensional coordinates of each pin characteristic point and the plastic package surface characteristic point under a camera coordinate system, and the three-dimensional coordinates are respectively as follows:
TABLE 3
TABLE 4
Fitting a plastic package surface equation by using a least square method to obtain the plastic package surface equation under a camera coordinate system as follows: -0.0598X-0.1122Y +0.9919Z-471.3773 ═ 0.
As shown in fig. 5, 23 is a plastic package surface plane equation under a camera coordinate system, 24 is three-dimensional coordinates of character feature points of the plastic package surface, and 25 is a three-dimensional position of a pin corner point.
And then, the vertical distance from the pin corner points to the plastic package surface can be realized by utilizing the obtained three-dimensional distance from the three-dimensional coordinates of the pin corner points to the plastic package surface, and the detection of whether the pin corner points of the chip have faults such as pin warping, pin collapsing and the like is finished.
The above description is only of the preferred embodiments of the present invention, and it should be noted that: it will be apparent to those skilled in the art that various modifications and adaptations can be made without departing from the principles of the invention and these are intended to be within the scope of the invention.
Claims (4)
1. An on-line precision vision measuring method for the three-dimensional geometric dimension of a pin of a semiconductor chip is characterized in that the chip is provided with a plastic package surface and a chip pin, and comprises the following steps:
s1, initial image acquisition: shooting the same chip transmitted to the measuring position by two cameras at the same time to respectively obtain a left-eye initial image and a right-eye initial image; the two cameras can shoot a complete image of the chip positioned on the measuring position;
s2, feature point extraction: respectively preprocessing the left-eye initial image and the right-eye initial image, and capturing the interested regions of the images to obtain a corresponding left-eye image and a corresponding right-eye image; extracting coordinate information of chip pins and plastic package surface characteristic points from the left eye image and the right eye image, wherein the plastic package surface characteristic points comprise characters; determining the positions of corner points of chip pins in the image by adopting a corner point detection method based on a gradient correlation matrix; extracting characteristic points of the plastic package surface of the chip by adopting an SURF algorithm;
s3, feature point matching and three-dimensional coordinate obtaining: acquiring two coordinates of the same characteristic point P on a left-eye image and a right-eye image at the same moment, obtaining a parallax relation between the two coordinates, and obtaining a three-dimensional coordinate of the characteristic point P in a camera coordinate system according to the coordinates and the parallax relation between the coordinates; calculating the width of the chip pins and the distance between the adjacent pins by calculating the three-dimensional geometric dimension between the corner points of each pin;
the two cameras are arranged at the same vertical position, and the step S3 includes:
s3.1, obtaining the coordinates of the same feature point P on the left-eye image and the right-eye image, wherein the coordinates are as follows:
P left ＝(X left ,Y left ),P right ＝(X right ,Y right )；
s3.2, when the images of the two cameras are on the same plane, the Y coordinates of the image coordinates of the feature points P are the same, and the Y coordinates are obtained through a geometrical relation:
wherein f is the focal length of the cameras, and B is the base line distance, namely the distance between the connecting lines of the projection centers of the two cameras;
s3.3, calculating the three-dimensional coordinates of the feature point P in the camera coordinate system as follows:
wherein, Disparity ═ X left -X right (ii) a Disparity represents parallax;
s4, judging whether the pins of the chip are qualified: performing plane fitting on the three-dimensional characteristic points on the surface of the plastic package, respectively solving the three-dimensional geometric distances from the corner points of the chip pins to the reference plane by taking the three-dimensional plane of the plastic package surface as the reference plane, and judging whether the calculated three-dimensional geometric distances are within a preset range so as to judge whether the pins of the chip are qualified;
the step S4 includes:
s4.1, in the binocular stereo vision system, recording the space point to be measured as a point P, and recording the optical centers of the two cameras as C l And C r Connecting line of optical centers of two cameras
Taking the baseline as a baseline, wherein the intersection point of the baseline and the left-eye image and the right-eye image is a pole of the image, and the connecting line of the pole in the left-eye image and the projection point corresponds to the polar line of the projection point in the right-eye image and is marked as l pl (ii) a The line connecting the pole and the projection point in the right eye image corresponds to the polar line of the projection point in the left eye image and is marked as l pr ；
S4.2, in the binocular stereo vision system, the projection point P of any space point P in the left eye image and the right eye image l And p r The following relationship is satisfied: left eye image projection point p l Corresponding point p on the right eye image r Polar line l in the right-eye image plane pl Upper, right eye image projection point p r Corresponding point p on the left eye image l Polar line l which must be in the plane of the left eye image pr The above step (1);
according to the relation, the internal and external parameters between cameras in the stereoscopic vision measuring system are obtained;
s4.3, solving the equation of the left and right epipolar lines in the corresponding image according to the perspective projection relation of the camera:
wherein F is a basis matrix, A l And A r The three-dimensional coordinate system is characterized by comprising internal parameter matrixes of a left camera and a right camera respectively, R is a rotation matrix in external parameters of two camera coordinate systems, and S is an antisymmetric matrix of a translation matrix T in the external parameters of the two camera coordinate systems;
and S4.4, fitting the plastic package surface equation by using a least square method to obtain a chip plastic package surface equation under a camera coordinate system.
2. The on-line precise visual measurement method for the three-dimensional geometric dimension of the pin of the semiconductor chip according to claim 1, characterized in that: if the braid bears different types of chips, sequentially executing steps S1-S3 to complete the detection of the pins of each chip; and if the same type of chip is loaded in the braid, determining the region of interest of the image by using the prior information.
3. An online precision vision measurement system for three-dimensional geometric dimensions of a semiconductor chip pin is characterized in that: the measuring system comprises a measuring position, two cameras and an image processing module, wherein the measuring system is built on a chip packaging production line, a braid for carrying a chip to be detected is arranged on the chip packaging production line, the measuring system comprises the measuring position, the two cameras and the image processing module, when the braid is moved to the measuring position one chip at a time during transmission, the two cameras can both shoot images of the chip positioned on the measuring position, and the image processing module is used for executing the method of any one of claims 1-2.
4. The system of claim 3, wherein the system comprises: the cameras arranged in the measuring system comprise a left camera and a right camera, and each camera is provided with a matched telephoto lens.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011037463.7A CN112161997B (en) | 2020-09-28 | 2020-09-28 | Online precise visual measurement method and system for three-dimensional geometric dimension of semiconductor chip pin |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011037463.7A CN112161997B (en) | 2020-09-28 | 2020-09-28 | Online precise visual measurement method and system for three-dimensional geometric dimension of semiconductor chip pin |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112161997A CN112161997A (en) | 2021-01-01 |
| CN112161997B true CN112161997B (en) | 2022-09-27 |
Family
ID=73861754
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011037463.7A Active CN112161997B (en) | 2020-09-28 | 2020-09-28 | Online precise visual measurement method and system for three-dimensional geometric dimension of semiconductor chip pin |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112161997B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114445335B (en) * | 2021-12-22 | 2024-04-12 | 武汉易思达科技有限公司 | Vehicle running state monitoring method based on binocular machine vision |
| CN114152200A (en) * | 2022-02-07 | 2022-03-08 | 北京航空航天大学杭州创新研究院 | Chip pin measuring method and device |
| CN114252000B (en) * | 2022-03-01 | 2022-05-27 | 北京京瀚禹电子工程技术有限公司 | Device and method for testing coplanarity and spacing of chip pins |
| CN114577135B (en) * | 2022-03-01 | 2024-01-19 | 合肥图迅电子科技有限公司 | 3D detection method and system for chip pin warpage based on single lens |
| CN115266743B (en) * | 2022-05-17 | 2024-02-02 | 江苏汤谷智能科技有限公司 | System and method for evaluating chip quality under nondestructive testing |
| CN115248017B (en) * | 2022-09-21 | 2022-11-29 | 江苏新恒基特种装备股份有限公司 | Rapid detection method and detection system for geometric dimension of branch pipe fitting |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102032878B (en) * | 2009-09-24 | 2014-06-18 | 黑龙江省科学院自动化研究所 | Accurate on-line measurement method based on binocular stereo vision measurement system |
| CN103247053B (en) * | 2013-05-16 | 2015-10-14 | 大连理工大学 | Based on the part accurate positioning method of binocular microscopy stereo vision |
| CN104574367B (en) * | 2014-12-21 | 2017-04-12 | 浙江大学 | Method for detecting and calculating geometric size of integrated chip pin based on machine vision |
| CN106767399B (en) * | 2016-11-11 | 2018-11-09 | 大连理工大学 | The non-contact measurement method of logistics goods volume based on binocular stereo vision and dot laser ranging |
| CN107907048A (en) * | 2017-06-30 | 2018-04-13 | 长沙湘计海盾科技有限公司 | A kind of binocular stereo vision method for three-dimensional measurement based on line-structured light scanning |
| CN107588721A (en) * | 2017-08-28 | 2018-01-16 | 武汉科技大学 | The measuring method and system of a kind of more sizes of part based on binocular vision |
| CN107884767A (en) * | 2017-10-31 | 2018-04-06 | 暨南大学 | A kind of method of binocular vision system measurement ship distance and height |
| CN110009690A (en) * | 2019-03-23 | 2019-07-12 | 西安电子科技大学 | Binocular stereo vision image measuring method based on polar curve correction |
| CN114255286B (en) * | 2022-02-28 | 2022-05-13 | 常州罗博斯特机器人有限公司 | Target size measuring method based on multi-view binocular vision perception |
-
2020
- 2020-09-28 CN CN202011037463.7A patent/CN112161997B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN112161997A (en) | 2021-01-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112161997B (en) | Online precise visual measurement method and system for three-dimensional geometric dimension of semiconductor chip pin | |
| US8121352B2 (en) | Fast three dimensional recovery method and apparatus | |
| Kim et al. | A camera calibration method using concentric circles for vision applications | |
| JP7037876B2 (en) | Use of 3D vision in automated industrial inspection | |
| CN109859272A (en) | A kind of auto-focusing binocular camera scaling method and device | |
| CN104713885A (en) | Structured light-assisted binocular measuring method for on-line detection of PCB | |
| KR20120058828A (en) | System for extracting 3-dimensional coordinate and method thereof | |
| CN109345597B (en) | Camera calibration image acquisition method and device based on augmented reality | |
| CN110310331B (en) | Pose estimation method based on combination of linear features and point cloud features | |
| CN109727290A (en) | Zoom camera dynamic calibrating method based on monocular vision triangle telemetry | |
| CN106996748A (en) | A kind of wheel footpath measuring method based on binocular vision | |
| CN109636857B (en) | Alignment method and calibration system | |
| CN112489193A (en) | Three-dimensional reconstruction method based on structured light | |
| CN115359127A (en) | Polarization camera array calibration method suitable for multilayer medium environment | |
| CN116188558A (en) | Stereo photogrammetry method based on binocular vision | |
| CN112802114A (en) | Multi-vision sensor fusion device and method and electronic equipment | |
| CN112595262B (en) | Binocular structured light-based high-light-reflection surface workpiece depth image acquisition method | |
| JP2008275366A (en) | Stereoscopic 3-d measurement system | |
| Kochi et al. | 3D modeling of architecture by edge-matching and integrating the point clouds of laser scanner and those of digital camera | |
| CN108537831B (en) | Method and device for performing CT imaging on additive manufacturing workpiece | |
| CN116205993A (en) | Double-telecentric lens high-precision calibration method for 3D AOI | |
| CN206583440U (en) | A kind of projected image sighting distance detecting system | |
| CN111998834B (en) | Crack monitoring method and system | |
| CN114963981A (en) | Monocular vision-based cylindrical part butt joint non-contact measurement method | |
| CN115082538A (en) | System and method for three-dimensional reconstruction of surface of multi-view vision balance ring part based on line structure light projection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20240102 Address after: No. 22-3, No. 988 Gangcheng Avenue, Shengang Street, Jiangyin City, Wuxi City, Jiangsu Province, 214443 Patentee after: Jiangsu Kangtan Technology Co.,Ltd. Address before: 1 No. 211167 Jiangsu city of Nanjing province Jiangning Science Park Hongjing Road Patentee before: NANJING INSTITUTE OF TECHNOLOGY |