CN112161997A - Online precise visual measurement method and system for three-dimensional geometric dimension of semiconductor chip pin - Google Patents

Online precise visual measurement method and system for three-dimensional geometric dimension of semiconductor chip pin Download PDF

Info

Publication number
CN112161997A
CN112161997A CN202011037463.7A CN202011037463A CN112161997A CN 112161997 A CN112161997 A CN 112161997A CN 202011037463 A CN202011037463 A CN 202011037463A CN 112161997 A CN112161997 A CN 112161997A
Authority
CN
China
Prior art keywords
image
chip
point
pin
dimensional
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011037463.7A
Other languages
Chinese (zh)
Other versions
CN112161997B (en
Inventor
路绳方
张健
何睿清
焦良葆
陈烨
高阳
曹雪虹
史金飞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jiangsu Kangtan Technology Co.,Ltd.
Original Assignee
Nanjing Institute of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing Institute of Technology filed Critical Nanjing Institute of Technology
Priority to CN202011037463.7A priority Critical patent/CN112161997B/en
Publication of CN112161997A publication Critical patent/CN112161997A/en
Application granted granted Critical
Publication of CN112161997B publication Critical patent/CN112161997B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/95—Investigating the presence of flaws or contamination characterised by the material or shape of the object to be examined
    • G01N21/956—Inspecting patterns on the surface of objects
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/002—Measuring arrangements characterised by the use of optical techniques for measuring two or more coordinates
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/02—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/02—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness
    • G01B11/06—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness for measuring thickness ; e.g. of sheet material
    • G01B11/0608—Height gauges
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/14—Measuring arrangements characterised by the use of optical techniques for measuring distance or clearance between spaced objects or spaced apertures
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/26—Measuring arrangements characterised by the use of optical techniques for measuring angles or tapers; for testing the alignment of axes
    • G01B11/27—Measuring arrangements characterised by the use of optical techniques for measuring angles or tapers; for testing the alignment of axes for testing the alignment of axes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20228—Disparity calculation for image-based rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30148—Semiconductor; IC; Wafer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30204—Marker
    • G06T2207/30208—Marker matrix

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Geometry (AREA)
  • Health & Medical Sciences (AREA)
  • Quality & Reliability (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Biochemistry (AREA)
  • General Health & Medical Sciences (AREA)
  • Immunology (AREA)
  • Pathology (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

The invention discloses an on-line precise visual measurement method and a system for the three-dimensional geometric dimension of a pin of a semiconductor chip, wherein the method comprises the following steps: acquiring a semiconductor chip image in the braid in a fixed area; extracting and analyzing the features of the image shot in the stereo vision measuring system, and extracting two key corner features on the outer side of each pin on the chip and the feature points on the surface of the chip; performing feature matching on the pin feature points of the image and the surface feature points of the chip plastic package body to realize three-dimensional reconstruction of the pin feature points and the surface feature points of the plastic package body in a camera coordinate system; whether the chip pins are qualified or not can be detected by calculating the three-dimensional geometric dimension between the corner points of each pin and the three-dimensional characteristic points of the surface of the plastic package. The method of the invention not only can calculate the width distance of the chip pins, the spacing distance between adjacent pins and the coplanarity of the pins, but also can realize the detection of whether the pins warp and other common faults.

Description

Online precise visual measurement method and system for three-dimensional geometric dimension of semiconductor chip pin
Technical Field
The invention belongs to the technical field of chip packaging detection, and particularly relates to an on-line precise vision measurement method and system for three-dimensional geometric dimensions of a pin of a semiconductor chip based on a binocular stereo vision technology.
Background
The semiconductor chip manufacturing process includes the steps of distributing the chip and other elements on a frame or a substrate by using a film technology and a fine processing technology, pasting, fixing and connecting, leading out a wiring terminal, and embedding and fixing the wiring terminal through a plastic insulating medium to form an integral three-dimensional structure. With the rapid development of semiconductor chip detection technology, the current market has higher and higher requirements on the quality of semiconductor chip products.
The application and popularization of the computer vision technology provide an idea for the current chip packaging and quality intelligent detection technology. The measurement of the critical geometric dimensions of the semiconductor chip pins, including the critical dimensions such as the width distance of the pins, the spacing distance between adjacent pins, and the height of the pins (whether there is a failure such as tilting or collapsing), is a problem that needs to be solved urgently in the current chip packaging process.
In a common solution, key feature points of a pin are usually extracted, pixel coordinate point information of the feature points is obtained, and then a conversion ratio of pixels is used to obtain an actual geometric size. This method requires accurate knowledge of the actual physical distance represented by each pixel of the camera, which has a good effect on solving between planar objects in the field of view of the camera. In the actual packaging process of the chip, the chip in the braid always has some inclination, so that the solution of the method has certain errors. In addition, the method cannot measure three-dimensional information of the depth direction of the target in the field of view of the camera, and the three-dimensional information is an important parameter index in the chip packaging process.
Disclosure of Invention
The technical purpose is as follows: aiming at the defects of the prior art, the method and the system for online and high-precision visual measurement of the key three-dimensional geometric dimension of the pin of the semiconductor chip are provided, and the online detection of the quality of the chip product is realized by detecting the three-dimensional information such as the width, the height, the coplanarity and the like of the pin of the chip in the chip packaging process in real time.
The technical scheme is as follows: in order to achieve the technical purpose, the invention adopts the following technical scheme:
an on-line precision vision measuring method for the three-dimensional geometric dimension of a pin of a semiconductor chip is characterized in that the chip is provided with a plastic package surface and a chip pin, and comprises the following steps:
s1, initial image acquisition: shooting the same chip transmitted to the measuring position by two cameras at the same time to respectively obtain a left-eye initial image and a right-eye initial image; the two cameras can shoot a complete image of the chip positioned on the measuring position;
s2, feature point extraction: respectively preprocessing the left-eye initial image and the right-eye initial image, and capturing the interested regions of the images to obtain a corresponding left-eye image and a corresponding right-eye image; extracting coordinate information of chip pins and plastic package surface characteristic points from the left eye image and the right eye image, wherein the plastic package surface characteristic points comprise characters;
s3, feature point matching and three-dimensional coordinate obtaining: acquiring two coordinates of the same characteristic point P on a left-eye image and a right-eye image at the same moment, obtaining a parallax relation between the two coordinates, and obtaining a three-dimensional coordinate of the characteristic point P in a camera coordinate system according to the coordinates and the parallax relation between the coordinates; and calculating the width of the chip pins and the distance between the adjacent pins by calculating the three-dimensional geometric dimension between corner points of each pin.
Preferably, the method further comprises the steps of:
s4, judging whether the pins of the chip are qualified: and performing plane fitting on the three-dimensional characteristic points on the surface of the plastic package, respectively solving the three-dimensional geometric distance from the corner point of the pin of the chip to the reference plane by taking the three-dimensional plane of the plastic package surface as the reference plane, judging whether the calculated three-dimensional geometric distance is within a preset range, and further judging whether the pin of the chip is qualified.
Preferably, the two cameras are disposed at the same vertical position, and the step S3 includes:
s3.1, obtaining the coordinates of the same characteristic point P on the left-eye image and the right-eye image, wherein the coordinates are respectively as follows:
Pleft＝(Xleft,Yleft),Pright＝(Xright,Yright)；
s3.2, when the images of the two cameras are on the same plane, the Y coordinates of the image coordinates of the feature points P are the same, and the geometric relationship is obtained as follows:
Figure BDA0002705481350000021
wherein, B is the base line distance, namely the distance of the connecting line of the projection centers of the two cameras;
s3.3, calculating the three-dimensional coordinates of the feature point P in the camera coordinate system as follows:
Figure BDA0002705481350000022
wherein f is the focal length of the camera, and the Disparity is Xleft-Xright(ii) a Disparity represents parallax.
Preferably, the step S4 includes:
s4.1, in the binocular stereo vision system, the space point to be measured is marked as a point P, and the optical centers of the two cameras are respectively marked as ClAnd CrConnecting line of optical centers of two cameras
Figure BDA0002705481350000031
Taking the baseline as a baseline, wherein the intersection point of the baseline and the left-eye image and the right-eye image is a pole of the image, and the connecting line of the pole in the left-eye image and the projection point corresponds to the polar line of the projection point in the right-eye image and is recorded as
Figure BDA0002705481350000032
The line connecting the extreme point and the projection point in the right image corresponds to the epipolar line of the projection point in the left image and is recorded as
Figure BDA0002705481350000033
S4.2, in the binocular stereo vision system, the projection point P of any space point P in the left eye image and the right eye imagelAnd prThe following relationship is satisfied: left image projection point plCorresponding point p on the right imagerPolar lines defined in the right image plane
Figure BDA0002705481350000034
Upper and right image projection points prCorresponding point p on the left imagelPolar lines that are determined in the left image plane
Figure BDA0002705481350000035
The above step (1);
according to the relation, the internal and external parameters between cameras in the stereoscopic vision measuring system are obtained;
s4.3, according to the perspective projection relation of the camera, the equation of the left and right epipolar lines in the corresponding image can be obtained:
Figure BDA0002705481350000036
wherein F is a basis matrix, AlAnd ArThe three-dimensional coordinate system is characterized by comprising internal parameter matrixes of a left camera and a right camera respectively, R is a rotation matrix in external parameters of two camera coordinate systems, and S is an antisymmetric matrix of a translation matrix T in the external parameters of the two camera coordinate systems;
and S4.4, fitting the plastic package surface equation by using a least square method to obtain a chip plastic package surface equation under a camera coordinate system.
Preferably, in step S2, the positions of the corner points of the chip pins in the image are determined by using a corner point detection method based on a gradient correlation matrix.
Preferably, SURF algorithm is adopted to extract the characteristic points of the plastic package surface of the chip.
Preferably, if different types of chips are loaded in the braid, the steps S1-S3 are sequentially executed to complete the detection of each chip pin; and if the same type of chip is loaded in the braid, determining the region of interest of the image by using the prior information.
An online precision vision measurement system for three-dimensional geometric dimensions of a semiconductor chip pin is characterized in that: the measuring system comprises a measuring position, two cameras and an image processing module, wherein the measuring system is built on a chip packaging production line, the braid for carrying the chip to be detected is arranged on the chip packaging production line, the measuring system comprises the measuring position, the two cameras and the image processing module, when the braid is moved to the measuring position one chip at a time during transmission, the two cameras can both shoot the image of the chip positioned on the measuring position, and the image processing module is used for executing the method.
Preferably, the cameras arranged in the measuring system comprise a left camera and a right camera, and each camera is provided with a matched telephoto lens.
Has the advantages that: due to the adoption of the technical scheme, the invention has the following technical effects:
the invention discloses a binocular stereoscopic vision technology-based semiconductor chip pin key three-dimensional size precise online measurement method. The method of the invention not only can calculate the width distance of the chip pins, the spacing distance between adjacent pins and the coplanarity of the pins, but also can realize the detection of whether the pins warp and other common faults.
Drawings
FIG. 1 is a flow chart of the method for on-line precise visual measurement of the three-dimensional geometric dimension of a semiconductor chip pin according to the present invention;
FIG. 2 is a schematic structural diagram of a measurement system to which the three methods of the present invention are applied;
FIG. 3 is a schematic diagram of a head-up binocular stereo imaging;
FIG. 4 is a polar line geometric constraint relationship of a binocular stereo vision measurement system;
fig. 5 is a schematic diagram of a plastic package surface plane equation, plastic package surface character feature points and three-dimensional position structures of pin corner points under a camera coordinate system.
Detailed Description
The invention utilizes the stereoscopic vision measurement technology and combines the characteristics of the chip packaging process to research an on-line measurement method capable of realizing the three-dimensional size of the key part of the chip pin, and completes the on-line measurement of the key geometric size and the on-line detection of the chip packaging quality in the chip packaging process. The method comprises the following steps: firstly, extracting and analyzing the features of an image shot in a stereo vision measuring system, and extracting two key corner features on the outer side of each pin on a chip and the feature points on the surface of the chip. And then, performing feature matching on the pin feature points of the two images in the measuring system and the surface feature points of the chip plastic package body by using the calibrated binocular stereo vision measuring system to realize three-dimensional reconstruction of the pin feature points and the surface feature points of the plastic package body in a camera coordinate system. By calculating the three-dimensional geometric dimension between corner points of each pin, the width of the pins of the chip and the distance between adjacent pins can be realized. Meanwhile, the three-dimensional characteristic points on the surface of the plastic package are subjected to plane fitting, so that the three-dimensional distance between each pin and the surface of the plastic package can be realized, and further, whether the chip pin has faults such as pin warping or not can be detected.
Fig. 2 shows a schematic structural diagram of a measuring system, which includes a left camera 1 and a right camera 2, each camera is provided with a matched telephoto lens 3, a braid 4 carrying a semiconductor chip 5 to be detected is carried, and an arrow 7 shows a moving direction of the braid. The flow chart of the invention is shown in fig. 1, and comprises the following steps:
(1) a binocular stereoscopic vision measuring system is built for a chip packaging production line, so that when a braid carrying a chip device passes through a precision mechanical conveying device, the position of one chip is moved at each time, and the purpose is to enable two CCD vision sensors in the measuring system to respectively obtain a chip clear image at a fixed position of a mechanical device when the two CCD vision sensors shoot at each time.
(2) The braid chips pass through a precision mechanical conveyer, and the position and the direction of the chips in the braid are basically fixed. The precise mechanical conveying device moves for one braiding chip interval every time, the visual measurement sensor acquires a chip image of a fixed area every time, and chip pins are basically located at the image fixing positions. Therefore, aiming at the chips with the same type carried by the braid, the region of interest (ROI) of the image can be determined by utilizing the prior information, and the chip pin corner points and the plastic package surface feature points are quickly extracted in the ROI of the image.
(3) Binocular stereoscopic vision measurement principle:
the binocular stereo vision three-dimensional measurement is based on the parallax principle, as shown in fig. 3, a simple head-up binocular stereo imaging principle diagram is shown, 9 and 10 are image planes of a left camera and a right camera respectively, 8 is a target point P to be measured, 11 and 12 are feature points of the target point on the left image plane and the right image plane respectively, and 13 is the distance between the projection centers of the two cameras, namely the base line distance is B.
Two cameras view the same characteristic point P of a space object at the same time, and images of the point P are acquired on the left eye and the right eye respectively, and the image coordinates of the point P are as follows:
Pleft＝(Xleft,Yleft),Pright＝(Xright,Yright)
assuming that the images of the two cameras are on the same plane, the Y coordinates of the image coordinates of the feature points P are the same, i.e., Yright＝YrightLet Y be the focal length of the camera, and f, then we get from the geometric relationship:
Figure BDA0002705481350000051
then the parallax is Disparity Xleft-Xright. Therefore, the three-dimensional coordinates of the feature point P in the camera coordinate system can be calculated as follows:
Figure BDA0002705481350000052
therefore, any point on the left camera image plane can determine the three-dimensional coordinates of the point as long as the corresponding matching point (the two points are the points of the same point in space on the left and right camera image planes) can be found on the right camera image plane. The method is point-to-point operation, and all points on an image surface can participate in the operation as long as corresponding matching points exist, so that corresponding three-dimensional coordinates of the points are obtained.
(4) Fast extraction of chip characteristic points:
according to the binocular stereo vision measurement principle, if three-dimensional of the related information of the pins of the semiconductor chip is realized, the extraction of the image characteristic points is very critical. According to the pin measurement characteristics of the semiconductor chip, the invention analyzes feature point extraction from two aspects:
extracting angular points of pins of the A chip: due to the action of the braid and external light, edges and corners at two ends of a chip pin angular point sometimes present an elliptical state, so that the pin angular point characteristics are not obvious, and a chip pin target point cannot be accurately extracted by using a traditional angular point extraction algorithm, so that the three-dimensional reconstruction precision of a subsequent target point is influenced.
The invention adopts an angular point detection method (Zhang X, Wang H, Smith A W B, et al. corner detection based on gradient correlation matrix of planar currents [ J ]. Pattern Recognition,2010,43(4):1207-1223.) based on a gradient correlation matrix to effectively extract a chip image pin target point.
Firstly, extracting the image edge of the ROI by using a Canny edge extraction algorithm, and then smoothing the edge contour by using a Gaussian function, so that the algorithm has stronger robustness to image noise. Because the pin and the braid background color are obviously contrasted, the pin edge is easily extracted.
Next, assume that the above-mentioned extracted one-pin plane curve equation is
C(t)＝(x(t),y(t)) (3)
Wherein t is a curve equation parameter, x (t) and y (t) are coordinate functions.
From equation (3), the gradient vector of any point on the curve can be found:
C′(t)＝(x′(t),y′(t))＝(dx,dy) (4)
where dx, dy are the gradients of the curve C (t) in the x and y directions, respectively.
For a curve segment centered at any P point on the curve, the gradient vector around that point is denoted as (dx)i,dyi). To point (dx)i,dyi) The perpendicular distance between the point and the characteristic line, i.e. the projection of the point on the unit normal vector n of the characteristic line, is represented by di(n) represents the distance, i.e.
Figure BDA0002705481350000061
When the sum of squared distances of all gradient vector distribution points satisfies the minimum, the unit normal vector n satisfies the following minimum:
Figure BDA0002705481350000071
wherein k is the serial number of the kth edge point in the support area in the curve segment,
Figure BDA0002705481350000072
is the autocorrelation matrix of the gradient. The eigenvector corresponding to the minimum eigenvalue of the M matrix is the most matched gradient unit normalized vector, and the minimum eigenvalue reflects the distribution of the characteristics, namely the dispersion degree of the vectors in the gradient coordinates. The discrete form of the M matrix may be defined as:
Figure BDA0002705481350000073
wherein (Δ x)ki,Δyki) W is the ith order difference in the support region, and W is the radius of the support region.
The response degree of the corner points can be determined by analyzing the determinant of the M matrix, and the points with local maximum values larger than a preset threshold value in the response degree are screened out to be the solved corner points. According to the characteristics of the chip pin, the scale sigma of the Gaussian function is set to be 2.0, the radius W of the contour support area is set to be 1, the preset threshold value is 0.004 and 0.008, and the pin corner effect obtained by using the parameters is good.
The algorithm has moderate calculation complexity, high detection speed and good detection effect, and is suitable for extracting the pin corner points of the chip.
B, extracting characteristic points of the plastic package surface of the chip:
the printed characters on the surface of the chip plastic package have strong angular point information, and the character characteristic points on the surface of the plastic package can be subjected to three-dimensional reconstruction by using the extracted character characteristic points on the surface of the plastic package, so that the three-dimensional characteristic information on the surface of the plastic package is fitted. And (3) taking the three-dimensional plane of the plastic package surface as a reference surface, and respectively calculating the three-dimensional geometrical distance from the three-dimensional point information of the corner point of the chip pin to the plane by using a computer binocular stereo vision measurement principle, so that whether the chip pin has foot collapse and foot tilting faults, and the measurement of key geometrical information such as coplanarity of each pin can be judged.
Characters on the surface of each plastic package are clear and obvious, and feature information is convenient to extract. In consideration of the characteristic extraction efficiency, the invention adopts SURF algorithm (H.Bay, T.Tuytelaars, L.V.Gool, SURF: Speed Up Robust Features, ECCV,2006) to extract the character characteristic points on the surface of the plastic package, has stable characteristic extraction, stronger robustness on information such as rotation and scale, and fast extraction Speed, and is suitable for extracting the character characteristic on the surface of the plastic package.
(5) Feature point matching and three-dimensional reconstruction:
in the binocular stereo vision system, as shown in fig. 4, it is assumed that 14 is the spatial point P to be measured, 19 and 20 represent the optical centers C of the left and right cameras, respectivelylAnd CrConnecting line of optical centers of two cameras
Figure BDA0002705481350000074
As a baseline, the intersections 17 and 18 of the baseline with the two image planes are called the poles of the image, and the line 15 connecting the pole 17 with the projection point 21 in the left image is called the epipolar line corresponding to the projection point 22 in the right image (denoted as epipolar line)
Figure BDA0002705481350000081
) (ii) a Similarly, the line 16 connecting the pole 18 and the proxel 22 in the right image is called the epipolar line corresponding to the proxel 21 in the left image (denoted as epipolar line)
Figure BDA0002705481350000082
). That is, in the binocular stereoscopic vision system, the projection point P of an arbitrary spatial point P in the left and right imageslAnd prThe following relationship must be satisfied: left image projection point plCorresponding point p on the right imagerPolar lines defined in the right image plane
Figure BDA0002705481350000083
Upper and right image projection points prCorresponding point p on the left imagelPolar lines that are determined in the left image plane
Figure BDA0002705481350000084
The above. This relationship is known as the epipolar constraint between image planes in a stereoscopic system. Between cameras in a system for determining stereo visionAccording to the perspective projection relationship of the camera, the equation of the left and right epipolar lines in the corresponding image can be obtained:
Figure BDA0002705481350000085
wherein F is a basis matrix, AlAnd ArThe three-dimensional coordinate system is characterized by comprising internal parameter matrixes of a left camera and a right camera respectively, R is a rotation matrix in external parameters of two camera coordinate systems, and S is an antisymmetric matrix of a translation matrix T in the external parameters of the two camera coordinate systems.
The three-dimensional reconstruction of the image characteristic points in the binocular stereo vision measurement system is established on the basis of the parallax of the corresponding points, so that the matching of the corresponding edge points in the image is a key link for realizing the thickness measurement of the skateboard. And epipolar constraint gives an important relation of corresponding point matching search, and the search of corresponding matching points is compressed to a straight line from the whole image, so that the search range of target points is greatly reduced.
The extracted pin corner points and the plastic package surface characteristic points are registered by polar line constraint on the characteristic points on the two images shot by the binocular stereoscopic vision camera, so that the matching of target points can be accurately realized. The epipolar geometric constraint has certain significance on the rapid matching of the pin feature points of the semiconductor image, and is a key technology for realizing the three-dimensional reconstruction of the pin feature stereoscopic vision of the semiconductor chip. And the matched image characteristic point pairs are subjected to three-dimensional extraction of target characteristic points under a camera coordinate system by using a binocular vision measurement principle, so that the precise measurement of the key geometric dimension of the chip pin can be realized.
Examples
The vision measuring camera in FIG. 2 is Basler acA2500 camera, whose lens is RICOHFL-CC2518-5MX, and the left camera internal reference calibration result
Figure BDA0002705481350000086
Calibration result of internal parameters of right camera
Figure BDA0002705481350000091
Rotation matrix between left and right cameras
Figure BDA0002705481350000092
Translation matrix T ═ 95.5963-1.4835-142.8034]The extraction results of the corner points of the semiconductor chip pins (4 pins on the left and right, 2 corner points on each pin) in the left and right cameras are as follows:
TABLE 1
Figure BDA0002705481350000093
The extraction results of the 4 character SURF characteristic points corresponding to the plastic package surface of the semiconductor chip in the left camera and the right camera are respectively as follows:
TABLE 2
Figure BDA0002705481350000094
The algorithm and the process provided by the invention can be used for respectively calculating the three-dimensional coordinates of each pin characteristic point and the plastic package surface characteristic point under a camera coordinate system, and the three-dimensional coordinates are respectively as follows:
TABLE 3
Figure BDA0002705481350000101
TABLE 4
Figure BDA0002705481350000102
Fitting a plastic package surface equation by using a least square method to obtain the plastic package surface equation under a camera coordinate system as follows: -0.0598X-0.1122Y +0.9919Z-471.3773 ═ 0.
As shown in fig. 5, 23 is a plastic package surface plane equation under a camera coordinate system, 24 is three-dimensional coordinates of character feature points of the plastic package surface, and 25 is a three-dimensional position of a pin corner point.
And then, the vertical distance from the pin corner points to the plastic package surface can be realized by utilizing the obtained three-dimensional distance from the three-dimensional coordinates of the pin corner points to the plastic package surface, and the detection of whether the chip pin corner points have faults such as pin warping, pin collapsing and the like is completed.
The above description is only of the preferred embodiments of the present invention, and it should be noted that: it will be apparent to those skilled in the art that various modifications and adaptations can be made without departing from the principles of the invention and these are intended to be within the scope of the invention.

Claims (9)

1. An on-line precision vision measuring method for the three-dimensional geometric dimension of a pin of a semiconductor chip is characterized in that the chip is provided with a plastic package surface and a chip pin, and comprises the following steps:
s1, initial image acquisition: shooting the same chip transmitted to the measuring position by two cameras at the same time to respectively obtain a left-eye initial image and a right-eye initial image; the two cameras can shoot a complete image of the chip positioned on the measuring position;
s2, feature point extraction: respectively preprocessing the left-eye initial image and the right-eye initial image, and capturing the interested regions of the images to obtain a corresponding left-eye image and a corresponding right-eye image; extracting coordinate information of chip pins and plastic package surface characteristic points from the left eye image and the right eye image, wherein the plastic package surface characteristic points comprise characters;
s3, feature point matching and three-dimensional coordinate obtaining: acquiring two coordinates of the same characteristic point P on a left-eye image and a right-eye image at the same moment, obtaining a parallax relation between the two coordinates, and obtaining a three-dimensional coordinate of the characteristic point P in a camera coordinate system according to the coordinates and the parallax relation between the coordinates; and calculating the width of the chip pins and the distance between the adjacent pins by calculating the three-dimensional geometric dimension between corner points of each pin.
2. The method for on-line precise visual measurement of the three-dimensional geometric dimension of the pin of the semiconductor chip according to claim 1, further comprising the steps of:
s4, judging whether the pins of the chip are qualified: and performing plane fitting on the three-dimensional characteristic points on the surface of the plastic package, respectively solving the three-dimensional geometric distance from the corner point of the pin of the chip to the reference plane by taking the three-dimensional plane of the plastic package surface as the reference plane, judging whether the calculated three-dimensional geometric distance is within a preset range, and further judging whether the pin of the chip is qualified.
3. The method of claim 1, wherein the two cameras are disposed at the same vertical position, and step S3 includes:
s3.1, obtaining the coordinates of the same characteristic point P on the left-eye image and the right-eye image, wherein the coordinates are respectively as follows:
Pleft＝(Xleft,Yleft),Pright＝(Xright,Yright)；
s3.2, when the images of the two cameras are on the same plane, the Y coordinates of the image coordinates of the feature points P are the same, and the geometric relationship is obtained as follows:
Figure FDA0002705481340000011
wherein f is the focal length of the cameras, and B is the base line distance, namely the distance between the connecting lines of the projection centers of the two cameras;
s3.3, calculating the three-dimensional coordinates of the feature point P in the camera coordinate system as follows:
Figure FDA0002705481340000021
wherein, Disparity ═ Xleft-Xright(ii) a Disparity represents parallax.
4. The method for on-line precise visual measurement of three-dimensional geometric dimension of pin of semiconductor chip according to claim 1, wherein said step S4 comprises:
s4.1, in the binocular stereo vision system, the space point to be measured is marked as a point P, and the optical centers of the two cameras are respectively marked as ClAnd CrConnecting line of optical centers of two cameras
Figure FDA0002705481340000022
Taking the baseline as a baseline, wherein the intersection point of the baseline and the left-eye image and the right-eye image is a pole of the image, and the connecting line of the pole in the left-eye image and the projection point corresponds to the polar line of the projection point in the right-eye image and is recorded as
Figure FDA0002705481340000023
The line connecting the extreme point and the projection point in the right image corresponds to the epipolar line of the projection point in the left image and is recorded as
Figure FDA0002705481340000024
S4.2, in the binocular stereo vision system, the projection point P of any space point P in the left eye image and the right eye imagelAnd prThe following relationship is satisfied: left image projection point plCorresponding point P on the right imagerPolar lines defined in the right image plane
Figure FDA0002705481340000025
Upper and right image projection points PrCorresponding point p on the left imagelPolar lines that are determined in the left image plane
Figure FDA0002705481340000026
The above step (1);
according to the relation, the internal and external parameters between cameras in the stereoscopic vision measuring system are obtained;
s4.3, according to the perspective projection relation of the camera, the equation of the left and right epipolar lines in the corresponding image can be obtained:
Figure FDA0002705481340000027
wherein F is a basis matrix, AlAnd ArThe three-dimensional coordinate system is characterized by comprising internal parameter matrixes of a left camera and a right camera respectively, R is a rotation matrix in external parameters of two camera coordinate systems, and S is an antisymmetric matrix of a translation matrix T in the external parameters of the two camera coordinate systems;
and S4.4, fitting the plastic package surface equation by using a least square method to obtain a chip plastic package surface equation under a camera coordinate system.
5. The method as claimed in claim 1, wherein in step S2, the positions of the corner points of the chip pins in the image are determined by a corner point detection method based on a gradient correlation matrix.
6. The method for on-line precise visual measurement of the three-dimensional geometric dimension of the pin of the semiconductor chip according to claim 1, wherein the SURF algorithm is adopted to extract the characteristic points of the plastic package surface of the chip.
7. The on-line precise visual measurement method for the three-dimensional geometric dimension of the pin of the semiconductor chip according to claim 1, characterized in that: if the braid bears different types of chips, sequentially executing steps S1-S3 to complete the detection of the pins of each chip; and if the same type of chip is loaded in the braid, determining the region of interest of the image by using the prior information.
8. An online precision vision measurement system for three-dimensional geometric dimensions of a semiconductor chip pin is characterized in that: the measuring system comprises a measuring position, two cameras and an image processing module, wherein the measuring system is built on a chip packaging production line, a braid for carrying a chip to be detected is arranged on the chip packaging production line, the measuring system comprises the measuring position, the two cameras and the image processing module, when the braid is moved to the measuring position one chip at a time during transmission, the two cameras can both shoot images of the chip positioned on the measuring position, and the image processing module is used for executing the method of any one of claims 1 to 7.
9. The system of claim 8, wherein the system comprises: the cameras arranged in the measuring system comprise a left camera and a right camera, and each camera is provided with a matched telephoto lens.
CN202011037463.7A 2020-09-28 2020-09-28 Online precise visual measurement method and system for three-dimensional geometric dimension of semiconductor chip pin Active CN112161997B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011037463.7A CN112161997B (en) 2020-09-28 2020-09-28 Online precise visual measurement method and system for three-dimensional geometric dimension of semiconductor chip pin

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011037463.7A CN112161997B (en) 2020-09-28 2020-09-28 Online precise visual measurement method and system for three-dimensional geometric dimension of semiconductor chip pin

Publications (2)

Publication Number Publication Date
CN112161997A true CN112161997A (en) 2021-01-01
CN112161997B CN112161997B (en) 2022-09-27

Family

ID=73861754

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011037463.7A Active CN112161997B (en) 2020-09-28 2020-09-28 Online precise visual measurement method and system for three-dimensional geometric dimension of semiconductor chip pin

Country Status (1)

Country Link
CN (1) CN112161997B (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114152200A (en) * 2022-02-07 2022-03-08 北京航空航天大学杭州创新研究院 Chip pin measuring method and device
CN114252000A (en) * 2022-03-01 2022-03-29 北京京瀚禹电子工程技术有限公司 Device and method for testing coplanarity and spacing of chip pins
CN114445335A (en) * 2021-12-22 2022-05-06 武汉易思达科技有限公司 Vehicle running state monitoring method and system based on binocular machine vision
CN114577135A (en) * 2022-03-01 2022-06-03 合肥图迅电子科技有限公司 3D detection method and system for warpage of chip pin based on single lens
CN115248017A (en) * 2022-09-21 2022-10-28 江苏新恒基特种装备股份有限公司 Rapid detection method and detection system for geometric dimension of branch pipe fitting
CN115266743A (en) * 2022-05-17 2022-11-01 江苏汤谷智能科技有限公司 System and method for evaluating chip quality under nondestructive testing

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102032878A (en) * 2009-09-24 2011-04-27 甄海涛 Accurate on-line measurement method based on binocular stereo vision measurement system
CN103247053A (en) * 2013-05-16 2013-08-14 大连理工大学 Accurate part positioning method based on binocular microscopy stereo vision
CN104574367A (en) * 2014-12-21 2015-04-29 浙江大学 Method for detecting and calculating geometric size of integrated chip pin based on machine vision
CN106767399A (en) * 2016-11-11 2017-05-31 大连理工大学 The non-contact measurement method of the logistics measurement of cargo found range based on binocular stereo vision and dot laser
CN107588721A (en) * 2017-08-28 2018-01-16 武汉科技大学 The measuring method and system of a kind of more sizes of part based on binocular vision
CN107884767A (en) * 2017-10-31 2018-04-06 暨南大学 A kind of method of binocular vision system measurement ship distance and height
CN107907048A (en) * 2017-06-30 2018-04-13 长沙湘计海盾科技有限公司 A kind of binocular stereo vision method for three-dimensional measurement based on line-structured light scanning
CN110009690A (en) * 2019-03-23 2019-07-12 西安电子科技大学 Binocular stereo vision image measuring method based on polar curve correction
CN114255286A (en) * 2022-02-28 2022-03-29 常州罗博斯特机器人有限公司 Target size measuring method based on multi-view binocular vision perception

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102032878A (en) * 2009-09-24 2011-04-27 甄海涛 Accurate on-line measurement method based on binocular stereo vision measurement system
CN103247053A (en) * 2013-05-16 2013-08-14 大连理工大学 Accurate part positioning method based on binocular microscopy stereo vision
CN104574367A (en) * 2014-12-21 2015-04-29 浙江大学 Method for detecting and calculating geometric size of integrated chip pin based on machine vision
CN106767399A (en) * 2016-11-11 2017-05-31 大连理工大学 The non-contact measurement method of the logistics measurement of cargo found range based on binocular stereo vision and dot laser
CN107907048A (en) * 2017-06-30 2018-04-13 长沙湘计海盾科技有限公司 A kind of binocular stereo vision method for three-dimensional measurement based on line-structured light scanning
CN107588721A (en) * 2017-08-28 2018-01-16 武汉科技大学 The measuring method and system of a kind of more sizes of part based on binocular vision
CN107884767A (en) * 2017-10-31 2018-04-06 暨南大学 A kind of method of binocular vision system measurement ship distance and height
CN110009690A (en) * 2019-03-23 2019-07-12 西安电子科技大学 Binocular stereo vision image measuring method based on polar curve correction
CN114255286A (en) * 2022-02-28 2022-03-29 常州罗博斯特机器人有限公司 Target size measuring method based on multi-view binocular vision perception

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
DONG LI等: "A Robust 3-D Abrasion Diagnosis Method of Pantograph Slipper Based on Stereo Vision", 《IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT》 *
ZHIJING YU等: "The Research On Modeling of Stereo Vision Measurement and System Accuracy In Non-ideal Situation", 《APPLIED MECHANICS AND MATERIALS》 *

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114445335A (en) * 2021-12-22 2022-05-06 武汉易思达科技有限公司 Vehicle running state monitoring method and system based on binocular machine vision
CN114445335B (en) * 2021-12-22 2024-04-12 武汉易思达科技有限公司 Vehicle running state monitoring method based on binocular machine vision
CN114152200A (en) * 2022-02-07 2022-03-08 北京航空航天大学杭州创新研究院 Chip pin measuring method and device
CN114252000A (en) * 2022-03-01 2022-03-29 北京京瀚禹电子工程技术有限公司 Device and method for testing coplanarity and spacing of chip pins
CN114577135A (en) * 2022-03-01 2022-06-03 合肥图迅电子科技有限公司 3D detection method and system for warpage of chip pin based on single lens
CN114577135B (en) * 2022-03-01 2024-01-19 合肥图迅电子科技有限公司 3D detection method and system for chip pin warpage based on single lens
CN115266743A (en) * 2022-05-17 2022-11-01 江苏汤谷智能科技有限公司 System and method for evaluating chip quality under nondestructive testing
CN115248017A (en) * 2022-09-21 2022-10-28 江苏新恒基特种装备股份有限公司 Rapid detection method and detection system for geometric dimension of branch pipe fitting
CN115248017B (en) * 2022-09-21 2022-11-29 江苏新恒基特种装备股份有限公司 Rapid detection method and detection system for geometric dimension of branch pipe fitting

Also Published As

Publication number Publication date
CN112161997B (en) 2022-09-27

Similar Documents

Publication Publication Date Title
CN112161997B (en) Online precise visual measurement method and system for three-dimensional geometric dimension of semiconductor chip pin
US11720766B2 (en) Systems and methods for text and barcode reading under perspective distortion
Kim et al. A camera calibration method using concentric circles for vision applications
CN109859272A (en) A kind of auto-focusing binocular camera scaling method and device
CN109579695B (en) Part measuring method based on heterogeneous stereoscopic vision
EP2104365A1 (en) Method and apparatus for rapid three-dimensional restoration
CN109345597B (en) Camera calibration image acquisition method and device based on augmented reality
CN104713885A (en) Structured light-assisted binocular measuring method for on-line detection of PCB
CN110310331B (en) Pose estimation method based on combination of linear features and point cloud features
CN106996748A (en) Wheel diameter measuring method based on binocular vision
CN109727290A (en) Zoom camera dynamic calibrating method based on monocular vision triangle telemetry
CN110966956A (en) Binocular vision-based three-dimensional detection device and method
CN107957246A (en) Article geometrical size measuring method on conveyer belt based on binocular vision
Wang et al. Error analysis and improved calibration algorithm for LED chip localization system based on visual feedback
CN109636857B (en) Alignment method and calibration system
CN112802114B (en) Multi-vision sensor fusion device, method thereof and electronic equipment
CN115359127A (en) Polarization camera array calibration method suitable for multilayer medium environment
CN112489193A (en) Three-dimensional reconstruction method based on structured light
CN112595262B (en) Binocular structured light-based high-light-reflection surface workpiece depth image acquisition method
CN108537831B (en) Method and device for performing CT imaging on additive manufacturing workpiece
CN108180825A (en) A kind of identification of cuboid object dimensional and localization method based on line-structured light
CN116205993A (en) Double-telecentric lens high-precision calibration method for 3D AOI
CN206583440U (en) A kind of projected image sighting distance detecting system
Delmas et al. Stereo camera visual odometry for moving urban environments
CN114963981A (en) Monocular vision-based cylindrical part butt joint non-contact measurement method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right

Effective date of registration: 20240102

Address after: No. 22-3, No. 988 Gangcheng Avenue, Shengang Street, Jiangyin City, Wuxi City, Jiangsu Province, 214443

Patentee after: Jiangsu Kangtan Technology Co.,Ltd.

Address before: 1 No. 211167 Jiangsu city of Nanjing province Jiangning Science Park Hongjing Road

Patentee before: NANJING INSTITUTE OF TECHNOLOGY

TR01 Transfer of patent right