KR20010099825A - Apparatus and method to measure three- dimensional data - Google Patents

Abstract

The present invention provides a light source 8 for illuminating an object to be observed and means 11 for generating a plurality of parallel bundles of parallel rays which are separated from each other and which contain image information of the object being illuminated, respectively. Means (9) for collecting and sending light rays of parallel bundles of parallel rays to the imaging means, generating a image of a parallel bundle of parallel rays, the two-dimensional imaging area in which the generated image is depictable by a set of two coordinate axes Imaging means 2 which are separated from each other at 12 and corresponding corresponding to the two-dimensional imaging area 12 from the object at different bundles of parallel rays, which occur at exactly the same point h ₁ of the observed object. And an analysis device 10 for extracting the length, width, and height information of the object to be inspected 7 from the coordinate values of the incident point of the light beam. Three-dimensional data of which comprises a bundle of parallel corresponding blood observation unit 7 to an apparatus for optically measured.

Moreover, the present invention relates to a corresponding method for optically measuring three-dimensional data of an object to be observed.

Description

3D data measuring device and method {APPARATUS AND METHOD TO MEASURE THREE- DIMENSIONAL DATA}

Humans use both eyes to measure, for example, three-dimensional (3-D) objects of length, width, and third dimension. This three-dimensional can also be referred to as the distance from the viewer.

The present invention provides a novel apparatus and method for measuring 3-D data of length, width and height. For the purposes of the present invention the terms height and distance may be alternating with each other.

To make the field of the present invention easier to understand, the following general notes are provided in connection with the following well known apparatus and methods for measuring 3-D data.

1. Stereoscopic Vision

2. Depth from Focus

3. Laser Range Finding

4. Structured lighting

1. Stereoscopic Vision

It is well known that two-dimensional (2-D) measurements can be formed with one eye or one observation. To make three-dimensional measurements, a second eye or a second observation is required. However, if the eye is moved at a position corresponding to each observation, or if an object is moved at a position relative to the eye, a single eye can generate two or more observations.

In general, various devices are useful for three-dimensional measurements using methods of stereoscopic vision or stereopsis. 1 shows a conventional method using two imaging devices (e.g., two video cameras 2 and a lens 3), in which the path along the light rays propagated during image formation is critical due to the use of conventional lenses. Row, two images (one for each camera) of the observed object are generated.

FIG. 2 shows another conventional method using a single video camera 2 and a lens 3 used with a set of mirrors or prism mirrors 4. In addition, the path along the ray that proceeds during image formation is non-parallel. Moreover, this leads to the boundary of two images in a single video camera 2.

However, the well-known method as described in connection with FIGS. 1 and 2 using nonparallel light rays for generating at least two images of the object to be observed is characterized by the use of non-parallel light rays that proceed with multiple individual components. The space provided is relatively complex and large.

2. Depth from Focus

It is also possible to measure the third dimension by using a single observation from a single imaging sensor. This is accomplished by using an optical imaging system, for example a microscope, having an extremely narrow field depth. This microscope is mounted vertically on a calibrated stand. To measure the height (z-axis), the microscope is moved up and down until the point of interest is in focus. Height data on the calibration stand provides height measurements. Multiple points can be measured in this way. However, a drawback of this method is the low speed in taking measurements as servo-mechanical means must be used to focus the optical imaging system from one point to another. Further details can be found in FIG. 3 in which the camera 2 is shown with a lens 3 with an extremely narrow field width. The camera 2 and the lens 3 are in the up-and-down direction with respect to the object under observation along a calibration stand 5 having a height meter 6 indicating a tentative position of the camera 2 and the lens 3. Can be moved. The distance of the object to be observed corresponds to the reading of the height meter 6 corrected where the image of the object generated by the lens and the camera is out of focus, for example, not blurred or distorted.

3. Laser Range Finding

The third dimension of the distance can also be obtained by using a pulse or amplitude modulated wave or by calculating the phase change of the transmitted and reflected waves. Lasers, radars, and ultrasonic devices (so-called sound wave detection devices) are examples of devices using this technique. Measurements can be based on interferometers or echo detection techniques. Such techniques require expensive and highly sophisticated opto-electronic devices having transmitters, waveguide means and receivers for propagating waves transmitted directly to the object to be observed and reflected back by the object.

4. Structured lighting

Structural light includes the protection of light patterns on the surface of an object to be observed and studies shadows such as those generated by projected light or observing light reflected from one or more different angles. The classic example of structured light is the sundial.

Examples of structured light include projection of a point of light, observation of a line or lattice scale from an angle onto an object surface, and an optical pattern using a video camera from another angle. Light projection and camera are similar for the use of two observations.

The present invention relates to an apparatus and method for measuring three-dimensional data, and to measuring not only the length and width of an object, but also its height along with a relative distance to an observer.

1 is a view showing the use of two video cameras and lenses in a conventional stereoscopic view,

2 shows a conventional use of a mirror with one single video camera and lens to obtain stereoscopic vision;

3 is a view showing a conventional use of a progressive microscope obtained from the distance traveled in the vertical direction of the progressive microscope as the depth or height moves up and down for focus adjustment,

4 shows an apparatus for measuring three-dimensional data according to the present invention;

5A shows a single camera observation composed of left and right images obtained by using a stereoscopic image beamsplitter as shown in FIG. 4;

FIG. 5B illustrates the functional equivalent of the device shown in FIG. 4 when two cameras are used;

6 shows a path of light rays traveling through an uncoated right angle prism,

7 shows a representation of a 3-D coordinate system of a point in space h ₁ with coordinates (x ₁ , y ₁ , z ₁ ),

FIG. 8 is a graphical representation of two paths traveled by double beams from point h1 (x1, y1, z1) shown in FIG. 7 through the stereoscopic image beamsplitter shown in FIG.

9 is a diagrammatic representation of more image points and their respective relationships corresponding to FIG. 8;

FIG. 10 is a graphical representation corresponding to FIG. 8 showing the effect of skew and inconsistency prism in the left and right images of the device shown in FIG. 4;

FIG. 11 illustrates the use of a flatness correction device positioned flat in relation to the stereoscopic image beamsplitter shown in FIG. 4;

12 shows an elevated triangle (wedge) used as a calibration device for the z-axis and y-axis in relation to the stereoscopic image beamsplitter shown in FIG.

13 illustrates the use of a compound calibration object for use in calibrating all three dimensions;

14 shows an embodiment of a complex assembled device according to the invention,

FIG. 15 shows a mounting configuration of the device as shown in FIG. 14, showing that a barrel extension is used to provide convenience in the alignment of the prism with respect to the telecentric lens and camera. drawing,

16A illustrates the use of an integrated light source using transparent glass as a flat surface on which the three-dimensional prism used in the apparatus shown in FIG. 14 is aligned;

FIG. 16B illustrates an integrated light source as used in the integrated light mounting geometry shown in FIG. 16A using the addition of discrete LEDs or a continuous strip of fiber light; FIG.

17 illustrates the use of two concentric barrel extensions to allow rotation of the stereoscopic image beamsplitter shown in FIG. 4;

18 shows the use of a tri-image beamsplitter in the device according to the invention,

19 shows the use of a coaxial beamsplitter in an apparatus according to the invention to allow light projection through the plane of the three-phase beamsplitter shown in FIG. 18, FIG.

20 shows the height profiling of a wire performed by the use of the device according to the invention,

FIG. 21 shows height profiling of skewed wire that is not horizontally oriented in the representation corresponding to FIG. 20;

FIG. 22 shows the limits of height profiling of horizontally oriented wires in a representation corresponding to FIG. 20;

FIG. 23 illustrates the use of a quad-image pyramidal beamsplitter. FIG.

24 shows the use of a penta-image triangular pyramid prism;

25 illustrates a flowchart of the software process and methodology;

26a, b show a flowchart of a calibration sub-process,

FIG. 27 shows a flowchart of a teaching sub-process; FIG.

28A is a view showing 3-D observation of an inverted semiconductor device;

FIG. 28B is a view showing the side observation of the semiconductor device shown in FIG. 28A and the height change of the lead;

29 is a view showing height profiling of the leaded semiconductor device shown in FIGS. 28A and 28B;

30 shows a line projected onto the surface of a leaded semiconductor device with bumps;

31 illustrates the use of a three-beam splitter with an integrated light projection system in an apparatus according to the present invention;

32 shows the use of incremental scan along the horizontal axis in a method according to the invention,

33a, b schematically show a plan view and a perspective view of a ball grid array (BGA) with an array of solder balls;

34 is a diagram illustrating the height profiling of a BGA by using the method according to the invention integrated with line projection.

The present invention has been invented in view of the above, and an object thereof is to provide a simple apparatus and a simple method for measuring three-dimensional data in addition to known apparatus and methods for measuring 3-D data.

In particular, it is an object of the present invention to provide an apparatus and method for measuring 3-D data in only a single eye and a single observation as needed.

It is another object of the present invention to provide an apparatus and method for measuring 3-D data that does not require a calibrated high precision altimeter indicating the tentative position of the eye being observed.

It is yet another object of the present invention to provide an apparatus and method for measuring 3-D data that does not require sophisticated opto-electronic devices.

The invention also requires that 3-D data analysis be performed to perform very complex and time-consuming mathematical operations to extract relevant 3-D object data such as length, width and height from the image data provided by the imaging means. It is another object to provide an apparatus and method for measuring 3-D data that can be performed with conventional calculators, such as standard microprocessors based on personal computers, without requiring extremely high computing power.

This object is achieved in accordance with independent claim 1 in relation to the device and independence claim 2 in the method.

Dependent claims relate to embodiments of other advantages of the invention.

4 shows a device for measuring three-dimensional data according to the invention, which is a light source 8 for illuminating the object to be observed 7 and which is radiated from the illuminated object 7 to be observed. A stereo image beamsplitter (11) as a means for generating a plurality of parallel bundles of parallel rays, each of the parallel bundles of parallel rays containing image information of the object to be observed and separated from each other. Telecentric lens 9 as a means for collecting and sending light beams of parallel bundles of parallel rays to imaging means, generating an image of a parallel bundle of parallel rays, the resulting image being depicted by a set of two coordinate axes. An imaging sensor and an image forming apparatus such as a video camera 2 or the like as the imaging means separated from each other in the dimensional imaging area, NOTE Analysis for extracting the length, width and height information of the object to be observed 7 from the coordinates of the point of incidence of the corresponding light beam, which originates at the same point but is conducted from the object to the two-dimensional imaging area in another bundle of parallel rays. It is configured with an analysis device 10 using analysis device software as the device, characterized in that each image corresponds to one particular parallel bundle of parallel rays.

An embodiment of the device according to the invention is shown in FIG. 14, in which a micro-processor based on a personal computer 10 is connected to the video camera 2 via an interface cable 16. The video camera 2 has a telecentric lens 9 attached thereto. The wedge shaped prism that functions as the stereoscopic image beam splitter 11 is aligned and attached to the telecentric lens 9 using the barrel extension 17. The object to be inspected 7 is illuminated according to the light source 8.

The image information contained in the parallel bundles of parallel rays passing through the opposite side of the stereoscopic image beam splitter 11 is displayed in another part of the image area of the video camera 2, and then the PC 10 via the interface cable 16. ), Where the analysis results are displayed on the PC and appropriate action is taken.

Imaging Sensors and Image Forming Devices

The present invention utilizes one-dimensional (1-D) or two-dimensional (2-D) imaging sensors to detect the formation of an image formed on the image plane. The imaging sensor used is integral with a number of pixels with each pixel sensing the intensity of the formed image. Certain sensors of visible and invisible light may be used that serve this purpose. Examples of such sensors include video cameras, line scan cameras and X-ray imaging sensors.

Parallel light as shown represents the path through which the light traveled by the use of a telecentric lens. There is no parallax and no field of view distortion with the use of the telecentric lens 9.

Consideration should be given to ensuring that the image forming apparatus, such as the optical lens, has a large depth of focus. This is in contrast to the prior art for calculating the depth of focus described above with reference to FIG. Depth of focus requires a very narrow depth of focus. In addition, in order to maintain the area of the object of interest within the field of view of the imaging sensor, careful selection of the focal length and focal length is required.

As another feature, the present invention uses a telecentric lens 9 to simplify the calculation and analysis. Telecentric lenses different from ordinary lenses provide parallel viewing angles without parallax or field distortion as shown in FIG. 4. In addition, standard non-telecentric lenses, such as 50 mm macro optical lenses, can be used to make additional calculations needed during the analysis.

In another embodiment of the present invention, the present invention also uses a telecentric lens of variable magnification similar to a zoom lens. This allows various magnified views to see an object belonging to the field of view as one or more other detailed objects. In addition, in the zoom feature of the telecentric lens, a convenient configuration provides an optimal field of view.

Stereo Image Beam Splitter / Bi-Image Beam Splitter

In a first preferred embodiment of the invention, an image forming apparatus and a single imaging sensor with a stereo image beamsplitter are used to obtain a pair of stereo images. One example of a stereo image beamsplitter used in the present invention is an uncoated right-angle prism device as shown in FIG. 4 or FIG.

FIG. 5 shows a single camera field of view consisting of left and right images by use of a stereo image beamsplitter as shown in FIG. 4.

Prism device 11 splits two 1 / 2-image objects or images of objects that are divided perpendicular to the center of imaging area 12. For the purposes of the present invention, these images are referred to as "left" and "right" images as shown in FIG. 5A.

FIG. 5A is a simple representation of the rays split by the prism and formed through the telecentric lens 9 of the imaging sensor. The composite left and right images shown by the imaging sensor are shown in this figure. For the sake of simplicity, the optical elements representing the telecentric lens 9 are not shown and the image reversal is not shown due to the proper ray tracing.

The left and right images are similar but not identical. The composite image corresponds to the combined field of view of the two imaging sensors (one as substituted at the left angle and the other at the right angle, as shown in FIG. 5B). These two images form a stereo pair.

FIG. 5B shows the functional equivalent of the device shown in FIG. 4 when two cameras 2, 2 '(one is replaced with a left angle and the other with a right angle) are used. With the use of two telecentric lenses 9, 9 'the light beams are parallel.

6 shows the course of the light rays traveling through the uncoated right angle prism 11. In this case, the angle difference between the stereo pairs of the left and right images is = 45 degrees.

For example, if the present invention uses a right angle prism having an appropriate refractive index, a viewing angle of 22.5 degrees from vertical can be obtained by the left and right fields of view. The angle difference between two stereo pairs is 45 degrees as shown in FIG.

The choice of the right angle of the prism is one of convenience. Certain prisms are provided with narrower or wider apex angles in which the left and right images include associated objects.

The prism can also be made using a variety of materials and refractive indices, such as glass or plastic. Another example of a prism that can be used includes a pair of Brewster or wedge prisms or pentagonal prisms. In addition, the angle difference between the two stereo pairs need not be exactly 45 degrees, and other angles may be used.

Although right angle prisms can be readily used and used for right angle reflectors or for other purposes, the present invention uses a new method of uncoding right angle prisms for dividing a single field of view into two fields of view.

Light source

In a preferred embodiment, the present invention uses a plurality of mirrors or uncoated prisms rather than equivalent to prisms.

However, the use of a prism causes a prism effect of white light dispersion. When white light is refracted twice through the prism, the white light is dispersed in a spectrum of various colors other than rainbow. Although this effect is favorable to others, it distorts the image in the present invention. This color shift blurs the image.

At low magnification of the image, this effect can be neglected since the image quality still accepts field of view and low resolution measurements. However, for high magnification and high resolution measurements, these distortions interfere with accurate measurements.

To overcome this variation, the present invention uses light sources with a single frequency or monochromatic light with a narrow spectrum. One example of light having a single frequency is an array of red light emitting diodes (LEDs), such as a light source. In addition, color lasers or laser diodes can be used to achieve the same effect. In addition, white light may be filtered with a narrowband filter, and an imaging lens may be installed in such a filter.

If a color image is desired, the present invention illuminates the object with three colored lights (eg red, green and blue), respectively, to construct a color image.

Another design consideration of the present invention for reducing color shift is to minimize the size of the prism to reduce the distance that light travels within the prism.

The light source may be positioned at a predetermined position to achieve optimal lighting conditions. One example of a light source arrangement is shown in FIGS. 4 and 14. By the arrangement of the light source 8, the object 7 observed can be evenly illuminated.

Analysis device

The present invention uses the analyzer 10 to calculate three dimensions of an object as shown in FIG.

In the present invention, the analysis device is a computerized machine that allows a sequence of commands to be performed to input and output information. The sequence command is the software of the analyzer. One example of an analysis device is a personal computer (PC). The PC used in the present invention has means for interfacing with imaging sensors, other analyzers, machines and humans.

In the present invention, the analysis device may convert the image formed on the image sensor into information or data that can be calculated by analyzing the analysis device. For example, an image can be decomposed into a one or two dimensional array of pixels with digital intensity information.

Analyzer Software

The simplicity of the present invention can only be evaluated when considering the simplicity in three-dimensional calculation and analysis required for use of the present invention.

Conventional stereo imaging systems require synthetic three-dimensional calculations to calculate three-dimensional information. The method for this is described in an easy-to-use textbook.

Through the use of telecentric lenses and the selective use of right angle prisms, the present invention reduces the complexity of the analysis.

Corresponding pixel points and corresponding pixel distances

The present invention uses the corresponding inter-pixel spatial distances of the left and right images to calculate the height or distance data.

For simplicity, the following considerations are the basis. That is, a composite image beamsplitter is used as a means for generating two parallel bundles of parallel rays, wherein each parallel bundle of parallel rays contains image information of the illuminated object being observed and the parallel bundles of parallel rays The imaging area of the imaging means is separated from each other.

In this example, the stereo image beamsplitter splits the image into two images. Thus, an image point in one image can be seen in another image (provided that the dot is blocked).

7 shows a representation of a three-dimensional coordinate system of a point h _{1 in} space with coordinates (x ₁ , y ₁ , z ₁ ). The points h ₁ (x ₁ , y ₁ , z ₁ ) in space are located in X, Y and Z which are the coordinate axes of the points in space with respect to both images, i.e., the origin.

8 is a view through the stereo image beam splitter 11 shown in FIG. 4, represented by the left and right images of point h ₁ as seen in imaging region 12 (FIG. 5 a) by camera 2 (FIG. 4). 7 points _{h 1 (x 1, y 1} , z 1) it represents the course of two conducted by light from the pair shown in. Two corresponding points (Fig. 8) on the left and right images are called corresponding pixel points (CPPs), and the horizontal distance between them is called a corresponding pixel distance (CPD).

The points in the space shown by the left and right images as pixel points on the imaging sensor are shown in FIG. 8. Due to the vertical alignment of the two faces to the sensor and the configuration of the prism device, the two points on the left and right images fall along the same horizontal line. That is, a point shown in one image may be found in another image along the same horizontal line. This is called the corresponding pixel point (CPP) Corresponding PixelPoint. Each light beam is represented by a dashed line in FIG. 8.

FIG. 9 shows a number of points corresponding to those shown in FIG. 8 and the relationship between CPP and CPD. The farther object point is away from the beam splitter 11, the larger CPD. Moreover, even if the point is offset along the x and y axes, it does not change the CPD. This is explained in more detail below.

FIG. 10 shows the oblique results of the left and right images of the device according to the invention and the effect of misaligned prisms corresponding to those shown in FIG. 8. The distance between two corresponding pixel points in the left and right images seen by the imaging sensor is called the corresponding pixel distance (CPD). However, instead of searching along the horizontal axis, searching along the orthogonal axis may yield a corresponding CPP.

Referring again to FIG. 9, consider another point h ₂ (x ₁ , y ₁ , z ₂ ) with different z-coordinate positions at different heights. Far away from the imaging sensor, such as point h ₂ , each pixel point falls along the same horizontal line but does not fall further apart. Thus, the CPD is larger. Considering the closer points h ₂ (x ₁ , y ₁ , z ₃ ), the CPD is smaller.

Now consider the point off the center of the y-axis of FIG. 9, that is, h ₄ (x ₂ , y ₁ , z ₁ ). Even though the points are substituted to the left, the CPD is the same as the previous point h ₁ (x ₁ , y ₁ , z ₁ ) because they are at the same height. Similarly, if the point is off the center of the y-axis, the CPD remains the same as indicated by point h ₅ (x ₂ , y ₂ , z ₁ ).

Although the principle is that the imaging area 12 becomes a predetermined curved imaging area which can be described by two sets of coordinate axes of a preferred embodiment of the present invention described and described by means of Figs. 4-14, preferably imaging This uniform plane results in a strictly linear proportional relationship between the distance of the object point to the imaging sensor and the corresponding pixel distance between the left and right images of the object point of the imaging plane. This greatly simplifies the calculation and analysis of CPD calculations, and enables to perform three-dimensional analysis simply using standard commercially available personal computers without having to have extremely high computing power.

The use of a telecentric lens 9 and a precision prism simplifies the three dimensional correction process. In addition, the use of the telecentric lens 9 ensures that there is no parallax or visual distortion, and therefore the need for these distortion corrections is reduced.

In a preferred embodiment of the present invention, the search for the corresponding pixel point that produces the stereo image can be found by searching along the horizontal axis of another image. This simplifies the image processing required for the search as well as the search.

Functional operation of the analysis device

The functional operation of the analyzer is defined in two steps below. In other words,

a) calibration;

b) measurement.

correction

The calibration needs to calculate the pixels as relationships of physical size instead of calculating relationships using complex mathematical formulas.

FIG. 11 shows the use of the flatness correction device 13 positioned flat against the stereo image beamsplitter 11 shown in FIG. 4. The results of the left and right images are shown. It also shows that the prism edge is normalized at right angles to the horizontal.

FIG. 12 shows raised triangles 14 (wedge shaped) used as the correctors for the z and y axes for the stereo image beamsplitter 11 shown in FIG. 4. Once the z-axis correction factor is found, a known diagonal distance between the two points z ₁ , z ₂ is used to calculate the actual y-axis correction factor.

FIG. 13 shows the use of a constitutive correction object 15 (in the form of a double wedge) used to correct all three dimensions.

In order to calibrate the invention with three dimensions, a three-dimensional object with known length (x-axis), width (y-axis) and height (z-axis) dimensions is placed in the field of view in a setup in front of the telecentric lens. In order to successfully calibrate each axis, each axis must have a known pair of points along that axis.

In the invention, the calibration can be performed using certain salient features that can be seen in left and right images. For example, corner points, reference points, edges, centers of features, blocks of textures, patterns, features enhanced by illuminated light or constructed light, and the like may be used as indicators for performing calibration.

Y axis calibration

To correct the y-axis, an object with a known width is used. One example of such a calibration device is a flat rectangular device 12 on opposite sides of parallel as shown in FIG. For calibration, the calibration device is placed in the imaging system and aligned vertically and horizontally. Also, the calibration device should be flat and preferably the split edges of the two faces at the apex of the prism are aligned perpendicular to the imaging sensor.

At this time, one starts with the left image. If the width (y-axis) of the rectangles (x ₁ , y ₁ , z ₁ ), (x ₁ , y ₂ , z ₂ ) is 10 mm, and the number of pixels used to represent this length by the analyzer is 100 pixels, The vertical resolution of each pixel is equal to 0.1 mm. Using standard image processing subpixel methodologies, it is possible to represent small portions of pixels.

This process can be repeated in the right image to check a plurality of things, namely to align the assembly and proper structure of the prism device to the calibration process and the arrangement of the calibration device.

If the prism is properly configured and the alignment is correct, the corrected readings from both images should be identical.

The use of a telecentric lens in the present invention means that the pixel resolution of the y-axis of the left and right images is constant throughout the field of view and does not depend on space invariant, ie, spatial coordinates.

Z axis calibration

Likewise, to calibrate the z-axis, an object with a known height is used for calibration. For example, as shown in FIG. 12, a rectangular wedge shaped device 14 with raised sides is used. The calibration device is aligned as before, with the CPP determined, and as shown, the CPDs at two points are calculated at two different heights. At this time, the height resolution relationship for the CPD is calculated. For example, if the heights (z-axis) of the rectangles (x ₁ , y ₁ , z ₁ ), (x ₁ , y ₁ , z ₂ ) are 15 mm, the number of CPDs in the pixels used to represent this length by the analyzer Is 100 pixels, the height resolution of each pixel is equal to 0.15 mm.

Again, the subpixel resolution can be used for high resolution representation.

X axis calibration

The x-axis calibration can be performed in a similar manner as the calibration of the y and z axes.

However, because it is difficult for the calibration device to align and guarantee flatness with respect to the imaging sensor, other methods using absolute distance may be used.

The two diagonal points representing the length of the object are chosen as before.

Since the points are at different heights, the absolute distance equation between two points, i.e., z ² = x ² + y ² (Pythagorean Theorem) is used, where x, y, z are each distance between two points.

Combined X, Y, Z Axis Calibrator

The calibration of all three axes can be combined using a single calibration device. For example, the present invention uses a single double wedge shaped calibration device 15 as shown in FIG. 13 to perform calibration of three axes.

Measure

Once the present invention is calibrated, 2-D data can be obtained from either the left image or the right image, which is more convenient either way and has less chance of occlusion. The measured distance in the pixel is converted into physical dimensions using calibration factors. As an example, using a faster ycalibration factor of 0.1 mm per pixel—if the measurement is 80.2 pixels, it is transformed into a physical dimension of 8.02 mm.

However, determination of 3-D height data requires both left and right images. The height is obtained from the calibrated CPD, ie, the CPP pair of certain feature points on the object.

If the CPD difference between the two points is 42.5 pixels, using a faster z correction factor of 0.15 mm per pixel, the height difference between these two points is 6.375 mm.

Software Process and Methodology

25-27 show flowcharts for visualizing an inventive method for obtaining 3-D data of an object to be viewed.

As shown in Fig. 25, the steps of those main processes numbered sequentially are as follows.

Main processing:

1) System Initialization

-Boot up the analyzer.

-Initialize all variables and parameters.

-Other system initialization function.

2) System startup

-Activate the image sensor and image acquisition device.

-Start up the brightness and contrast values.

-Activate the proper focus and field depth.

-Activate peripherals, eg printers, etc.

-Different starting functions.

3) calibration

Has the calibration been done before and has not changed?

If the answer is YES, go to step 4)

-If the answer is NO, go to step A) calibration subprocess before proceeding to the next step.

4) Teaching

Has teaching been done before and has not changed?

If the answer is YES, go to step 5)

If the answer is NO, go to step B) Teaching Subprocess before proceeding to the next step.

5) Perform measurement of current unit

Teaching information is used to extract points of interest in the image.

Extract the first set of CPPs from the left image in 2-D coordinates.

For each CPP, search for the corresponding CPP in 2-D coordinates along the direction perpendicular to the prism axis of the right image in 2-D.

Correlate the CPP pairs between the left and right images.

For each CPP pair, calculate the CPD in the pixel.

For each CPD pixel measurement, the CPD is transformed into a physical height measurement (eg millimeters or thousands of inches).

Integrate the physical measurements of all three axes of all points of interest in the image.

6) Analyzing the measured data

Compute and analyze the relationships between points of interest in the image, for example, to track the profile along one or more lines or edges.

Compute and analyze the physical dimensions between points of interest in the image, such as the height of the profile along the line.

7) Decision and Execution

From the analyzed and measured data, determine and execute.

-Accept a checked unit that is within acceptable 3-D measurement tolerances, and send it to a PASS bin, or

Reject the checked object outside the acceptable 3-D measurement tolerance and send it to the FAIL bin.

Another example is to classify objects of different heights into their respective bins.

In other words, the value determined for the dimension of the object to be viewed is compared with a set of predetermined dimension values of the reference object, and includes at least one attribute value (eg, "PASS", "FAIL"; After being assigned to an object to be viewed based on the result of the comparison of "SIZE CATEGORY 1", "SIZE CATEGORY 2", etc.), the object to be viewed is further manipulated based on at least one attribute value previously assigned to the object. (Eg, an object resulting in the attribute value "FAIL" is sent to the FAIL bean).

8) next unit

If the next unit is valid, go to step 5).

If the unit is not valid, go to step 9).

9) Termination of Inspection

Terminate and terminate processing properly and take the necessary steps to shut down the system if necessary.

The steps of the calibration process numbered sequentially as shown in Figs. 26A and 26B are as follows.

Calibration subprocess

i)-Insert the calibration device into the field of view of the system.

ii)-start at the Z-axis feature point.

iii) extract the known feature points of interest of the calibration device starting by calculating the first set of CPPs from the left image in 2-D coordinates.

iv)-for each CPP, retrieve the corresponding CPP in 2-D coordinates along the horizontal direction.

v)-if the corresponding CPP is not found along the horizontal direction,

a) Intelligently and automatically retrieves the entire right image for a corresponding CPP.

An example of intelligent search is

b) prioritize the search very close to the expected location of the corresponding CPP.

c) If the automatic search does not succeed, manually search by the human operator.

d) Check for additional points if higher precision, reliability and repeatability are required.

e) Calculate the actual vertical axis and direction.

vi) Compute the CPD of the pixel for each CPP pair.

viii) For each CPD pixel measurement, a calibration factor is calculated from the CPD at the physical height measurement per pixel (eg, millimeters or thousands of inches).

ix) Similarly, calculate the correction factor for the Y axis.

x) Similarly, calculate the correction factor for the X axis.

xi) End the subprocess and return to main processing.

As shown in Fig. 27, the steps of the teaching process numbered sequentially are as follows.

Teaching Subprocess

i) teaching the sample form

Using the user interface, the analytical device (PC, etc.) taught the SOT23 semiconductor device type, which relates to the type of sample to be examined, for example having two wire profiles.

ii) teaching the properties of the device to be examined;

For example, the color and gray level histogram of the wires of the SOT23 device were taught to the PC.

iii) teaching two areas of interest;

A two-wire SOT23 device with two areas of interest, for example, with the starting and ending points of the wire profile in each of the left and right images, each area of interest surrounding each wire.

iv) teaching search zones and landmarks;

Since the original area of the location of interest may not be accurate, a search in a larger search zone would have to be done to locate each wire.

Also, these search zones can be related relative to the landmark. A landmark is a distinctive feature or point in an image that can be easily located.

v) teaches and defines a range of search parameters

-Optimal form and optimized parameters of the search methodology for searching.

Other teaching parameters as required.

According to the present invention, if the permutation of the sequence still achieves the final result, it does not need to be accompanied by the correct functional sequence of operation. For example, the steps of calibration and teaching are interchanged in the sequence. That is, teaching may be performed before calibration. Also, it is not important to perform the correction on the left image before the right image.

In another embodiment, the sequence of finding the complete list of CPPs of the left image need not be strictly followed before finding the list of CPPs of the right image. For example, each pair of CPPs can be found from the left and right images carried by the next pair. Another embodiment is that the sequence of calibration of each axis can be interchanged.

Moreover, if steps are completed and cannot be interchanged, the information associated with the steps can be saved and recalled for later use. This step includes, for example, setup and calibration teaching.

The present invention allows the use of a parallel computation architecture as an analysis device. The parallel analyzer allows you to calculate the entire left and right images, CPP and CPD in parallel rather than sequence. In addition, parallelism can be extended to include many other forms of analytical devices of the present invention, such as extraction and matching and searching, correlation, calculation and calculation.

In a parallel system, the flowcharts need not be continuous and can be performed in parallel.

It is not critical that the exact dimensions of each component of the invention be provided so that the functional purpose of the component and the invention are met. Similarly, the arrangement of the components or the location of the features of each component need not be provided so that the functional purpose of the components and the present invention are met.

Height profiling

As noted above, the inspection of the CPP along the horizontal or orthogonal axis makes the search for the corresponding stereo point very simple. For example, if an object tracks wire height in a vertical orientation, matching feature points along the corresponding CPP allows the swamp profile to be found.

FIG. 20 shows the height profiling of the wire as performed by using the apparatus according to the invention, which is shown in FIG. 20 with the use of five CPP points. The present invention allows the use of multiple CPP points limited only by the resolution of the imaging device and the number of feature points that can be extracted from the image.

For more complex objects, the points and edges of the object are similarly profiled to map the complete feature points of the object.

In another example, the wires are not completely horizontally oriented as shown in FIG. 21 showing the height profiling of the beveled wires, which are not oriented horizontally in a representation corresponding to FIG. 21. Even if the wire is oblique, the position of the CPP can still perform profiling.

22 shows the limitation of the height profiling of the horizontally oriented wire. If the wire is oriented exactly with the horizontal axis as shown in FIG. 22, only the end point can be determined. However, the present invention allows for a suitable orientation in which the complete 3-D measuring device is rotated 90 degrees or allowing profiling. The present invention also allows the prism component of the assembly to be rotated.

The present invention allows a gain for CPP points that are extracted simultaneously for height profiling from one image. Additional observation is required after proper reorientation, except when the object is oriented horizontally.

In the present invention, there is no need to mechanically or electrically shift the focus. In addition, in the present invention, it is not necessary to select another observation of the object in order to move the object with respect to the imaging device and determine the height or distance.

Lead height measurement

In addition, the present invention can be used to measure the height profile of a lead of a semiconductor device.

FIG. 28A is a 3-D view of the inverted semiconductor device, and FIG. 28B is a side view of the inverted semiconductor device emphasizing the height of the deviation in the lid 30.

29 is a view showing height profiling of a semiconductor device in which leads are installed in the apparatus of the present invention.

FIG. 28A shows an inverted 6-lead semiconductor device such as an SOIC package, and FIG. 28B shows a side of the semiconductor device. The invention can be used to measure the distance of each of the 6-leads spaced from the reference plane.

As shown in Fig. 28, the leads 30 and 30 'differ in distance from the reference plane (e.g., the bottom plane of the semiconductor device placed upside down).

As shown in FIG. 29, the center of each lead tip can be found by finding the tip of each lead and by taking the center of the left and right sides of the dip. Calculate the six lead tip centers from the left image. These six points are the CPPs of the left image, namely CPP1, CPP2 and the like. Similarly, the corresponding six lead tip centers, namely CPPa, CPPb, etc., are calculated from the right image. If the prism is properly aligned, the corresponding point can be found along the vertical axis, otherwise search along the orthogonal axis.

Calculating the CPD of each CPP point, namely CPP1 and CPPa and CPPb, results in the required 3-D height measurement. In the embodiment shown in FIG. 29, the intermediate lead is further spaced away from the camera system. Therefore, the CPD becomes larger compared to the CPD of other leads.

The height measurement may be based on the camera, which may alternatively be changed to the seating surface of the semiconductor device.

In another preferred embodiment of the present invention shown in FIG. 15, a mounting feature is provided in which the barrel extension 17 is provided for convenient alignment of the prism 11 with respect to the telecentric lens 9 and the camera 2. do. In FIG. 15, the telecentric lens 9 is connected to the threaded barrel extension 17. The other end of the barrel extension 17 has a flat surface 19 with a hole 18 for the prism. The flat surface 19 is attached to the end of the barrel extension 17. Alternatively, the flat surface bearing the prism can be rotated and fixed by fastening means such as screws or some other fastening means.

16A and 16B show integrated light-mounting features integrated with the barrel extension 17.

FIG. 16a shows the use of an integrated light source 8 using transparent glass as a flat surface 19 in which the three-dimensional prism 11 used in the apparatus shown in FIG. 4 is aligned. The prism 11 is adhered to the glass surface, but a hole need not be provided in the glass. The light source shown is an LED mounted on the inner side of the flat surface 19.

FIG. 16B shows the integrated light source 8 as used for the addition of discontinuous LEDs or for the integrated optical properties shown in FIG. 16 used as the continuous strip 28 of the optical fiber.

For the optical lens image formation system, the present invention uses a hollow thin cylinder formed with a barrel extension 17-threaded as shown in Fig. 15B. At one end, a flat surface 19 having a hole 18 cut in the center is attached as shown in FIG. 15. The arrangement of the prism device 11 with respect to the center of the barrel 17 and the hole 18 is centered.

Correct horizontal alignment of the prism is not important in the present invention where the analyzer software can correct fine misalignment. The diameter of the barrel 17 is the same as that of the lens 9, and by matching the threads, the barrel extension 17 can be attached to the lens 9. The length of the barrel is selected to provide the optimum depth of field when the prism device is used in combination with a telecentric lens. On the other hand, the present invention uses a fixed length barrel extension or a combination of a barrel and a variable length extension.

On the other hand, this invention uses transparent flat glass as a flat surface. The prism device is then bonded to the surface using an optical adhesive. In this case, it is not necessary to cut the hole in the center. The use of a transparent surface has the additional advantage that the light source can be integrated in the barrel, as shown in FIGS. 16A and 16B, and can be illuminated through a transparent flat surface.

Stereo Image Beam Splitter

The present invention uses prismatic devices instead of mirrors or sets of prisms for simple alignment. One consideration is to physically align the prism device alignment with the imaging sensor. This is done by aligning the horizontal edge of the prism device at right angles with the line of sight of the imaging sensor shown in FIG. An example of oblique alignment is shown in FIG. 10.

Stereo Image Beam Splitter Rotating Features

The present invention provides features for horizontal alignment alignment. In order to allow the stereo image splitter to rotate relative to the image sensor, the present invention has an optional feature. The barrel extension of the present invention consists of two interleaved central hubs 17 and 20 with a locking screw 21 for locking the rotational orientation. This is shown in FIG. 17.

On the other hand, the present invention can search in the orthogonal direction by software for CPP when the stereo beamsplitter is mounted and the orientation is fixed.

Alternatives to the bi-image described above may be employed as a means for generating multiple parallel bundles of parallel ray multi-beam splitters.

Tri-Image Beam Splitter Features

The present invention includes features that can achieve three-dimensional observation using the three-phase beam splitter 22 or the dove prism shown in FIG. Additional faces parallel to the horizontal base are made in the prism device. This additional surface provides planar image observation. As described above, the three-phase beamsplitter splits the image into stereo pairs and includes a planar image in the center of the image, as shown in FIG. Although dove prisms are available as optics for rotating an image, the present invention uses a novel method of the dove prism concept to split a single observation of an object in three directions.

FIG. 19 shows that light projection through the plane of the three-beam splitter shown in FIG. 18 is allowed using the coaxial beamsplitter 24 in the apparatus of the present invention. The light pattern is projected by the light projector 23 to the coaxial beam splitter 24 from the left side. When the projected pattern is cast on the object, the planar image as well as the left and right images observe the object simultaneously.

Quad-image beam splitter features

The present invention can be further extended by using a pyramid prism as shown in FIG. 23 using a four-phase pyramid beamsplitter 26. Beamsplitter 26 splits the observation of an object into four images, allowing simultaneous observation of life and vertical profile modes. An additional two-view is equivalent to a 90 ° prism rotation, resulting in a stereo image of the other 2-D axis.

The resulting image is shown in FIG. Using the four images, the present invention can easily calculate the CPD without any physical rotation of certain portions of the present invention, with both horizontal and vertical profiles.

Penta-Image Beam Splitter Features

In the present invention, as shown in Fig. 24, the apex of the pyramid prism is cut out to generate five phases. This is similar to a three-phase beamsplitter with two additional orthogonal planes to produce two orthogonal observations as shown in FIG. 24.

The use of a five-phase beamsplitter 27 is similar to a three-phase beamsplitter where the horizontal plane of the prism can be used for planar observation and light projection described in the next section.

Light projection features

The present invention provides an alternative use of certain aspects of the prism in a novel way. Instead of observing alone through this facet, the light pattern is projected through one or more facets of the prism device onto an object plane or surface.

For example, a dot beam or line light or grid can be projected onto the object surface via a stereo or three-phase beamsplitter device. These light patterns can be used for object analysis using an analyzer. For example, by projecting a grid light pattern from the top surface onto an uncharacterized and regular surface, the reflected light rays can be observed by all three sides at the same time for analysis. This feature is useful for objects that have an inflated portion, as shown in FIG. 19, for example, where there is no pronounced feature of a smooth, smooth curved surface.

If the light projection is coaxial in the light of the field of view of the characteristic and image formation system, the present invention may be integrated with an additional beamsplitter 24 for coaxial light projection as shown in FIG. 19.

Height Profiling Using Light Projection

30 shows a 3-D observation of a straight line projected from the line projector 23 onto a flat surface with a bump 31. For the sake of simplicity, the line projection in this embodiment is projected perpendicularly and directly to the top of the bump.

Since the light is not projected on the rest of the object and is dark, only the reflectance of the projected light, ie the hump of the bump 31 and two adjacent straight profiles, can be seen. Since there is no height deviation on the surface, the adjacent profile is straight.

When line projection is observed using the present invention with a three-beam image splitter 22 as shown in FIG. 31, three images are shown. The three images are planar images and left and right images. In the embodiment of the present invention, the size of the light projection beamsplitter 24 is made narrow and integrated on top of the three-phase beamsplitter 22 shown.

As the bump is indistinguishable from the planar image, the planar image represents a straight line. However, the two side images look at the bent height profile shown. The higher the bump, the shorter the CPD between the pairs of CPPs.

Thus, using the methodology according to the invention as described above, the complete height profile of the line projected onto the surface can be extracted. Thus, the height profile of the surface can be inferred.

The present invention also provides for projection of grids or patterns on multiple lines or surfaces or objects. Finding the corresponding CPP and calculating the CPD produces the measured 3-D data.

On the other hand, as shown in Figure 32, the device according to the invention can be attached to a system of motor drive of a single axis and move along the horizontal axis to provide a 3-D line profile measurement in an incremental step. Thus, a composite 3-D image can be formed by scanning across the entire area.

An application of the present invention is the investigation of the surface flatness of a given area. If the area is flat, there will be no predetermined deviation from the expected CPD of the projected line.

Height Profiling of Ball Grid Array Using Light Projection

FIG. 33A shows a top view of a Ball Grid Array (BGA) consisting of an array of solder balls on a flat carrier surface, and FIG. 33B shows a corresponding schematic perspective view.

Figure 34 shows the height profiling of the BGA according to the present invention incorporating line projection.

In the described embodiment of the present invention, the projected light beam and the powering system are used to scan across the surface of the BGA. As described above, by calculating CPP and CPD of several points along the horizontal or vertical axis, 3-D measurement of the height of each ball of the array can be performed as shown in FIG. By scanning across the surface of the BGA in an incremental step as shown in FIG. 32, the 3-D height profile of the BGA can be established.

In this invention, the horizontal resolution of the 3-D image depends on the granularity of the incremental step, i.e. the smaller the step, the higher the horizontal resolution. The vertical resolution is proportional to the amount of sampling point of the imaging sensor. Additional points are sampled, and the closer the placement of each point is, the higher the resolution.

Height resolution is determined by the magnification of the image. The larger the magnification, the greater the variation of the CPD for the same physical height. In addition, by increasing the difference angle (Fig. 6), it is possible to increase the variation and height resolution of the CPD. For example, increasing the difference angle from 45 degrees to 50 degrees will increase the variation and height resolution of the CPD.

Claims (18)

A light source 8 for illuminating the object to be observed, Means 11, 22, 26, 27 for generating a plurality of parallel bundles of parallel rays, each containing image information of the to-be-observed object 7 which is illuminated and separated from each other, Means (9, 17) for collecting and sending light rays of parallel bundles of said parallel rays to imaging means, Generating an image of a bulging bundle of parallel rays, the imaging means 2 being separated from each other in a two-dimensional imaging area 12 delineable by a set of two coordinate axes; The observed object from the coordinates of the point of incidence of the corresponding light ray, which occurs at the very same points h _{1 to} h ₅ of the observed object, but which is conducted from the object to the two-dimensional imaging area 12 in another bundle of parallel rays. (7) is provided with an analysis device (10) for extracting length, width and height information, And wherein each image corresponds to one particular parallel bundle of parallel rays. Illuminating the subject; Generating a plurality of parallel bundles of parallel rays each containing image information of the to-be-observed object 7 which are illuminated and separated from each other, Collecting light rays of parallel bundles of parallel light rays, Generating an image of a parallel bundle of parallel rays and separating the generated images from each other in a two-dimensional imaging area 12 delineable by a set of two coordinate axes; Extracting the length, width, and height information of the object to be observed from the coordinates of the point of incidence of the corresponding light beam that occurs at the very same point of the object but is conducted from the object to the two-dimensional imaging area in another bundle of parallel rays. With steps, A method for optically measuring three-dimensional data of an object to be observed, wherein each image corresponds to one particular parallel bundle of parallel rays. The two-phase beam splitter 11, the three-phase beam splitter 22, the four-phase beam splitter 26, or the five-phase beam as a means for generating a plurality of parallel bundles of parallel rays. Apparatus or method for optically measuring three-dimensional data of an object to be observed, characterized in that a splitter (27) is used. 4. The wedge-shaped prism of claim 3, wherein the two-phase beam splitter 11, the three-phase beam splitter 22, the four-phase beam splitter 26, or the five-phase beam splitter 27, respectively, is cut off. An apparatus or method for optically measuring three-dimensional data of an object to be observed, characterized by using a wedge shaped prism, a pyramidal prism, and a truncated pyramid prism. 5. The telecentric lens 9, a macro optical lens or a telecentric lens of various magnifications is used as a means for collecting light rays of parallel bundles of parallel rays. An apparatus or method for optically measuring three-dimensional data of an object to be observed. The individual pixel according to claim 1, wherein the individual pixel is sensitive to visible and / or infrared and / or ultraviolet and / or x-rays as an imaging means for generating an image of an incident parallel bundle of parallel rays. An apparatus or method for optically measuring three-dimensional data of an object to be observed, characterized in that an image sensor with elements is used. 7. Optically measuring three-dimensional data of an object to be inspected according to any one of claims 1 to 6, characterized in that the image sensor is a video camera (2), a line scan camera, a CCD camera or an x-ray imaging sensor. Apparatus and method for. 8. The three-dimensional data of the observation object according to claim 1, wherein a computer machine is used as the analysis device 10 for extracting length, width and height information of the observation object. Apparatus or method for measuring optically. 9. The means according to claim 1, wherein the means 9, 17 for generating a plurality of parallel bundles of parallel rays are provided to the imaging means 2 for generating an image of the incident parallel bundles of parallel rays. An apparatus or method for optically measuring three-dimensional data of an object to be observed, characterized in that it is rotatably installed. 10. A monochromatic light source 8, or a blue light source, a red light source and a green light source, according to any one of claims 1 to 9, for illuminating the object to be observed, is used. Apparatus or method for optically measuring three-dimensional data of an object to be observed. 11. The light source (8) according to any one of the preceding claims, wherein the light source (8) is integrated in a housing of the means (9) for collecting and sending light beams of parallel bundles of parallel beams to the imaging means, An apparatus or method for optically measuring three-dimensional data of an object to be observed, characterized in that it is arranged in a predetermined spatial relationship with respect to a means for generating parallel bundles of s. 12. The method according to any one of claims 1 to 11, in order to provide a calibration coordinate value of the two-dimensional imaging area 12 which serves as a standard dimension for an unknown observation dimension. Optically display the three-dimensional data of the object to be observed, characterized by disposing a reference object 13, 14, 15 of known dimensions at a predetermined distance from the means 11, 22, 26, 27 for generating parallel bundles. Method for measuring. The method of claim 12, wherein the reference object is a parallel hexagonal block-shaped object, a wedge shape, or a double wedge-shaped object. The method according to any one of claims 1 to 13, wherein the salient features of the reference object are analyzed and stored in the analysis device 10, and the data of the observed object are compared with the data obtained for the salient features of the reference object. A method for optically measuring three-dimensional data of an object to be observed. The method according to any one of claims 1 to 14, wherein the analysis device 10 determines the two-dimensional coordinate data of the incident point of the corresponding light beam in the other images of the plurality of images occurring at the same point of the observed object. And calculating the distance between the incident points (CPD). The method according to any one of claims 1 to 15, wherein the imaging means (2) moves along the observation object (7) to optically scan the three-dimensional data of the observation object. Method for measuring. 17. A property according to any one of the preceding claims, wherein the value determined for the dimension of the observed object (7) is compared with a set of predetermined dimensional values of the reference object and is determined in advance on the basis of the comparison result. And at least one attribute value of the values is assigned to the observed object. 18. The method according to claim 17, wherein the observation object (7) is further processed based on at least one attribute value previously assigned to the object.