WO2008124392A1 - 2d/3d inspection method and system - Google Patents

Abstract

A method and system for inspecting a Device Under Test include deriving a model of a Device Under Test, obtaining a 2D image of the Device Under Test, obtaining a 3D image of the Device Under Test, and comparing the 2D and 3D images to the model.

Description

2D/3D INSPECTION METHOD AND SYSTEM

Cross-Reference to Related Application

This application claims the benefit of U.S. Provisional Patent Application Serial No. 60/909,865, filed on April 3, 2007, which is incorporated by reference.

Background

Many manufacturing processes electronic for electronic components and assemblies include inspection and test procedures, which can be either manual or automated. For example, the surface mount assembly process (SMT) consists fundamentally of three value added process steps: Solder paste printing, component mounting and reflow. These are schematically illustrated in Figure 4. Un-stack bare board 30 removes a single bare circuit board from a stack of them and inserts it into the assembly line. Solder paste print 32 prints solder paste onto the bare circuit board. Component mount 34 moves components from a component bulk feed apparatus (not shown) and places them onto the circuit board. Reflow oven 36 melts the solder paste and then allows it to cool and re-solidify. Stack populated board 38 takes at least partially assembled circuit boards and stacks them into a portable batch.

There are many extensions to the above, including inspection and test strategies, flipping the circuit boards so that components can be mounted onto each side, accommodation for through-hole components, glue deposition for those components that need to be held during assembly, odd- form assembly, under fill of flip-chip components, etc. But at its most basic level, the above three steps describe the SMT line.

Figure 1 is a simplified schematic side view of a small section of a bare circuit board. The coordinate system 20 shows a Z and X direction, where Z is the direction of height perpendicular to the substrate 10 surface, X (Y could be substituted for X with no consequence to this discussion) is one of two lateral directions parallel to the substrate 10 surface. The substrate 10 is typically formed from a rigid fiberglass called FR4, but other materials are sometimes used, including flexible materials.

The substrate has formed onto its surface several important features. The electrically conducting material 12 on the surface of the substrate is most often copper. Solder mask 11 is often deposited everywhere that no electrical contact is intended, including over the electrical conductive material as illustrated at 13. Because copper readily oxidizes and can then become less conducive to forming good electro-mechanical solder joints, most often an additional material (not shown) designed to prevent oxidation is deposited on top of the copper. The thickness of the solder mask as it is situated on top of the substrate is illustrated at 14. The thickness of the conductive material (and if it is present, the additional material used to prevent oxidation) is illustrated at 15. A typical range of values for thickness 14 is 10-20μm. A typical range of values for thickness 15 is 10-20μm. Figure 2 is a simplified schematic side view of the circuit board after solder paste printing has occurred. Here, two distinct solder paste deposits are illustrated by the aggregates of micro solder balls at 17. The height of the solder paste deposits 16 is typically 100-200μm.

Figure 3 is a simplified schematic side view of the circuit board after a component 18 has been mounted. The height of the component can be 0.5mm to 25mm 19. The final step of electronic assemble is solder reflow, where the solder paste 17 is melted and then cooled to form solder joints (not illustrated).

While the SMT assembly process has been refined over the last two decades, SMT assembly lines continue to make a variety of errors that lead to defective assemblies. These errors are herein grouped into classes: One class of errors is related to solder paste printing and includes (non-exhaustive list):

Incorrect height, volume or area of each individual solder paste deposit; and Incorrect position; Creation of a solder bridge (paste connecting two nominally separate deposits).

Another class of errors is related to components and includes (non-exhaustive list): Missing components (one or more not located where they should be); Extra components (one or more located where they shouldn't be); and Incorrect components (wrong component type or value); Incorrectly positioned components (linear or rotational errors).

Another class of errors comes from the reflow oven (non-exhaustive list): Incorrect temperatures; Incorrect dwell time in temperature zones; and Uneven heating.

Yet another class of error comes from raw materials imperfections (non-exhaustive list): Component lead oxidation; Component lead non-coplanarity; Panel (circuit board) surface contamination; Panel warp; Panel stretch (relative to the solder paste stencil); Stencil stretch

(relative to the panel); Stencil apertures incorrectly sized, shaped, positioned; and Insufficient solvents in the paste.

Yet another class of error comes from design imperfections (non-exhaustive list): Components are placed too close together or otherwise positioned so that, when the panel is in the oven, uneven heating causes solder paste to melt in a non-uniform way; and Copper pads and/or solder mask are incorrectly sized/positioned causing incorrect solder wicking. Most of these errors have some visible manifestation when viewed after solder reflow. These manifestations include (non-exhaustive list): Missing components (compared to the design intent); Extra components (compared to the design intent); Wrong component (as compared to the design intent); Wrong value (the resistance or capacitance of a component is not correct, even though the package is right. This is a subset of wrong component); Tombstone (a two lead component flipped up on end so that it is making contact only on one lead); Billboard (a two lead component flipped up on its side so that it is making contact on both leads, but not with a correct solder fillet); Dead Bug (a component flipped upside down "feet up"); Wrong polarity (a device whose orientation must be controlled for correct electrical behavior but is oriented incorrectly); Bad solder joint (one or more solder joints is improperly formed. Solder joints that are located under the body of a component would not normally be visible but can be observed via x-ray inspection); Lifted lead (not all the leads of a component are soldered well to the panel. Caused by component coplanarity errors, panel warp or both); and Solder bridge (two leads that should be electrically isolated have electrical continuity) Defects can result in electronic assemblies that do not work correctly while they are still in the factory. It is possible to catch most of these before the assembly is shipped by electrically testing the completed assembly, depending on the thoroughness of the test. Thorough electrical testing is often quite difficult and time consuming, especially for more complex electrical assemblies. For some devices, thorough electrical test is so difficult as to be considered impractical.

Sometimes, assemblies will work properly when electrically tested at the factory and then fail after only a short time in the field. These failures are often caused by visually evident errors. For example, a partially formed solder joint will provide good electrical contact but possibly tenuous mechanical contact. Thermally or mechanically induced physical stresses can cause this solder joint to crack leading to premature field failure. However, improperly formed solder joints can be visually detected if they are not hidden from view (e.g. by a component body or a radio frequency (RF) shield, or the like).

For the above reason, electrical test is generally understood to be an incomplete quality control approach. To supplement electrical test, SMT operators nearly always implement some sort of visual inspection approach at the end of the assembly line (after reflow). One type of inspection is by human visual. Another, often more suitable approach, is in-line AOI (Automatic Optical Inspection) and sometimes X-Ray (automatic or manual) inspection. Here, an in-line inspection machine is defined to mean a machine that is able to receive a circuit panel from an upstream machine through a first port, perform operations on that panel, and then pass the panel out through a second port to a downstream machine, where the first and second ports are physically not coincident. An off-line machine is defined to mean a machine that is able to receive a circuit panel through a first port, ftorn a machine or from a human, perform operations on that panel, and then must pass the panel out through the same first port.

Training is herein defined to be the process by which solder paste inspection (SPI) or AOI machines are programmed so that they are able to perform their inspection tasks. AOI machines are notoriously difficult to Train. For AOI machines, in the best case, Training can take ten minutes and in the worst case a day or more. AOI machines are prone to False Calls, which occur when any inspection machine determines that an error has occurred when, in fact, it has not. They are also prone to False Accepts, which occur when any inspection machine determines that an error has not occurred when, in fact, one has. AOI machines are able, at the expense of extensive and expert programming, to suppress the False Call rate to low levels (~50 parts per million (PPM)) for short periods of time while maintaining an acceptably low (D5PPM) False Accept rate. But to keep the false call rate at these low levels requires ongoing programming efforts to accommodate raw materials variation. This variation changes the appearance of assemblies, sometimes substantially, yet the assembly may still be considered "good" in that it will exhibit proper electro-mechanical behavior.

AOI Devices that are able to be Trained in only a few minutes rely on Training by single example. They have very high False Call rates and their function is often more aptly referred to as machine assisted human inspection rather than automatic inspection. This is true because an operator must be constantly present at such a machine to promptly overrule it when it generates a false call, which might happen ten times or more on a single panel of only moderate complexity.

Some SPI inspection systems are able to be Trained in only a few minutes and rely on board level Computer Aided Design (CAD) data based Training. Many SMT operators of moderate to substantial size maintain their own, custom data bases. Some SPI machines support custom interfaces to these custom databases in order to reduce Training time to the few minute range. However, as there are many such custom databases, it is expensive to develop and support custom interfaces to a majority of them. 2D SPI machines are prone to False Call problems, but somewhat less so than are AOI machines. 3D SPI machines are even less prone to false calls than are 2D SPI machines. This board level CAD is a description of how the bare circuit board or panel or solder stencil is constructed and is not a description of how the final populated panel should appear.

SPI inspection machines that Train from board level CAD data, an example of which is called stencil Gerber data, take approximately one hour to Train and require a medium level of expertise.

Because of the above, board level CAD based Training works well for SPI inspection devices. However, there are some inspection functions, mostly relevant to AOI, which do not lend themselves well (or at all) to Training from board level CAD. Not (or only partially) present in such data are descriptions of (non-exhaustive list): Color and brightness changes of raw materials (components, bare boards, etc.); Font changes of text labels; Orientation or location changes of text labels; Movement of silk screen (within limits); Position changes of components (within limits); Component body dimensions (in all three dimensions); Appearance of polarity marks; Appearance of fiducial marks; and Appearance and dimensions of solder joints.

The above is a non-exhaustive list of Attributes of Appearance of the circuit assembly. Board level CAD, at best, only partially or approximately describes these Attributes of

Appearance. Silk screen size and position and the sizes and positions of components are two

Attributes of Appearance that are described approximately. Depending on the data set, the others may not be described at all.

However, component body dimensions are, or become, known to the modern-day mounter before, or as, it places components onto the circuit board. A-priori knowledge of the component X and Y dimensions allows the mounter's internal vision system to use optimal parameters to align the component. Some optical systems used for in-mounter component alignment do not require a-priori knowledge of the X and Y dimensions of the component, but these optical systems measure those two parameters as a byproduct of the alignment operation. In either case, the X and Y dimensions of all components are known at least by the time the component is placed by the mounter onto the circuit board.

The Z dimension of the component must be known before the mounter can safely pick up the component from the bulk feed mechanism. Without this knowledge, the vacuum nozzle used to pick up the component would either not descend far enough and would fail to pick up the component properly, or conversely, descend too far and smash into the top of the component. Mounters typically implement laser range finders or the like to aid the SMT operator in learning the Z dimension of components. Importantly, this Z dimension may not be well controlled from one component vendor to the next, even though the components are supposed to be interchangeable. Therefore, even if a-priori data of the component Z dimension were known, the mounter would preferably learn the actual Z value of the components that were physically present.

Another example of data that must be known to a mounter is the lighting configuration required for fiducial finding. Fiducials are marks located on the circuit panel specifically to allow the mounter to find their XY positions. Once these fiducial mark positions are known to the mounter, in some coordinate frame also known to the mounter, the mounter can calculate the XY coordinates at which to place the components. Fiducials have a variety of optical finishes, some shiny, others dull. The lighting configuration to optimally illuminate these fiducials is determined by operator interaction with the mounter's lighting system. The operator will adjust the mix of specular (vertical, sometimes coaxial with the camera, light source) with the diffuse (lower angle lighting) sources so as to achieve a fully lit fiducial suitable for vision processing.

Thus, some data are available on the SMT line without any additional work beyond that already required to build circuit boards. One type is board level CAD data and is derived from the circuit board design. Another type are that data which becomes known to the mounter through some user interaction with it or as a byproduct of the mounter's activities related to its job of placing components. Still more types are data available to other machines on the SMT line including the circuit board ID (often read via a barcode), or its sequence number, or the status of its "block-sldp-marks", or some (often partial) inspection results performed, for example, by solder paste inspection apparatus within a solder paste printer.

Such data, required by or available to other machines on the SMT line are, together, herein called Machine Readable data. Because Machine Readable data are required before the SMT line can operate or becomes available as a byproduct of such operation, no work is required by the line operator or engineer to make this data available to an inspection device other than the act of transporting those data to the inspection device. Of course, this transport can be electronic and automated eliminating the burden on the user entirely.

In summary, Machine Readable data describe completely, or nearly completely, the 3D physical structure of the assembly. Conversely, Machine Readable data describes the 2D appearance of the assembly, at best, only partially. Training by example has been available in a number of extant 2D inspection devices. The user typically delineates what is good from bad based on his comprehensive knowledge of the assembly. An example of this is the appearance of a solder joint, which an experienced user can visually determine is good or not, but which is not described in any extant data set. However, such Training by example suffers from a major difficulty: The 2D appearance of assemblies can vary from board to board or from lot to lot. Such appearance variations are caused by variations in the Attributes of Appearance, described above and may, or may not, be indicative of an assembly error. Inspection systems have been deployed that can accommodate these sorts of 2D appearance changes based upon multiple-example based Training. In this paradigm, the SMT operator is required to Train the inspection system using a number, typically between ten and one hundred, of known-good (Golden) examples. In many, but not all, cases, the variations described above will be observed by the inspection system and, since they are declared Golden by the user, the system can learn that such variation is not indicative of a faulty assembly. Other variation, outside that which has be taught as acceptable, is then flagged for user review as a putative assembly error.

The drawbacks of this approach are (non-exhaustive list):

^■ The user must actually know that the examples are good. If, inadvertently, a bad example is added to the Training set, then other assemblies of the same bad nature will be accepted as good. Of course this means that a user who can distinguish good assemblies from bad is required to review the assembly. This expertise is not always available.

^■ Many Golden examples are required. If the SMT operator is building fewer panels than are required by this example set, then this approach is a non-starter. Even if the number of panels is somewhat larger than the required Training set, this still presents a major problem to the SMT operator: The time, effort and expertise required to Train the inspection system is not worth it when only a few assemblies will be scanned by a well- taught system.

■ The expected range of good variation is required in the Training data set. If this isn't the case, when a new variation appears, the system will flag it as defective. An expert user will then be required to review this result and confirm the defect, or else add it as another Golden example to expand the knowledge of the inspection system. This results in the need for constant updating of the Training data set by a not-always-available expert.

^■ The system, when Trained in this way, generates errors based on detected novel differences between the Trained good example set and everything else. Thus it is not able to sort errors into categories, for example, into tombstone, billboard or component- missing error categories without additional training using examples of specific errors.

Such training is very difficult to perform because it is not practical to deliberately generate numerous examples of all categories. Of course, while the assembly line is running, errors will occur and the resulting imagery can be used for Training. But such Training then becomes an on-going process requiring the presence of an expert. An additional difficulty that arises when Training from example comes from the need to describe Regions of Interest (ROI). Board CAD data often has some information that will allow a system to know where to look in an image for certain features. An example of this is Gerber data for a circuit board or a screen stencil that can be used to learn where solder paste is intended to be printed. This information can be used by an SPI machine to learn where to look for solder paste or by an AOI machine to learn where to look for a solder joint.

Without board level CAD data to program the inspection device, SMT operators today are required to manually draw boxes that describe ROIs. This can require a lot of time from an expert and is particularly onerous. Training of an inspection system by Golden example can be divided into two logically separate steps. The first allows the system to partition the image into different target types and is referred to in the image processing industry as Image Segmentation. An example of this is distinguishing image segments of solder paste from image segments of something else. The second is to establish upper and lower thresholds or tolerances that will be later used to detect and categorize errors.

Traditional 2D inspection devices rely on a variety of Train-by-Golden-example methods to characterize the contents of (sometimes board CAD defined) ROIs. A common way to do this is to Train from a single Golden example to establish a correlation kernel of an ROI. This kernel is used later to score image segments acquired of the device under test (DUT). Normalized correlation scores above a certain (expert user defined) threshold are deemed to be good. As already mentioned, this scoring approach has the distinct drawback of being prone to False Calls caused by 2D visual variations. It is a one dimensional approach in the sense that a single score value, in this example the normalized correlation coefficient, is being used to establish whether the assembly is good or not.

Expanding this approach to multiple mathematical (not physical) dimensions from preferably independent measures derived from the 2D image of the circuit board is also known and deployed in 2D AOI systems, but as described above, requires Training by numerous examples. Another comparatively simple and robust way to perform Image Segmentation, relies on the concept of difference processing wherein a single instance of a circuit board is observed by an optical inspection system before and after a value add operation is performed by the SMT line. For example, the bare circuit board 10 of Figure 1 does not yet have any solder paste on it. If the circuit board, while it is in this condition, is observed by an SMT inspection machine and later, after solder paste 17 of Figure 2 is added, again observed by an SMT inspection machine that has access to the previously acquired image, in principle areas with where the two images differ can only be caused by the presence of solder paste 17.

Similarly, component 10 of Figure 3 can be segmented from their surroundings by difference processing applied to images acquired before component mounting and after. This sort of difference image processing is well known in the public domain, but is not deployed as a Training or run time methodology in existing SMT inspection systems at least in part because no one machine is deployed to easily implement both SPI and AOI. Additionally, the optical characteristics of systems optimized for one or the other purpose are dissimilar. This dissimilarity itself can cause difference image processing to become extraordinarily complicated. For example differences in illumination, magnification, distortion, color sensitivity, etc. will cause image differences to appear where, in fact, the object being imaged may not have changed in those regards. Training a system to look for expected features within an ROI facilitates one goal:

Confirmation (or not) of what is expected within the ROI. However, inspection devices would preferably also be able to determine that what is not expected elsewhere is, in fact, not there. For example, Training (however it is done) positions ROIs over portions of an image where components on a circuit board are intended to be situated. As a result of this Training, AOI machines are able to detect that the component is missing from the ROI; from its intended position. But this leaves open the question: Where did the missing component go? Existing AOI systems are unable to locate this component because:

^■ In the interest of throughput, inspection only occurs within defined ROIs which often cover only a fraction of the entire circuit board. Throughput (speed of inspection, including image acquisition and image processing times) is a significant factor for inspection machines, especially when they are situated in-line. Restricting areas that need to be acquired and processed to ROIs enables inspection machines to run faster.

■ No extant Training paradigm teaches the inspection system the expected appearance of regions outside of ROIs. ■ CAD data does not describe the appearance of regions outside of ROIs.

No extant AOI system performs this sort of negative confirmation inspection. A similar line of reasoning applies to SPI; extra solder paste located in places where it is not supposed to be will likely cause assembly defects. Yet, with the notable exception of detection of solder bridges (extra solder located between intended print locations), no extant SPI system looks for extra solder paste.

Some extant inspection machines are able to use a mix of Train-by-example and board CAD data. But in all cases, the inspections that are predicated on Train-by-example either have a high false call rate, or require a broad set of examples that encompass the (nearly) entire range of variation in appearance. For these and other reasons, there is a need for the present invention. Summary

A method and system for inspecting a Device Under Test include deriving a model of a Device Under Test, obtaining a 2D image of the Device Under Test, obtaining a 3D image of the Device Under Test, and comparing the 2D and 3D images to the model. In exemplary embodiments, the model is a 2D/3D model, and can be derived from machine readable data and/or 2D and 3D images of an example of the Device Under Test. The system is typically implemented using an appropriately programmed computer having access to a memory storing the necessary software code, the Device Under Test models, etc.

Brief Description of the Drawings

Embodiments of the invention are better understood with reference to the following drawings. The elements of the drawings are not necessarily to scale relative to each other. Like reference numerals designate corresponding similar parts.

Figure 1 is a simplified schematic side view of a portion of a bare circuit board. Figure 2 is a simplified schematic side view of the circuit board shown in Figure 1 after solder paste printing has occurred.

Figure 3 is a simplified schematic side view of the circuit board shown in Figures 1 and 2 after a component has been mounted.

Figure 4 schematically illustrates portions of an SMT assembly process. Figure 5 schematically illustrates portions of an SMT assembly process in conjunction with an inspection machine.

Figure 6 is a histogram of solder paste heights.

Figure 7 schematically illustrates a Tombstone component.

Figure 8 is a data flow diagram of the use of Machine Readable data to allow for detection of Zero Training Defects.

Figure 9 is a data flow diagram of the use of 2D and 3D images from example assemblies to form 2D and 3D models to allow for detection of defects on DUTs.

Figure 10 is a data flow diagram of the use of 2D and 3D images from example assemblies and Machine Readable data to form 2D and 3D models to allow for detection of defects on DUTs.

Figure 11 is a data flow diagram of a Learning/Running phase that constructs 2D models of good assemblies suitable for use during a Running phase. Figure 12 is a data flow diagram of a Learning/Running phase that constructs 2D models of bad assemblies suitable for use during a Running phase.

Figure 13 is a data flow diagram of a Running phase that makes use of the good and bad 2D models constructed during the Learning/Running phase. Figure 14 is a high resolution 2D image of a portion of a circuit board.

Figure 15 is a 3D image of the same region of the circuit board as Figure 14.

Detailed Description hi the following Detailed Description, reference is made to the accompanying drawings, which form a part hereof, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. In this regard, directional terminology, such as "top," "bottom," "front," "back," "leading," "trailing," etc., is used with reference to the orientation of the Figure(s) being described. Because components of embodiments of the present invention can be positioned in a number of different orientations, the directional terminology is used for purposes of illustration and is in no way limiting. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present invention. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.

In accordance with aspects of the present invention, an inspection system is disclosed that is able to generate high fidelity 3D maps of a populated or unpopulated Device Under Test (DUT), such as a circuit board, and acquire high resolution 2D color imagery of a populated or unpopulated circuit board. The inspection machine can be Trained by a human having reduced expertise, from Machine Readable data and/or Golden example(s).

Figure 8 is a flow diagram illustrating aspects of an inspection method in accordance with embodiments of the invention. The method is typically implemented in an inspection system coupled to and controlled by a computer system that includes necessary processing devices, memory, input/output devices, peripheral devices, etc. Referring to Figure 8, Machine Readable Data 82 containing 3D information about the assembly can be used by a model generator 84 to construct nominal 3D models of the assembly which are stored in memory 88 of the system. To the extent 2D Attributes of Appearance are specified in the Machine Readable data, this can be used to generate nominal 2D models also stored in memory 88. However, as discussed previously, most Machine Readable data does not contain complete information about Attributes of Appearance and therefore, the 2D nominal models will not be complete enough to allow for reliable inspection based on 2D data alone. Thus, an image acquisition system 81 obtains 2D and 3D images 86 of the DUT, which are compared using inspection algorithm 90 to the models 88 to obtain an inspection result

One key advantage obtained from performing AOI in 3D is that, with the use of Machine Readable data, no manual Training and no example (either Golden or not) is required before a system based on this approach can detect defects such as those in the following defect categories (partial list): Missing components; Improper XY component position; Tombstone components; Billboard components; and Extra components located where they are not supposed to be. These defect categories are herein referred to as Zero Training Defects. Detecting Zero Training Defects and determining their category is straightforward even with no Training, e.g.: Missing components will have 3D height map regions that are substantially coplanar with the surrounding substrate. Referring to Figure 7, tombstone components 18 (those up on end as shown), will have a different height profile than billboard components (those up on their side (not illustrated)) which in turn will have a different height profile from a component that is properly placed (components are very rarely cubes). Categorizing errors becomes possible because the disclosed system is not sensitive only to appearance differences, but is in fact making measurements of physical dimensions: An error has occurred when these measurements fall outside certain limits indicated by the Machine Readable data; when these measurements fall outside limits in certain ways the error can be categorized. Similarly, areas where no component is expected can be easily examined for extra or stray objects that will have height. This is a strong signal, using only 3D models generated from Machine Readable data, that a component is located where it is not supposed to be.

Figure 15 is an example 3D image (a height map) of a region of a circuit board generated with a system that operates in the manner described in the incorporated co-pending application entitled "INSPECTION SYSTEM AND METHOD". In this 3D image, height is encoded as brightness; taller objects are brighter while shorter ones are darker. It is easy to see that the colors of portions of the image are suppressed in this height map. The substrate (in this case the circuit board) 100 is of substantially uniform height although, on close inspection, it is possible to see small height changes where there are circuit traces 101 and silk screen lettering 102. These two objects do change the height, but only slightly compared to the components 103 through 108, which are evidently very tall (and therefore bright in Figure 15) in comparison. Referring to Figure 14, the same circuit traces 101 and silk screen lettering 102 are very clearly evident. Their appearance, in the 2D image of Figure 14, is not any less clear or less apparent than the components 103 through 108. Systems reliant entirely on 2D images like the one in Figure 14 must be able to learn the relative importance of variations in silk screen appearance (for example) from the variations in the components' appearance.

Referring again to Figure 15, it is now easy to see how such a 3D height map, coupled with the nominal descriptions of the components' sizes and positions, can allow the system of this invention to detect zero training defects without the benefit of any example assemblies: The components' presence or absence is clearly evident because the color and brightness changes inherent to the 2D color image of the DUT are suppressed, while the dimensional aspects of the DUT are revealed. Note that the height map of Figure 15 reveals that component 108 is taller (brighter) than all the other components. Note also that the leads 109 of component 103 are getting darker as they leave the component 103 body and descend toward the substrate 100.

Figure 9 illustrates an embodiment where 2D and 3D images of an example DUT 83 are used to generate the models 88. Thus, with minimal Training from a single Golden example, detection of the following errors that evidence themselves in 3D can additionally be implemented (partial list): Lifted leads; Solder bridges; and some polarity errors.

At run-time, the height maps in the neighborhood of the above targets can be compared to those previously recorded from a Golden example. For example, a solder bridge error seen in the run-time height map will have heights different from those of the corresponding region in the Golden example. Some polarity errors are detectable because, for some components, the polarity mark is a

3D mark such as a divot or a bevel. These marks are sometimes very difficult to see in 2D, but are large compared to the 3D resolution envisioned herein. Again, the location of the 3D polarity mark would be compared to that learned from a single Golden example.

It is also envisioned that all the above mentioned measurements, error detection and categorization can be made even if no Machine Readable data is available by using one or more Golden examples to generate 2D and 3D models as illustrated in Figure 9. When only one Golden example is available, data derived from the corresponding height map and/or 2D imagery is used to form a set of nominal values, e.g. for each component's XY position, height, luminance and chrominance. When multiple Golden examples are available, they can be used to establish normally occurring variations, such as the allowable range of XY positions or heights. The nominal value and normally occurring variations are then used for a basis of comparison against the panels or substrates of unknown quality. As mentioned, Training of an inspection system by golden example requires the first step of Image Segmentation and the second step of application of thresholds or tolerances.

An aspect of the disclosed system and method relies upon the use of a mix of Machine Readable data and color 2D and 3D data acquired from a single Training example (Figure 10) to establish a multiple dimensional approach to image segmentation and Training in general that does not require multiple examples.

SMT assemblies consist of components placed onto (locally) flat substrates. The most common substrate (fiberglass FR4) item 10 of Figure 1 is rigid and nominally flat over large lateral extent. Others are flexible (polyimide) but are still flat (or at least during the assembly process must be held flat) over the lateral extent of the component lateral dimension. In any case, the component's height is large compared to the allowed substrate height variation over a lateral extent the size of the component. Of course, the population of components on a panel will have varying heights, but the shortest among them is still tall compared to the allowed substrate height variation according to the above rule. It is for this reason that it is possible and even easy to detect and categorize Zero Training Defects once a 3D height map of the populated circuit board is available.

Other optically evident features on a circuit board have nominal heights as well. These include silk screen (typically 25-50um), solder paste (typically 100-200um), solder joints

(different height profiles depending on the component contact type), bare copper or aluminum (this would be the ideal zero height reference), bare substrate, solder mask covered substrate and solder mask covered copper or aluminum.

It is possible to reliably group each of the above features based upon the following Independent Characteristics (non-exhaustive list):

^■ Height (3D) ■ Height Signal Strength or Vector Length: A by-product 3D height computation

^■ Brightness (2D luminance from a 2D camera)

■ Color (2D color from a 2D color camera)

■ Texture (computed for example, by a well known filtering approach, such as the Law's Texture Filter) ■ Lateral extent (2D and/or 3D)

Using Machine Readable data and only a single example (Figure 10) it is possible to generate the necessary Training information. The example need not be known-good in order for the segmentation portion of Training to be successful. The required Machine Readable data consists of:

^■ The intended placement location of the component (contained within a "Centroid File") that mounters must have in order to place components

^■ The XY and Z sizes of each of the components that are placed. ■ The fiducial illumination parameters

In this aspect of the invention, the single example circuit board is scanned by the inspection device. To Train for solder paste inspection for example, the user will indicate a segment of the image that has solder paste (using a mouse for example) and tell the inspection device that he is so indicating. Similarly, he will indicate examples of some or all of the various features on the circuit board. The Independent Characteristics will then be evaluated for each feature. Feedback to the user is immediate in that the image of the circuit board can be colorized or otherwise highlighted to indicate the results of Image Segmentation. For example, once the set of Independent Characteristics for solder paste is established, all solder paste in the image can be highlighted, regardless of its location. This Training can be machine assisted in a variety of ways:

■ The image content can be pre-analyzed into different regions of the multi-dimensional space defined by the Independent Characteristics. Pixels with a multi-dimensional hyper- volume can be highlighted and the user can the select from a list that would declare what that volume's "name" should be; e.g. "solder paste". The hyper volume corresponding to a single feature, such as solder paste, need not be contiguous.

■ The image content can also be analyzed so as to adjust filter parameters or filter structures so as to better separate hyper- volumes belonging to different feature types.

^■ Knowledge of previously Trained names and their hyper- volume can be used as a guide for current and future assemblies; e.g. if solder paste does not vary much in its appearance at one SMT facility, then it would be redundant for the user to perform this process for each circuit board assembly type.

^■ Machine Readable data related to the component dimensions and locations can be used to create 3D ROIs for those components.

As mentioned above, the example need not be Golden to allow for successful image segmentation; If the solder is printed incorrectly in some places, this is OK because it will still have a signature of solder paste in one or more of the Independent Characteristics.

However, multiple Golden examples are useful, although not required, for establishing optimal parametric thresholds of acceptability (tolerances). The multiple Golden examples may exist on a single Golden circuit board, or not. For example, if the circuit board has numerous identical components on it (and this is quite common for some part types), then the solder paste print, the solder joint, the package dimensions, etc. and the Independent Characteristics thereof are nominally identical. Additionally, even if there is only one component of a certain type on a single Golden circuit board, that part may have many solder contacts, possibly hundreds. The solder paste print for each of these contacts is almost always nominally identical, so it would be possible using that population of solder paste contacts to establish tolerances. The same argument applies to that component's solder joints. Under these conditions it is possible to gather the statistics useful in to establish optimal tolerance thresholds for those components from the numerous examples present on the single circuit board.

One way to use these statistics to establish tolerances is by using histograms. For example, it is possible to establish tolerances for the heights of solder paste deposits on the circuit board from a single Golden example by creating a histogram of those heights, as illustrated in Figure 6.

Figure 6 is a histogram 50 of solder paste heights and shows two distinct height populations 51 and 52, which would be the case when a step stencil, a stencil with two nominal thicknesses, is used. Such stencils are not typical, but are also not uncommon. Solder paste inspection systems used to qualify boards printed with two or more stencil thicknesses must at least be able to accommodate two separate height tolerance sets or thresholds.

The left distribution 51 has a mean height of lOOμm and a standard deviation of lOμm. The right distribution 51 has a mean height of 150μm and a standard deviation of 5μm. When the example is known Golden, it is reasonable to set thresholds so that the probability of examples in the present set failing is very low to nil. For example, for population 51, upper threshold 54 and lower threshold 53 could be set ±30μm (three times the standard deviation) from the mean. Under these conditions, the chance of there an error (solder paste deposit with height outside the threshold or tolerance limits established by 53 and 54) in the Golden population of solder paste deposits corresponding to population 51 would be about 0.2%. Alternatively, the threshold might simply be set to ensure that no paste deposits on the

Golden example are defective, regardless of their statistics. Or a margin can be added to such thresholds so that other examples, that won't be Golden, have an elevated probability of passing inspection. A second set of thresholds can also be automatically generated for population 52 according to one or another of the above methods.

The user can adjust the thresholds however they are established via a user interface by entering new values for them, using a pointing device to move the threshold lines, such as 53 or 54, on the histogram, or the like.

Real time results are presented to the user as he modifies the thresholds, hi the example of solder paste heights, solder deposits falling outside the thresholds can be highlighted, for example using color. As the thresholds are modified according to the above, the number of so highlighted solder paste deposits will change providing real-time feedback to the user. In the absence of multiple examples required to establish statistics useful in setting tolerance thresholds, it is still possible to establish the nominal value and use operator supplied thresholds. For example, the amount of solder paste on a single example from the Golden circuit board can be used to establish the nominal value. The operator can then enter an allowed percentage change to establish the tolerances. When a single circuit board does not contain a sufficient number of examples to gather the statistics required to establish tolerance thresholds from statistics, multiple Golden circuit boards will be needed. Or the user may supply his application specific knowledge to enter thresholds.

Regardless of the circuit board design, the number of Golden examples required to establish tolerances can be small (compared to 2D AOI Trained by multiple examples) because the Independent Characteristics available to a 3D AOI machine are robust against appearance changes. For example, once component height is known from Machine Readable data, determining if a component is present or not need not depend on component color or brightness (which can vary in ways not necessarily indicative of assembly defects). Solder paste deposits on a single Golden example can be categorized by their dimensional extents. Histograms of deposits that are observed to be nearly identical can be used to compute Golden variances from which error thresholds can be computed. This categorization process can be assisted from the Machine Readable data by associating those solder paste deposits for part of a given size into a given category. Once a single Golden example is used for Training, negative confirmation inspection can be easily achieved against the most commonly occurring "extra" causes. These are: Stray components landing in unintended locations; and Solder paste where there should not be any. The first can be done, for example, by inspecting everywhere that no component-sized height is found in the Golden example. As discussed, the Independent Characteristic of height is a strong signal of the presence or absence of a component. In fact, SMT operators perform a manual inspection technique that is, in principle, very similar to this: A custom Mylar sheet is made with apertures located to conform to expected locations of components. The apertures are sized and positioned so as to just enclose each component. The operator positions the sheet over the populated circuit board and attempts to lower it flush onto the circuit board's surface. If he is able to lower the sheet onto the surface (and it is easily visually evident if this is achieved) then there are no extra components on the panel, which, if they were present, would hold the Mylar sheet up. Additionally, the optical uniformity of the Mylar sheet assists the user to visually confirm that all the expected components are present; it is clearly evident if any of the apertures are empty.

It is possible for additional Independent Characteristics (in addition to height) to be used to make the detection of a stray component even more reliable; the set of Independent Characteristics that define a region where no component is expected can be known by virtue of the Golden example. Height (or the lack of it in this case) is one of them, but color, texture, or any other Independent Characteristic or mix of characteristics can also be used.

The above discussion has been constructed to illustrate how to determine that components have not landed where they shouldn't. The same principles can be applied to detecting that solder paste has not been located in "extra" places. One or more hyper-volumes of Independent

Characteristics that define the (typically the majority) of the circuit board where paste should not be located is established from the Golden example. For example, almost always, solder mask covered substrate is located between adjacent intended solder paste deposit locations. Solder paste will have quite different Independent Characteristics than solder mask covered substrate; it will have strong texture, it will be brighter, it will be silver in color (rather than the typically green or, less frequently, red color of solder mask).

For the above example, the metaphor of the "Mylar sheet" lowered onto the panel to determine whether extra solder paste is present isn't, in this case, sensitive to height. It can't be because extra solder paste can be of varying, even almost zero height. It is, instead a "sheet" of a composite score or hyper-volume based on multiple Independent Characteristics that most strongly separate or segment solder paste from solder mask covered substrate.

Another way to Train the inspection system from a Golden example uses difference image processing. For an inspection machine able to perform both SPI and AOI, it is natural for substantially the same optics and illumination to be employed to observe the circuit board as it progresses through various assembly steps, thereby overcoming the difficulty of differences caused by different optics, illumination, etc. Difference image processing still presents some challenges, because the target object (the circuit board in various stages of assembly) may not be in exactly the same X, Y, Z or rotation about Z coordinates when the two images are acquired. This problem is also solved in the public domain, for example, by the use of cross correlation. The peak in the normalized corelogram is an estimate of the differences in these four axes. Image processing can be used to resample either image so as to make it appear to have been acquired at the same coordinates as the other. This re-sampling mechanism is also public domain. hi this aspect of the invention, illustrated in Figure 5, an operator physically transports the circuit board to the off-line inspection machine 40 along path 42. After the inspection machine 40 gathers data suitable for forming 2D color images and 3D height maps, the operator returns the circuit board back to its previous position on the assembly line (reversing path 42). Similarly, the operator transports the same circuit board after each value add operation to and from the inspection machine 40 following paths 44, 46 and 48. Of course, it is not necessary for the inspection machine to scan the circuit board before and after every stage of assembly. For example, difference processing will still be possible for component AOI even if paths 42 (or 44) and 48 were the only ones used, hi another example, difference processing will be possible for SPI using only paths 42 and 44. The above discussion is not restricted to Training of off-line inspection machines, but can be used to train an in-line inspection machine as well, provided: a) That a circuit board in a earlier state of assembly can be physically delivered to that inspection machine in order to allow for acquisition of the 'before' image or; b) That a reference image can be delivered to the in-line inspection machine from another image acquisition apparatus typically situated upstream. As noted earlier, if the reference image is acquired by such an approach, the optics, lighting and mechanics will vary to some degree between the two systems (even if they are of the exactly the same design intent). Correction for these variations may be required to allow the system to respond to differences that are physically present on the circuit board only. Ways of characterizing and compensating for these variations are known. However, if the variations are large, then this task can become quite difficult, especially if it is necessary to detect true differences on the circuit board that are small compared to the optical/mechanical variations.

Thus, data suitable for generating 2D color images and 3D height maps of the circuit board are acquired and saved before and after it passes through one or more value add operation on the SMT line from its bare state to its completed state. The difference processing methods described would preferably be applied to all images or derivatives thereof to achieve the most reliable Image Segmentation and to establish optimal Independent Characteristics representative of each of the features segmented via difference processing. It is also envisioned that the Independent Characteristics of the features covered (and therefore blocked from view) by operations on the assembly line will be learned. Thus, the presence of a component or solder paste will not only be detectable by image aspects that fall into a hyper- volume of Independent Characteristics that describe it, but also by the absence of the image aspects that fall into the hyper- volume of Independent Characteristics that describe what it is obscuring.

The difference processing method described above can be used not only to assist with Image Segmentation for Training, but can be used also for run-time Image Segmentation. In one embodiment, each circuit board under test would be examined at the various stages of operation. In a second embodiment, a single representative set of images is acquired from a first circuit board and is used in difference processing of images acquired from other circuit boards. Variations in the appearance between instances of the bare circuit board make this somewhat challenging for difference processing techniques reliant solely on 2D data. The inclusion of 3D Independent Characteristics overcomes these difficulties because while instances of the bare circuit board may have varying color, or brightness, etc., they will not have substantial locally varying height.

Also, difference processing can be used to help establish the Independent Characteristics of components.

The inspection methods disclosed so far rely at least in part on 3D data. Acquisition of 3D data requires more time than the acquisition of 2D data but, as discussed, offers substantial improvements in the reliability of proper error determination and in the reduction of the training burden placed on the user. Also, as discussed above, Machine Readable data does not contain enough information to allow for construction of 2D models complete enough to allow for reliable inspection. In further aspects of this invention, the system is able to operate at speeds approaching those of 2D systems without compromising the benefits that accrue to 3D systems. Referring to Figure 11, high resolution 2D color images 87 of the DUT coupled with the

Inspection Results based at least in part on the comparison of the 2D/3D images 86 of the DUT to the models 88 are used, over the course of a Learning/Running Phase, to generate 2D Models 92 that describe how DUTs, determined to be "good" (proper assembly) using inspection based at least in part on 3D images, appear in 2D. During this Learning/Running phase, the system renders reliable decisions based, at least in part, on 3D data and nominal 3D models generated from Machine Readable data per Figure 8. Thus the system is running, albeit at a slower speed, and inspecting DUTs at the same time as it is generating the 2D models of "good" that it will need in order to enable it to deliver reliable results, while running faster, during the Running Phase.

Figure 12 illustrates an embodiment of a similar implementation, where 2D models 92 of "bad" are generated. Referring to Figure 12, the system uses high resolution 2D color images 87 of the DUT coupled with the Inspection Result based on the comparison 90, which is based at least in part on 3D images of the DUT, to enable the system to, over the course of a Learning/Running Phase, generate 2D Models 92 that describe how DUTs, determined to be in "bad" (defective assembly) using inspection based at least in part on 3D images, appear in 2D. During this Learning/Running phase, the system renders reliable decisions based, at least in part, on 3D data and 3D models generated from Machine Readable data per Figure 8. Thus the system is running, albeit at a slower speed, and inspecting DUTs at the same time as it is generating the 2D models of "bad" that will help it to deliver reliable and more complete results, while running faster, during the Running Phase.

Continuing with Figure 12, when the Inspection Result based at least in part on 3D data includes not just than an error occurred but, additionally, the error category, the 2D models 92 of 'bad' can be constructed so as to describe the 2D appearance of that category. As mentioned elsewhere, easily categorized errors are absence, tombstone, billboard and the like. Thus inspection based at least in part on 3D data is able to categorize defects during the Learning/Running phase enabling the generation of defect category specific 2D models 92. Hence DUTs that are inspected during the Learning/Running phase that contain assembly defects, and where one or more of those defects can be detected and categorized based at least in part on 3D images of the DUT 86, contribute to generation of defect category specific 2D models 92 that can, during the Running phase, permit the system to categorize the defect while running at high speed

Referring to Figure 13, the 2D models 92 as described above allow the system to generate fast and reliable Inspection Results during a Running phase. The image acquisition system 81 need only acquire 2D images 87 of the DUT (which can be done faster than the data necessary to form 3D images) and deliver them to the image comparison mechanism 91. The 2D models 92 of 'good' and 'bad', once thoroughly populated during the Learning/Running phase, are sufficient to allow for reliable defect detection. When the 2D models of 'bad' 92 have enough examples of category specific defects, then in addition to allowing reliable defect detection, reliable defect categorization can be achieved.

Inspection based at least in part on 3D data can be used to verify the true condition contingent on a defect condition arising from the use of 2D images and models during the Running phase. Similarly, the inspection based at least in part on 3D data can be used to verify the true condition contingent on an Uncertain Condition arising from the use of 2D images and models during the Running phase. The 2D models can then be updated based on the verified true conditions. The contingent inspections can be used to lower the false call rate as seen by the system's user, at the cost of a slight slowdown compared to 2D inspection only.

Further, the contingent inspections can be used to allow decision thresholds used by 2D Inspection Algorithm to be biased towards false calls and away from false accepts thereby reducing the false accept rate and the cost of a further slowdown as compared to 2D inspection only. In a simple but illustrative example, the 2D models discussed above can be a set of known good images of a component on a circuit board. Each image in the set is known to be good because during the Learning/Running phase, inspection based at least in part on 3D data determined that it was good. During the Learning/Running phase, the system automatically added those 2D images to the known good set. Later, during the Running phase when, in the interest of higher throughput, the system has transitioned to operating primarily from 2D images, the 2D image(s) acquired from a DUT are compared, for example, by normalized correlation, against all the known good images in the known good set. If the correlation score is high, compared to a threshold intended to separate "good" from "bad", but possibly biased towards generating "bad", the site is determined to be "good". An improvement on the above compares the DUT image's highest normalized correlation score obtained from the known "good" image set against the DUT image's highest normalized correlation score obtained from the known "bad" image set. Thus the system is not only able to measure the similarity of the DUT's 2D image against known "good" images, but it can additionally evaluate the similarity of the DUT's 2D image against all the known "bad" images. This makes defect detection more reliable because the highest normalized correlation score of the 2D image against known "good" can be gauged against the highest normalized correlation score of the 2D image against known "bad". If the known bad image set is subdivided into error categories, then, if a defect has been detected according to the above, the error category will also be known.

No matter how well populated the 2D models of known good and known bad are, 2D processing will still, occasionally, make an error with a probability that exceeds that of methods that take advantage of 3D data. To reduce the number of such errors that require User interaction, methods described above can be used as a 'backstop'. If during the Running phase, image processing based on 2D data alone has determined that an assembly error occurred, the system can re-examine the suspect site in 3D. Thus the putative error is confirmed or rejected based on the most reliable means. If enabled to do so automatically, the system achieves this without User involvement, at the cost of the extra time required to re-examine the site. In nearly all cases, this extra time will be short compared to the time required for the User himself to view the 2D data to verify the putative error. Additionally, in cases where this backstop method determined that the initial result based on the primary 2D inspection method found an error that wasn't truly there, the 2D model can be updated to include information to reduce the probability of future such occurrences. Again, if enabled to do this automatically, the User need not even know that this occurred.

Even with incomplete 2D models (of known good and/or bad), in some cases 2D inspection will confidently discover errors or not. In our simple example where the 2D models for "good" and "bad" is each a collection of known "good" and known "bad" images and the 2D Inspection Algorithm is based on normalized correlation, this confidence comes from high normalized correlation score for one category at the same time as low such score for the opposite. For example, if a 2D image from a particular component placement site has a maximum normalized correlation score against all known "good" images that is low, and a maximum normalized correlation score against all known "bad" images that is high, it is very likely that an error has occurred. In such a case, it may not be necessary to use the 3D backstop method to verify the failure because the probability is very high that the primary 2D method rendered the correct verdict. This is referred to as a Certain Condition. On the other hand, if the two maximum correlation scores are not very different from each other, even if for example they are both high, then the probability is not very high that the 2D method rendered the correct verdict; the confidence of the determination is lower leading to an Uncertain Condition, hi this case (which is illustrative of why having models of both "good" and "bad" can help make results more reliable) the 3D backstop method would be more useful as described above. It should be clear that the above methods can work for a variety of DUTs, including circuit boards populated with components and for solder paste inspection or a mix. In the case of solder paste inspection, the apparent 2D difference between good and bad may, under some conditions, not be very large. The system of this invention is then able to, during the Running phase, automatically revert to 3D inspection because (using normalized correlation as a simple but workable example of image processing) the normalized correlation scores of the 2D image against the known "good" and the known "bad" models will not be that different from each other. This is an Uncertain Condition that then causes the automatic use of 3D backstop inspection. Thus solder paste inspection of_sitesjwhere 2D images contain strong indications of "good" or "bad" will be reliably inspected in 2D, while those sites where 2D images do not strongly indicate "good" or "bad" will be automatically inspected in 3D. This can be generalized beyond solder paste: Any site to be inspected that carries a strong signal of correctness in its 2D image will, eventually, be reliably inspected using 2D and the converse is also true.

It should be clear that, although the prior discussion used normalized correlation as the Inspection Algorithm to be used with models comprising collections of known "good" and known "bad" images, other Inspection Algorithms and model generators may be more reliable, efficient or both and may be used without departing from the spirit and scope of the invention.

Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that a variety of alternate and/or equivalent implementations may be substituted for the specific embodiments shown and described without departing from the scope of the present invention. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is intended that this invention be limited only by the claims and the equivalents thereof.

Claims

WHAT IS CLAIMED IS:

1. A method of inspecting a Device Under Test, comprising: deriving a model of a Device Under Test; obtaining a 2D image of the Device Under Test; obtaining a 3D image of the Device Under Test; and comparing the 2D and 3D images to the model.

2. The method of claim 1, wherein the model includes a 2D model and a 3D model of the Device Under Test.

3. The method of claim 1, wherein the model is derived from machine readable data.

4. The method of claim 1, wherein the model is derived from 2D and 3D images of an example of the Device Under Test.

5. The method of claim 1, further comprising generating a 2D model of the Device Under Test based on the comparison and an image of an example of the Device Under Test.

6. The method of claim 5, wherein the comparison determines that the Device Under Test is good, and the generated 2D model describes the good Device Under Test.

7. The method of claim 5, wherein the comparison determines that the Device Under Test is bad, and the generated 2D model describes the bad Device Under Test.

8. The method of claim 5, further comprising comparing the 2D image of the Device Under Test to the 2D model.

9. The method of claim 1, wherein the 2D and 3D images are obtained and compared to the model to verify a result of a previous inspection of the Device Under Test.

10. An inspection system, comprising: a memory storing a model of a Device Under Test; an image acquisition device configured to obtain a 2D image and a 3D image of the

Device Under Test; and a processor having access to the memory and programmed to compare the 2D and 3D images to the model.

11. The inspection system of claim 10, wherein the processor is further programmed to derive the model based on machine readable data describing the Device Under Test.

12. The inspection system of claim 10, wherein the processor is further programmed to derive the model based on 2D and 3D images of an example of the Device Under Test.

13. The inspection system of claim 10, wherein the memory further stores a 2D model of the Device Under Test based on the comparison and an image of an example of the Device Under Test.

14. The inspection system of claim 13, wherein the processor is further programmed to compare the 2D image of the Device Under Test to the 2D model.