JP2011516860A - Method and apparatus for acquiring and processing multiplexed images - Google Patents

Abstract

A method and apparatus for inspecting an article is provided.
In a method and apparatus for inspecting an article, a light source having a spectral response that matches the spectral response of a Bayer filter (12) affixed to the image sensor (10) from two or more directions. Illuminate at (52, 56). The image data is separated by color and a derivative image is generated and then processed to detect defect groups.
[Selection] Figure 3a

Description

  The present invention relates to high-speed inspection of manufactured articles using image recognition technology. Specifically, the present invention relates to a technique for acquiring and processing a plurality of images of an article acquired simultaneously with a single camera while the article is moving. More specifically, the present invention relates to a technique for separating multiple image groups and simultaneously acquiring multiple images using different color light that illuminates an article from different directions and a color filter group attached to a sensor.

  Many manufactured articles are inspected by a machine vision system for small defects during the manufacturing process. In a typical inspection system, a 2D (two-dimensional) grayscale camera is used to detect a defect group of an article based on a difference in grayscale reflectance or a difference in 2D morphology (surface morphology). In some cases where there are defect groups that cannot be detected due to gray scale differences, a color camera group is used to detect defect groups that can be distinguished by differences in spectral reflectance. If the defect group cannot be identified by any grayscale difference or color image difference, the defect group characterized by the difference in 3D morphology can be determined using a 3D (3D) vision system. Can be detected.

A 2D vision system usually consists of a camera, an optical system, and an illumination source. Since camera technology is usually limited to commercially available sensor groups, optical systems and illumination sources are usually customized to achieve the correct magnification, field of view, and acquisition speed. One example of a machine vision system manufactured using a commercially available image sensor, custom optics, and illumination light source is an ESI Bullet (available from Electro Scientific Industries, the assignee of this patent application). Barrett) ^TM wafer ID reader. The system includes a video sensor, an optical system, and an illumination light source, and these components are configured to inspect identification marks provided on a semiconductor wafer, among other items. Although the system is configured to image very small marks formed on a specular surface, the system was unable to image the micro defects that are the subject of the present invention. The reason is that these flaws are not reflected in the camera as a difference from the surroundings. Since these defects can only be seen as minute dents on the surface of the article, they cannot be imaged by conventional 2D imaging methods.

  One possible way to image the defect group is to image the article while illuminating it with light that illuminates the article from at least two different directions with a shallow glazing angle. This light needs to shine from at least two directions because not all of these defects are visible from a single light angle. Processing all of these images allows these defects to be detected, but the process is unacceptably slow because a separate frame time is required for each acquisition. Images can be acquired in parallel, but this requires one sensor for each light direction along with a group of filters so that each light is captured only by the appropriate camera. This results in huge costs in the process.

  Another adoptable solution to the problem of imaging microdefects uses a 3D imaging method. The 3D vision system generates image data corresponding to the actual 3D shape of the article, not the wavelength intensity of light reflected by the article. 3D systems can be classified into several groups according to the related technology. The first classification is a passive 3D system. In this passive 3D system, an image group is composed of height information obtained from a cue included in a normal 2D image. An example of this is stereoscopic image reconstruction, in which two or more 2D images are acquired, multiple feature points are detected in each image, and then detected in each image Solve the correspondence problem to try to match these feature points. Since the discrepancy is due to a difference in height perception between these images, the height is estimated. This entails the same problem as the above-described plurality of 2D images, that is, the time required to shoot a plurality of images and the equipment cost required for parallel acquisition. Also, this approach does not work for the defect groups in question here, because the defects show feature points that can be used to estimate height. Because there is no.

  Another type of 3D imaging method involves projecting a group of projection lines, usually a laser line or other shape, onto an article and then imaging. The displacement of these lines represents 3D irregularities. This method further requires multiple acquisitions, so the method is unacceptably slow for this application. Other types of 3D imaging methods are performed by detecting the focus quality over a series of image steps (confocal imaging) or by acquiring a group of images of a special projection grid (Moire imaging). Both of these methods require multiple image acquisitions to construct a single 3D image, and for these reasons, both of these methods are unacceptable solutions to this problem. is there. In addition, these methods typically require that the article be held stationary during the scan period, which means that the article must be moved in stages, stopped, and stationary before scanning. All of these operations slow down the manufacturing process.

  An example of this method is a paper titled “Technology on Phase Measurement and Surface Reconstruction by Using Color Structured Light” by Alexander (Applied Optics Vol. 41, Issue 29, pp. 6104-6117 (2002). In this example, a plurality of color patterns are projected to discriminate the projection light and to improve the accuracy of 3D measurement. Alexander et al. Describe that 3D information is extracted from a structured light image by using structured light to determine the surface shape of an automotive windshield using a differential equation.

Applied Optics Vol. 41, Issue 29, pp. 6104-6117 (2002)

  What many of these methods have in common is a requirement for acquiring more than one image in order to detect a defect group of an article. The problem with acquiring multiple images of moving parts is that multiple exposures are required to acquire multiple images with a single camera. Since the part is moving past the camera, whether it is necessary to stop the part in place to acquire multiple images or whether the camera needs to be moved with the part Either. None of the solutions are acceptable because stopping the part reduces system throughput, and moving the camera is difficult and expensive. Another possible solution uses multiple cameras, which is expensive and requires special work to align the optics and sensors.

  Therefore, what is needed is an image acquisition method that can image a defect group that cannot be easily revealed by a 2D inspection method. In addition, this approach does not require multiple cameras or other additional expensive equipment and only works for a portion of a single frame time, thus keeping the article stationary during acquisition. The requirement to do can be minimized or eliminated.

  An object of the present invention is to provide a method and an apparatus as a form of an image acquisition system having an improved function of detecting defects in a group of articles that cannot be revealed by 2D inspection means. Another object of the present invention is to perform the acquisition without the need to hold the group of articles so that they do not move during acquisition. In order to achieve the above and other objects in accordance with the objects of the present invention as embodied and broadly described herein, methods and apparatus are disclosed.

  Two or more images of an article are illuminated in a single image frame, the article is illuminated with two or more different colors of light, and data is acquired using an image sensor with a Bayer filter structure applied. Is obtained by The Bayer filter is a color filter that is attached to the image sensor and includes a red filter, a green filter, and a blue filter. The Bayer filter is attached to the image sensor, and image data is acquired. When acquiring the image data and importing it into the controller built in the camera itself or the controller connected to the camera, the pixel group is classified so as to change depending on which color of the filter the pixel group corresponds to. That is, all of the red pixel groups are allocated to one image, all of the blue pixel groups are allocated to another image, and all of the green pixel groups are further allocated to another image. In this way, a sensor that acquires a monochromatic image can be made to simulate a color imaging system with a single sensor, even at a rather low spatial resolution. Since these color images are processed so that their registrations coincide with each other, one feature point of one image is generated at the same position in another image group.

  By matching the illumination wavelength to the transmission wavelength of the Bayer filter, three individual images can be generated from the acquired single image. For example, if the article is illuminated with two or more of three colors from different directions, these generated images can be processed to present inconsistencies in these images, thus representing a group of defects. Minor differences in morphology can be presented. In addition, the lighting can be turned on at the right time, or the image sensor can be operated at the right time, so that data can be acquired within a very short period of time, so that the three images are matched at the same time, and therefore the registration is matched. These images can be acquired without stopping the article.

  In one embodiment of the invention, the article is illuminated from two different directions with light of a wavelength that matches the transmission wavelength of two filters of the filter group of the Bayer filter. The third image is acquired using a diffuse illumination source with a wavelength that matches the wavelength of the third color of the Bayer filter. Image data is acquired and separated into three individual images corresponding to the three colors of the Bayer filter. Next, the difference between the two images is emphasized by subtracting the first two images because the feature points that occur in one image but not in another image may be defective. Because it is very expensive. The resulting image is then processed using conventional machine vision techniques to identify and classify defect groups. Diffuse images are also processed using conventional machine vision techniques to determine whether other defect groups such as contour deviations from nominal values are included, but these defects are directional illumination There is a possibility that it cannot be detected in the image obtained by.

  In one embodiment of the invention, the three illumination sources are illuminated at a certain timing or flashed for a very short period of time. As a result, data can be acquired from the moving parts by the sensor without causing motion blur due to movement. The sensor is set to collect light and then these illumination sources are set to emit at some time while collecting light on the sensor. After emitting light at a certain timing, the sensor is instructed to read out image data generated in the sensor by the light emitting illumination light source and take it in the controller. In this case, the image data corresponds to three color filters. Separated into three images.

  When an article is moving near the sensor's field of view at a very high speed, it is impractical to stop the movement of the article simply because the illumination light source emits light at a certain timing. In this embodiment, the movement of the article is slowed by implementing a sinusoidal velocity profile in the transport mechanism that moves the article as it passes through the field of view of the sensor. In this embodiment, the controller is operably connected to the transport mechanism and controls the speed at which the group of articles is moved. By increasing or decreasing the speed of the transport mechanism according to a sinusoidal speed profile so that the timing when the minimum speed is reached coincides with the timing when the illumination light source emits light, the speed of these articles is sufficiently slowed down, It is possible to eliminate motion blur while minimizing a decrease in throughput due to the reduction.

It is a schematic diagram which shows how an RGB Bayer filter structure is affixed on an image sensor. It is a chart which shows the spectrum response of a normal RGB Bayer filter structure. 1 shows a diagram of a directional image acquisition system that uses a Bayer filter. FIG. 1 shows a diagram of a directional image acquisition system that uses a Bayer filter. FIG. The directional illumination image processing is shown. The directional illumination image processing is shown. The directional illumination image processing is shown. Fig. 6 illustrates diffuse illumination image processing.

  The present invention relates to acquisition of image data for supporting automatic inspection of an electronic component group. An example of this type of component is a chip capacitor, which is formed by alternately laminating metal conductor layers and ceramic dielectric layers. These components have defects that are difficult or impossible to image using conventional 2D or 3D systems. Furthermore, these parts are formed at a very fast rate. Equipment that manufactures these parts can manufacture these parts at a rate of hundreds of thousands per hour. Eventually, these parts are assembled into a circuit group, and these circuits become almost all electronic devices currently on the market; therefore, before these parts are assembled into a circuit group, It is advantageous to inspect. With the present invention, the imaging cycle can be shortened to a single image exposure (2 ms) and a faster 18 ms movement profile can be used. A 20 ms cycle would allow 180,000 throughput rates per hour.

  A broadband spectral image sensor with Bayer filters attached thereto is widely used to acquire color image information. FIG. 1 is a schematic diagram of a semiconductor image sensor 10 with a Bayer filter 12 attached thereto. The Bayer filter is composed of three different color filters, namely a red filter 14, a green filter 16, and a blue filter 18. The Bayer filter 12 is affixed to the sensor 10 such that each color segment of the Bayer filter 12 is aligned with a pixel or pixel of the image sensor 10. One of these pixels or pixels is indicated by the numeral 20. In this way, only one color of light is collected at each pixel 20 so that the resulting image data is read from the sensor 10 by a controller (not shown), and these data are classified into three individual images. Each individual image represents a group of pixels that are illuminated with only one color. Typically, this function is used to generate a full color image group using a broadband monochromatic sensor, but in one embodiment of the invention, this function is used to generate an individual image group, each individual image being 1 Corresponds to two individual illumination sources. Subsequent processing can be facilitated by creating these three images so that the size, bit depth, and registration match.

  FIG. 2 shows a state in which the quantum efficiency 32 of three types of color filters found in a normal Bayer filter is plotted against the wavelength 34. Quantum efficiency indicates the percentage of photons of a particular wavelength that are captured by the sensor and converted into a charge, and corresponds to the spectral response. The three color peaks correspond to the blue filter 36, the green filter 38, and the red filter 40. As shown, the blue filter shows a peak at about 450 nm, the green filter shows a peak at about 550 nm, and the red filter shows a peak at about 625 nm. All three filters transmit energy in the infrared (IR) region of the spectrum (about 700 nm and above). Most sensors of this type further include an IR filter that prevents this excess energy from reaching the sensor. Examples of devices with these types of image sensors include cell phone cameras and most medium to low cost video cameras. An exemplary image sensor with a Bayer filter attached is a part number ICX445 EXview HAD CCD sold by Sony Corporation, based in Tokyo, Japan.

In one embodiment of the present invention, using the technology developed for color image detection, three individual images are taken by using a Bayer filter in one frame time. This image capture is performed with the sensor exposure reduced to one time by illuminating the part from at least two different directions to multiplex multiple views of the part into a single image. Each illumination source utilizes a unique light color or light wavelength that matches the color group of the Bayer filter. In one embodiment of the invention, the following color LED groups (light emitting diode groups) are used for the image illumination light source:
-Blue light emitting diode with -470 nm wavelength -used for north side image illumination light source -Green light emitting diode with 525 nm wavelength -used for image diffusion illumination light source (third image)
-636 nm wavelength red light emitting diode-used in the eastern image illumination light source As an exemplary LED that can be used for this application: 470 nm wavelength HLMP from Avago Technologies, based in San Jose, California CB30-M0000; LTST-C190TGKT with a wavelength of 525 nm manufactured by Ryton Semiconductor Co., Ltd. based in Taipei, Taiwan; and SML-LX04021C-TR manufactured by Lumex, Inc. based in Palatine, Illinois.

  In a typical Bayer filter, 50% of the number of pixels is used for green, 25% for blue, and 25% for red. This ratio reflects the fact that these sensors produce color images from monochromatic sensors and that the human eye is more sensitive to green light than blue or red light. This is because it was configured. It should be further noted that this proportion represents only one specific type of color filter. Other color filter groups that can be used to separate the color groups on the monochromatic filter are, for example, color stripe groups whose positions coincide with the pixel column groups on the sensor. For other filter groups, slightly different colors can be used. Any of these other filters can be used in accordance with one embodiment of the present invention without departing from the basic concept.

  Figures 3a and 3b illustrate one embodiment of the present invention. FIG. 3 a shows a perspective front view from above and FIG. 3 b shows a side elevation view of one embodiment showing the article 50, the article 50 passing by the camera 60 by a transport mechanism. The first illumination light source 52 that projects the first color collimated light 54 from the first direction and the second illumination light source 56 that projects the second color collimated light 58 are illuminated. Further, the third illumination light source 62 illuminates the article 50 with the diffused light 64 of the third color. A camera 60 including a sensor with a Bayer filter attached (not shown) acquires image data from the article to be illuminated 50.

  FIG. 4 a shows image data 70 obtained using the first color light from the first direction 74 from the article illuminated by the first illumination light source 72. The image data 70 acquired by this illumination method shows one defect 76, and this defect 76 appears as a relatively bright color area. FIG. 4 b shows image data 78 acquired from the article illuminated by the second illumination light source 80 using the second color light from the second direction 82. Note that in this case, the defect is not visible in the image data 78. FIG. 4 c shows composite image data 84 calculated by subtracting image data 70 from image data 78. In the composite image data 84, the defect 86 is clearly visible. The purpose of synthesizing image data based on two different directional images 70 and 78 is to suppress information contained in both images and to emphasize information contained in only one of these images. There is to do. The idea is that a defect group to be detected is visible when illuminated in only one direction. Feature points that appear in two directions are usually not classified as defects. Subtraction is an exemplary method for emphasizing the defect detection unit by synthesizing image groups, but other arithmetic operations, logical operations, order statistic calculations such as maximum / minimum statistical operators, or these It should be noted that other operations including combinations of these operations can be used. After image composition, defects are identified in the composite image using standard machine vision techniques.

  In the case where a defect appears in both directional images, the diffusion image of the article can be used to detect the defect. FIG. 5 shows a grayscale image 90 of the article acquired with diffuse illumination from above. Defect 92 is shown circled. These image data can be combined with one or both of these directional images, or only these image data can exist by processing using standard machine vision techniques. Further information regarding can be generated.

  In one embodiment of the invention, data is acquired in a single frame time using a technique that multiplexes three images on a single sensor acquired in one frame time. Limiting the acquisition time to a single frame time minimizes the time required to stop the part, acquire data, and then resume the movement of the part. Further, when the sensor is allowed to collect light only for a limited period, image data can be acquired without stopping the parts by executing the electronic shutter function with the sensor. These illumination light sources can also emit light when the part passes the sensor's field of view and stops moving the part. The part can continue to move beside the camera at the normal speed of the part, or the electronic light source function can be performed, or the illumination light source can emit light in a timely manner. If the image data contains motion blur, the speed at which the part is moving through the sensor's field of view by programming the movement of the part to make a sinusoidal movement And speed up when no image is acquired.

  It will be apparent to those skilled in the art that many modifications can be made to the details of the above-described embodiments of the invention without departing from the basic principles of the invention. Accordingly, the scope of the invention should be defined only by the following claims.

DESCRIPTION OF SYMBOLS 10 ... Semiconductor image sensor, 12 ... Bayer filter, 14 ... Red filter, 16 ... Green filter, 18 ... Blue filter, 20 ... Pixel, 32 ... Quantum efficiency, 34 ... Wavelength, 36 ... Blue filter, 38 ... Green filter, 40 ... red filter, 50 ... article, 51 ... transport mechanism, 52 ... first illumination light source, 54 ... first color collimated light, 56 ... second illumination light source, 58 ... second color collimated light, 60 ... camera, 62 ... 3rd illumination light source, 64 ... 3rd color diffused light, 70 ... Image data, 72 ... 1st illumination light source, 74 ... 1st direction, 76 ... Defect, 78 ... Image data, 80 ... 2nd illumination light source, 82 ... 2nd direction, 84 ... Composite image data, 86 ... Defect, 90 ... Grayscale image, 92 ... Defect

Claims (12)

An improved method for obtaining image data of an article using first and second illumination sources, a image sensor with Bayer filters attached, and a controller, wherein the improvements are:
Illuminating the article from the first direction with the first illumination light source with light having a spectrum that matches the spectrum of the first color of the Bayer filter;
Illuminating the article from the second direction with the second illumination light source with light having a spectrum that matches the spectrum of the second color of the Bayer filter;
Acquiring image data from the article illuminated by the first illumination light source and the second illumination light source by using the image sensor and importing the image data into the controller;
Separating the image data into a first color image including data corresponding to the first illumination light source and a second color image including data corresponding to the second illumination light source using the controller; ;
Obtaining image data of the article by generating a generated image by calculation between the first color image and the second color image using the controller;
Including a method.   The method of claim 1, wherein the first direction and the second direction differ by 180 degrees.   The method of claim 1, wherein the first color of the Bayer filter is about 550 nm wavelength light.   The method of claim 1, wherein the second color of the Bayer filter is about 635 nm wavelength light.   The method of claim 1, wherein the calculation between the first color image and the second color image is a subtraction.   The method according to claim 1, wherein the first illumination light source and the second illumination light source emit light at a certain timing. An improved system for acquiring image data of an article using a image sensor with a Bayer filter attached thereto, the system being operatively connected to the image sensor with a Bayer filter attached thereto, and the image sensor A controller, wherein the improvements include:
A first illumination light source that operates to illuminate the article from a first direction with light having a spectrum that matches a spectrum of a first color of the Bayer filter;
A second illumination light source that operates to illuminate the article from a second direction with light having a spectrum that matches a spectrum of a second color of the Bayer filter;
The image data is acquired from the image sensor with a Bayer filter attached thereto, and the image data is obtained from an image of a first color corresponding to image data illuminated by the first illumination light source and the second illumination light source. The controller operative to separate into a second color image corresponding to the image data to be illuminated,
The improved system further operates to obtain the image data of the article by generating a generated image by calculation between the first color image and the second color image.   The system of claim 7, wherein the first direction and the second direction differ by 180 degrees.   The system of claim 7, wherein the first color of the Bayer filter is about 550 nm wavelength light.   The system of claim 7, wherein the second color of the Bayer filter is about 635 nm wavelength light.   The system of claim 7, wherein the calculation between the first color image and the second color image is a subtraction.   The system according to claim 7, wherein the first illumination light source and the second illumination light source emit light at a certain timing.

Images