JP5778551B2 - Foreign matter detection method and apparatus for imaging device - Google Patents

Description

  The present invention relates to a foreign object detection method and apparatus for an image sensor, and more particularly to an image sensor package in which a transparent body such as a cover glass is disposed in front of a light receiving surface of an image sensor (image sensor chip). The present invention relates to a foreign object detection method and apparatus for an image sensor capable of specifying the position, size, and shape of a foreign object existing on the front and back surfaces of a transparent body.

  An image pickup device package mounted on a printed circuit board of various devices such as a digital camera, a digital video camera, a mobile phone, and an electronic endoscope is generally a solid-state image pickup device (such as a CCD or a CMOS) in a concave housing portion of the package body. The chip is housed and fixed, and a cover glass is attached to the package body so as to close the opening of the housing part, and the inside of the housing part is sealed.

  In such an image pickup device package, if foreign matter such as dust adheres to the light receiving surface of the image pickup device or the front and back surfaces of the cover glass, the foreign matter is reflected in the picked-up image and deteriorates the image. It has been known. Therefore, it is necessary to physically remove such foreign matter or to remove it by software by image processing.

  In Patent Document 1, the total amount of light received in a pixel area of n × n pixels is obtained with each pixel as a central pixel on the light receiving surface of an image sensor, and the total amount is reduced by a decrease from the total value when no foreign matter is present. Find the amount. Then, it has been proposed that the position and size (the size of n) of the pixel region where the amount of received light is reduced by the foreign matter is obtained based on the reduction amount, and the position where the foreign matter is present is detected. In addition, it has been proposed to perform correction on an image of a pixel region in which the amount of received light is reduced by a foreign object based on the peripheral image.

  In Patent Document 2, a cleaning image is displayed on a display unit when a diffusion plate is attached to the tip of a camera lens mount or a photographic lens, so that a cleaning operator can detect a low-luminance portion of the photographic image as a light receiving surface. It has been proposed to improve the convenience of cleaning so that it can be grasped as the position of a foreign substance adhering to the like.

  In Patent Document 3, a uniform light is photographed with a camera, and it is determined that a foreign substance is attached to a pixel position where the image signal is smaller than a specified value. It has been proposed to correct the signal.

  In Patent Document 4, when images having different apertures are compared and a region where the luminance difference is equal to or greater than a predetermined threshold is detected, it is determined whether or not foreign matter is attached to the lens. Has been proposed to warn.

JP 2003-298924 A JP 2004-172835 A JP 2009-17058 A JP 2011-78047 A

  By the way, when a foreign object exists on the light receiving surface of the image sensor of the image sensor package or the surface of the cover glass, the defect area in which the foreign object appears in the photographed image depending on the state of the foreign object in addition to the position and size of the foreign object. The position, size, etc. of this change. Therefore, when correcting a defect area of a captured image caused by a foreign object by image processing, it is necessary to accurately grasp the three-dimensional position, size, and shape of the foreign object.

  In Patent Documents 2 to 4, detection of presence / absence of a defect area (shadow of a light receiving surface created by a foreign object) of a captured image due to a foreign object and the position and size of the defective area are performed, but the foreign object is on a specific one surface. Except for the case where it exists on a light receiving surface or a specific lens surface, detection that specifies the three-dimensional position or size of the foreign substance itself is not performed.

  Further, Patent Document 1 proposes predicting the three-dimensional position of a foreign object on the assumption that the size of the foreign object is constant, but the size of the foreign object is not actually constant. Therefore, the error is large, and the size of the foreign matter cannot be specified.

  The present invention has been made in view of such circumstances, and the position of a foreign substance present on either the front surface or the rear surface of a transparent body such as a cover glass disposed in front of the light receiving surface of the image sensor. Another object of the present invention is to provide a foreign object detection method and apparatus for an image sensor that can specify the size and shape.

  In order to achieve the above object, a foreign object detection device for an image sensor according to a first embodiment of the present invention is directed to a foreign object detection target in which a transparent body is disposed on the front side of a light receiving surface of an image sensor. The first illumination means for irradiating the light receiving surface with parallel light as first illumination light, the second illumination means for irradiating the light receiving surface with diffused light diffused from a point light source as second illumination light, and the first illumination A first photographed image acquisition means for obtaining a first photographed image obtained by photographing the image of the light receiving surface when the first illumination light is irradiated by the means with the image sensor; and the second illumination means for performing the second illumination. A second captured image acquisition means for acquiring a second captured image obtained by capturing an image of the light receiving surface when illumination light is irradiated by the image sensor; and the light receiving surface and the light source based on the first captured image. Table on either the front or back of the transparent body A first dark region detecting means for detecting a first dark region on the light receiving surface which is blocked by the first illumination light by a foreign matter present on the light source, and based on the second photographed image, A second dark area detecting means for detecting a second dark area on the light receiving surface which is blocked by the foreign light and blocked by the second illumination light; and the received light detected by the first dark area detecting means. Based on the position coordinate range of the first dark area on the surface and the enlargement ratio of the second dark area on the light receiving surface detected by the second dark area detection unit with respect to the first dark area, Foreign matter specifying means for specifying the surface on which the foreign matter exists among the light receiving surface and the front and back surfaces of the transparent body and specifying the range of the position coordinates of the foreign matter on the surface is provided.

  According to the present invention, it is possible to specify the surface on which the foreign matter exists among the light receiving surface of the imaging device and the front and back surfaces of the transparent body, and it is possible to specify the range of the position coordinates of the foreign matter on the surface. Therefore, the three-dimensional position, size, and shape of the foreign matter can be specified, and for example, effective information can be obtained in correcting an image of a defective area caused by the foreign matter.

  In the present invention, the second illuminating means may irradiate the light receiving surface with the light that has passed through the aperture of the diaphragm among the light emitted from the predetermined light source as the second illuminating light.

  In the present invention, the first illumination means may irradiate parallel light perpendicularly to the light receiving surface as the first illumination light. The first illuminating means irradiates the light receiving surface with the parallel light as the first illuminating light from the light emitted from a predetermined light source, which has passed through the aperture of the diaphragm. Can be.

  In the present invention, the foreign object detection target may be an image sensor package.

  According to a first aspect of the present invention, there is provided a foreign object detection method for an image sensor, wherein parallel light is applied to the light receiving surface with respect to a foreign object detection target in which a transparent body is disposed on the front side of the light receiving surface of the image sensor. A first illumination step of irradiating as the first illumination light, and a first photographed image obtained by capturing the image of the light receiving surface when the first illumination light is irradiated in the first illumination step with the image sensor. A first captured image acquisition step, a second illumination step of irradiating the light receiving surface with diffused light diffused from a point light source as second illumination light, and when the second illumination light is irradiated by the second illumination step A second captured image acquisition step of acquiring a second captured image obtained by capturing the image of the light receiving surface with the image sensor, and based on the first captured image, the light receiving surface and the front and back surfaces of the transparent body Any foreign matter present on the surface of any of the above A first dark region detecting step of detecting a first dark region on the light receiving surface which is blocked by the one illumination light and is shadowed by the foreign material; and the second illumination light by the foreign material based on the second photographed image. A second dark region detecting step for detecting a second dark region on the light receiving surface that is blocked by the foreign matter and the first dark on the light receiving surface detected by the first dark region detecting step The light receiving surface and the transparent body are based on a range of position coordinates of the region and an enlargement ratio of the second dark region on the light receiving surface detected by the second dark region detecting step with respect to the first dark region. A front surface and a rear surface on which the foreign material exists, and a foreign material specifying step for specifying a range of position coordinates of the foreign material on the surface.

  According to the present invention, the position, size, and shape of a foreign substance existing on either the front surface or the back surface of a transparent body such as a cover glass disposed in front of the light receiving surface of the image sensor are specified. Can do.

Explanatory diagram showing the standard illumination state used to explain the principle of foreign object detection Explanatory drawing which showed the 2nd illumination state used for description of the 1st foreign material detection method Explanatory drawing used to explain the first foreign object detection method Explanatory drawing used to explain the first foreign object detection method Explanatory drawing which showed the 3rd illumination state used for description of the 2nd foreign material detection method Explanatory drawing used to explain the second foreign object detection method Explanatory drawing used to explain the second foreign object detection method The block diagram which showed the whole structure of the foreign material detection apparatus which concerns on this invention The figure which showed the state which set the light source part in FIG. 8 to the 2nd illumination state. The figure which showed the state which set the light source part in FIG. 8 to the 3rd illumination state. Flow chart showing the procedure for foreign object detection and processing

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  First, the principle of foreign object detection in the present invention will be described.

  An image pickup device 10 shown in FIG. 1A is accommodated in, for example, an image pickup device package (described later), and a cover glass 12 is disposed on the front side of the light receiving surface 10A of the image pickup device 10. Foreign matter may be present on the light receiving surface 10A of the image sensor 10 and the surfaces of the cover glass 12 (front surface 12A and back surface 12B) due to dust adhering, etc. In FIG. The case where foreign matter G1, G2, G3 exists in the part is illustrated.

  Here, the X axis is taken in the horizontal direction (lateral direction) on the light receiving surface 10A of the image sensor 10, the Y axis (perpendicular to the paper surface) is taken in the vertical direction (vertical direction), and the Z axis is perpendicular to the light receiving surface 10A. It is assumed that the optical axis O of the image sensor 10 is parallel to the Z axis. The light receiving surface 10A of the image sensor 10 and the front surface 12A and the back surface 12B of the cover glass 12 are arranged so as to be substantially parallel.

  In such foreign object detection of the image sensor 10 and the cover glass 12, first, the light receiving surface 10A of the image sensor 10 is irradiated as illumination light with parallel light parallel to the optical axis O (Z axis) as shown in FIG. To do. That is, parallel light incident perpendicularly to the light receiving surface 10A of the image sensor 10 is irradiated as illumination light. The parallel light is obtained, for example, by making diffused light emitted and diffused radially from the point light source 20 into parallel light by the collimator lens 30. You may make it irradiate parallel light using a laser light source.

  At this time, on the light receiving surface 10 </ b> A of the image sensor 10, there are generated a bright region where the illumination light is irradiated without being blocked by the foreign matter, and a dark region where the illumination light is not irradiated under the foreign matter. The range of the XY coordinate value of the dark region is the same as the range of the XY coordinate value of the foreign matter forming the dark region. FIG. 4A shows dark regions g11 to g13 that are not shaded by the foreign substances G1 to G3 and are not irradiated with illumination light. The range (large size) of the XY coordinate values of these dark regions g11 to g13 is shown. And the shape) coincide with the ranges (size and shape) of the XY coordinate values of the foreign substances G1 to G3.

  If the illumination state shown in FIG. 5A is referred to as a reference illumination state (first illumination state), an image on the light receiving surface 10A is captured by the image sensor 10 in the reference illumination state. A captured image P1 as shown in FIG. Assuming that each point on the captured image P1 is assigned the same XY coordinate value as the corresponding point on the light receiving surface 10A, the range of the XY coordinate value on the captured image P1 that matches the XY coordinate value in the bright area on the light receiving surface 10A. Are bright and form a bright region on the captured image P1. On the other hand, the pixels in the range of the XY coordinate values on the captured image P1 that coincide with the XY coordinate values in the dark region on the light receiving surface 10A are dark, and form a dark region on the captured image P1. FIG. 4B shows dark areas h1 to h3 on the photographed image P1 corresponding to the dark areas g11 to g13 on the light receiving surface 10A formed by the foreign substances G1 to G3.

  Therefore, in this captured image P1, the range of the XY coordinate value of the dark area formed on the light receiving surface 10A is detected by detecting the range of the XY coordinate value of the dark area where the pixel value is smaller than the predetermined threshold value. Then, the range of the XY coordinate values where the foreign matter exists in the XYZ coordinate space is specified. In addition, since it is rare that a foreign matter having a thickness in the Z-axis direction that has to take into consideration the influence on the photographed image exists, in this specification, the thickness of the foreign matter in the Z-axis direction is extremely thin. It will be treated as (none).

  Next, in order to specify the Z coordinate value (distance from the light receiving surface 10A) of the foreign substance in the XYZ coordinate space, the illumination state is changed and an image on the light receiving surface 10A of the image sensor 10 is captured. In the first foreign object detection method, the illumination state (second illumination state) as shown in FIG. 2A is switched to the reference illumination state shown in FIG. In this second illumination state, the diffused light emitted and diffused radially from the point light source 20 is irradiated as it is to the light receiving surface 10A of the image sensor 10 as illumination light. Note that the position of the point light source does not have to be the same as in FIG. Moreover, although the point light source 20 can be formed using a diaphragm as described later, a light source having an extremely small light emitting surface may be used.

  At this time, the light receiving surface 10A of the image pickup device 10 is illuminated with the bright region where the illumination light emitted from the point light source 20 is irradiated without being blocked by the foreign matter, and the illumination light emitted from the point light source 20 is blocked by the foreign matter. A dark region that is not irradiated with illumination light occurs. FIG. 4A shows dark regions g21 to g23 which are shaded by the foreign matters G1 to G3 and are not irradiated with illumination light.

  When an image of the light receiving surface 10A is picked up by the image pickup element 10 in this state, a picked-up image P2 as shown in FIG. Also in this captured image P2, the pixels in the range of the XY coordinate values on the captured image P2 corresponding to the bright region on the light receiving surface 10A are bright, and a bright region is formed on the captured image P2. On the other hand, the pixels in the range of the XY coordinate values on the photographed image P2 corresponding to the dark region on the light receiving surface 10A become dark and form a dark region on the photographed image P2. FIG. 4B shows dark regions i1 to i3 on the captured image P2 corresponding to the dark regions g21 to g23 on the light receiving surface 10A formed by the foreign substances G1 to G3.

Here, the size of the dark region on the light receiving surface 10A, that is, the size of the dark region in the captured image P2 varies depending on the Z coordinate value of the position where the foreign object exists in the XYZ coordinate space. That is, as shown in FIG. 3, the Z coordinate value of the light receiving surface 10A of the image sensor 10 is Z0 (for example, 0), the Z coordinate value of the point light source 20 is ZS, the Z coordinate value of the position where the foreign object G exists is ZG, and the foreign object When the size of G in the X-axis direction is WG, the size Wi in the X-axis direction of the dark region g formed on the light-receiving surface 10A (and the captured image P2) by the foreign matter G is expressed by the following equation (1):
Wi = (ZS−Z0) · WG / (ZS−ZG) (1)
Is required. A similar equation holds for the Y-axis direction.

  Accordingly, the farther the foreign object G is from the light receiving surface 10A, the larger the dark region g that the foreign object G forms on the light receiving surface 10A (and the captured image P2). The size (area S) of the dark region g formed on the light receiving surface 10A (and the photographed image P2) by the foreign matter is the size (area SG) of the foreign matter G itself, that is, formed on the photographed image P1 by the foreign matter. The enlargement rate K (= S / SG) is larger as the foreign object G is farther from the light receiving surface 10A.

Here, the enlargement ratio K is based on the above equation (1),
K = S / SG = (ZS-Z0) ² / (ZS-ZG) ² (2)
Led by.

Further, since the Z coordinate values Z0 and ZS are known values, the Z coordinate value ZG of the position where the foreign matter exists can be specified from the magnification K, but the Z coordinate value ZG of the foreign matter G is the light receiving surface 10A. Z coordinate value Z0, Z coordinate value Z1 of front surface 12A of cover glass 12, and Z coordinate value Z2 of back surface 12B are limited to any one of three values. It is close to the value. That is, the magnification K when the foreign matter G is present on the light receiving surface 10A is K0, the magnification K when the foreign matter G is present on the front surface 12A of the cover glass 12 is K1, and the foreign matter G is present on the rear surface 12B of the cover glass 12. In this case, when the enlargement ratio K is K2, the following values are obtained.
K0 = (ZS−Z0) ² / (ZS−Z0) ² = 1
K1 = (ZS−Z0) ² / (ZS−Z1) ²
K2 = (ZS−Z0) ² / (ZS−Z2) ²
(K0 <K2 <K1)
Therefore, first, in the photographed image P2, the range of the XY coordinate values of the dark area where the pixel value is smaller than the predetermined threshold is detected, and the photographed image P1 formed by the same foreign matter is detected for the detected dark area. Correlate the dark areas. For example, when there is only one foreign object, since there is also one dark area on the captured images P1 and P2, they are associated with each other, and when there are multiple foreign objects, on the captured image P2. The dark area on the captured image P1 that is closest to each dark area may be associated. In addition, since the arrangement of the dark areas on the captured image P1 formed by each of the plurality of foreign objects is the same as the arrangement of the dark areas on the captured image P2 corresponding to the foreign objects, the arrangement is the same. The dark area on the captured image P1 corresponding to the foreign object can be associated with the dark area on the captured image P2. Thereby, as shown in FIG. 4, each of the dark areas i1 to i3 exemplified on the photographed image P2 and each of the dark areas h1 to h3 exemplified on the photographed image P1 are the same foreign substances G1 to G1. Corresponding as a dark region formed by G3.

  Subsequently, for each dark region on the captured image P2, an enlargement factor K is obtained from the area S of the dark region on the captured image P2 and the area SG of the dark region on the captured image P1 corresponding to the dark region. Then, for the enlargement factor K of each dark region on the photographed image P2 (or the photographed image P1), the closest one of the enlargement factors K0 to K2 is detected. As a result, with respect to the dark region where the enlargement ratio K0 (1) is the closest value, it is possible to specify that the foreign matter forming the dark region is present at the position (Z coordinate value Z0) of the light receiving surface 10A. . For example, the enlargement factor K of the dark region i1 of the photographed image P2 with respect to the dark region h1 of the photographed image P1 illustrated in FIGS. 1B, 2B, and 4 is a value closest to the enlargement factor K0. The foreign object G1 that forms the dark areas h1 and i1 is specified to be present at a position on the light receiving surface 10A.

  For the dark area where the enlargement factor K1 is the closest value, it is possible to specify that the foreign matter forming the dark area exists at the position (Z coordinate value Z1) of the front surface 12A of the cover glass 12. For example, the enlargement factor K of the dark region i2 of the photographed image P2 with respect to the dark region h2 of the photographed image P1 illustrated in FIGS. 1B, 2B, and 4 is a value closest to the enlargement factor K1, The foreign matter G2 that forms the dark regions h2 and i2 is specified to be present at a position on the front surface 12A of the cover glass 12.

  For the dark area where the enlargement factor K2 is the closest value, it can be specified that the foreign matter that formed the dark area exists at the position (Z coordinate value Z2) of the back surface 12B of the cover glass 12. For example, the enlargement factor K of the dark region i3 of the photographed image P2 with respect to the dark region h3 of the photographed image P1 illustrated in FIGS. 1B, 2B, and 4 is a value closest to the enlargement factor K2. The foreign matter G3 that forms the dark regions h3 and i3 is specified to be present at a position on the back surface 12B of the cover glass 12.

  As described above, it is possible to specify the surface on which the foreign matter exists among the light receiving surface 10A of the image sensor 10 and the front surface 12A and the back surface 12B of the cover glass 12, and it is possible to specify the range of the position coordinates of the foreign matter on the surface. The three-dimensional position, size, and shape of the foreign matter can be specified.

  Note that the Z coordinate value of the position where the foreign substance exists may be obtained using the above formula (2) with respect to the enlargement ratio K obtained from the dark region. Further, the enlargement factor K is a value obtained by dividing the area of the dark region on the photographed image P2 formed by the foreign substance by the area of the dark region on the photographed image P1, but instead of the area, a specific direction (X The enlargement ratio may be obtained as a length such as the maximum width in the axial direction or the Y-axis direction.

  Next, the second foreign matter detection method will be described. Like the first foreign matter detection method, when the photographed image P1 is captured in the reference illumination state of FIG. 1A, illumination as shown in FIG. Switch to state (third illumination state). In the third illumination state, as shown in FIG. 5A, parallel light in a direction oblique to the optical axis O by a predetermined angle is irradiated onto the light receiving surface 10A of the image sensor 10 as illumination light. For example, the light receiving surface 10A is irradiated with parallel light at a predetermined incident angle θ with the XZ coordinate plane as the incident surface. The parallel light is obtained by inclining the optical axis O ′ of the point light source 20 and the collimator lens 30 (matching the traveling direction of the parallel light) in FIG. It is done.

  At this time, on the light receiving surface 10 </ b> A of the image sensor 10, there are a bright area where the illumination light is irradiated without being blocked by the foreign object and a dark area where the illumination light is not irradiated due to the foreign object. FIG. 6A shows dark regions g31 to g33 that are not shaded by the foreign substances G1 to G3 shown in FIG. As in the case of the reference illumination state, the XY coordinate value range of the dark region matches the XY coordinate value range of the foreign matter forming the dark region.

  When an image of the light receiving surface 10A is picked up by the image pickup element 10 in this state, a picked-up image P3 as shown in FIG. FIG. 4B shows dark regions j1 to j3 on the captured image P3 corresponding to the dark regions g31 to g33 on the light receiving surface 10A formed by the foreign substances G1 to G3.

Here, the position of the dark area on the light receiving surface 10A, that is, the position of the dark area on the photographed image P3 is a position where the foreign substance exists with respect to the position of the dark area formed on the photographed image P1 by the same foreign substance. Shift in the X-axis direction by an amount corresponding to the Z coordinate value. As shown in FIG. 6, the shift amount ΔX is such that the Z coordinate value of the light receiving surface 10A is Z0, the Z coordinate value of the position where the foreign object G is present is ZG, and the parallel light inclination angle (incident angle on the light receiving surface 10A). Is θ, the following equation (3),
ΔX = (ZG−Z0) · tan θ (3)
Is required.

  Therefore, the shift amount ΔX increases as the foreign object G is farther from the light receiving surface 10A.

Since θ and Z0 are known values, the Z coordinate value ZG of the position where the foreign substance exists can be specified from the shift amount ΔX, but the Z coordinate value ZG of the foreign substance G is the Z coordinate of the light receiving surface 10A. Since it is limited to any one of the value Z0, the Z coordinate value Z1 of the front surface 12A of the cover glass 12, and the Z coordinate value Z2 of the back surface 12B, the shift amount ΔX is close to any of the three values. Value. That is, the shift amount ΔX when the foreign object G is present on the light receiving surface 10A is ΔX0, the shift amount ΔX when the foreign object G is present on the front surface 12A of the cover glass 12 is ΔX1, and the foreign material G is present on the rear surface 12B of the cover glass 12. When the shift amount ΔX is ΔX2, the following values are obtained.
ΔX0 = (Z0−Z0) · tan θ = 0
ΔX1 = (Z1-Z0) · tanθ
ΔX2 = (Z2−Z0) · tanθ
(ΔX0 <ΔX2 <ΔX1)
Therefore, first, in the photographed image P3, the range of the XY coordinate values of the dark area where the pixel value is smaller than a predetermined threshold is detected, and the photographed image P1 formed by the same foreign matter is detected for the detected dark area. Correlate the dark areas. This association can be performed in the same manner as in the first foreign object detection method. Thereby, as shown in FIG. 7, each of the dark areas j1 to j3 illustrated on the captured image P3 and each of the dark areas h1 to h3 illustrated on the captured image P1 are the same foreign matter G1 to G1. Corresponding as a dark region formed by G3.

  Subsequently, for each dark region on the captured image P3, a shift amount ΔX in the X-axis direction with respect to the position of the corresponding dark region on the captured image P1 is obtained. Then, the shift amount ΔX of each dark region on the captured image P2 (or the captured image P1) is detected as the closest one of the shift amounts ΔX0 to ΔX2. As a result, for the dark region where the shift amount ΔX0 has the closest value, it is possible to specify that the foreign matter forming the dark region is present at the position (Z coordinate value Z0) of the light receiving surface 10A. For example, the shift amount ΔX of the dark region j1 of the captured image P3 with respect to the dark region h1 of the captured image P1 illustrated in FIGS. 1B, 5B, and 7 is closest to the shift amount ΔX0 (0). The foreign matter G1 forming the dark regions h1 and j1 is specified as being present at a position on the light receiving surface 10A.

  For the dark region where the shift amount ΔX1 is the closest value, it can be specified that the foreign matter forming the dark region is present at the position (Z coordinate value Z1) of the front surface 12A of the cover glass 12. For example, the shift amount ΔX of the dark region j2 of the captured image P3 with respect to the dark region h2 of the captured image P1 illustrated in FIGS. 1B, 5B, and 7 is a value closest to the shift amount ΔX1. The foreign object G2 that forms the dark areas h2 and j2 is specified to be present at a position on the front surface 12A of the cover glass 12.

  With respect to the dark region where the shift amount ΔX2 is the closest value, it can be specified that the foreign matter forming the dark region is present at the position (Z coordinate value Z2) of the back surface 12B of the cover glass 12. For example, the shift amount ΔX of the dark region j3 of the captured image P3 with respect to the dark region h3 of the captured image P1 illustrated in FIGS. 1B, 5B, and 7 is a value closest to the shift amount ΔX2. The foreign matter G3 that forms the dark regions h3 and j3 is specified to be present at a position on the back surface 12B of the cover glass 12.

  As described above, it is possible to specify the surface on which the foreign matter exists among the light receiving surface 10A of the image sensor 10 and the front surface 12A and the back surface 12B of the cover glass 12, and it is possible to specify the range of the position coordinates of the foreign matter on the surface. The three-dimensional position, size, and shape of the foreign matter can be specified.

  FIG. 8 is a configuration diagram showing the overall configuration of the foreign object detection device according to the present invention, and is a configuration diagram of the foreign object detection device that enables foreign object detection by the first and second foreign object detection methods.

  As shown in the figure, the foreign object detection device includes an image sensor package mounting unit 50 (simply referred to as the mounting unit 50), a light source unit 60, and a control / detection unit 80.

  The mounting unit 50 is a component for detachably mounting the image sensor package 2 before being incorporated into a predetermined product as an inspection target, and has a connection terminal to which a signal line from the control / detection unit 80 is connected, for example. These connection terminals are sockets that hold the connection terminals connected to the corresponding connection terminals 14B, 14B,.

  The image pickup device package 2 includes, for example, an image pickup device (chip) 10 such as a CCD or a CMOS, a package main body 14 having a concave housing portion 14A, a cover glass 12 that closes an opening of the housing portion 14A, and the like. . Note that the image pickup device 10 and the cover glass 12 of the image pickup device package 2 in the figure correspond to the image pickup device 10 and the cover glass 12 shown in FIGS.

  The image sensor 10 is housed in the housing portion 14A of the package body 14 and is fixed with the light receiving surface 10A facing the opening side of the housing portion 14A. The package main body 14 is provided with connection terminals (pins) 14B, 14B,... That are communicated from the housing portion 14A and exposed to the outside, and the connection terminals 14B and the bonding pads on the image sensor 10 are provided. Bonded by wire bonding or the like. And the cover glass 12 is affixed on the package main body 14 so that the opening of the accommodating part 14A may be closed, and the inside of the accommodating part 14A is sealed (sealed).

  The mounting unit 50 holds the light receiving surface 10 </ b> A of the image sensor 10 of the image sensor package 2 with the light source unit 60 facing toward the light source unit 60. In the following description of the configuration diagram of FIG. 5, the image pickup device package 2 refers to the image pickup device package 2 to be inspected attached to the attachment portion 50.

  The light source unit 60 is a component for irradiating the imaging device package 2 with illumination light, and corresponds to the components shown in FIGS. 1 (A), 2 (A), and 5 (A). The light source 20 and the collimator lens 30 and a drive mechanism 66 for changing the illumination state.

  The point light source 20 is a light source that emits diffused light that diffuses radially from a sufficiently small region that can be regarded as a point toward the collimator lens 30, and includes, for example, a light source (lamp) 62 having an arbitrary light emitting area and a circular minute opening. And a diaphragm (or aperture) 64 having a (hole). As a result, of the light emitted from the light source 62, the light that has passed through the aperture of the diaphragm 64 enters the collimator lens 30 as diffused light that diffuses radially with the position of the aperture as the position of the point light source.

  The collimator lens 30 is a lens for converting the diffused light from the point light source 20 into parallel light, and uses the light emitted from the point light source 20 as parallel illumination light as the imaging element package 2 (the light receiving surface 10A of the imaging element 10). ).

  The drive mechanism 66 is a means for changing the illumination state of the light source unit 60. In the present apparatus, when the first foreign matter detection method described above is employed, the drive mechanism 66 collimates the optical path through which the illumination light emitted from the point light source 20 and applied to the image sensor package 2 passes. A mechanism for inserting and removing 30 is shown. At this time, when the reference illumination state of FIG. 1A is set, the drive mechanism 66 sets the collimator lens 30 to be inserted in the optical path as shown in FIG. In FIG. 8, the optical axis O ′ of the light source unit 60 parallel to the direction in which the parallel light emitted from the collimator lens 30 travels is arranged in parallel to the optical axis O of the image sensor 10 of the image sensor package 2. The parallel light emitted as the illumination light from the light source unit 60 is parallel to the optical axis O of the image sensor 10 and is irradiated perpendicularly to the light receiving surface 10A.

  On the other hand, in the case of the second illumination state of FIG. 2A, the drive mechanism 66 sets the collimator lens 30 in the extracted state (withdrawn) from the optical path as shown in FIG. Thereby, as shown in FIG. 1A, a reference illumination state in which parallel light incident perpendicularly to the light receiving surface 10A of the image sensor 10 of the image sensor package 2 is irradiated from the light source unit 60 as illumination light, As shown in FIG. 2A, the light receiving surface 10A of the image sensor 10 can be switched between the second illumination state in which diffused light that diffuses radially from the point light source 20 is irradiated from the light source unit 60 as illumination light. it can.

  On the other hand, in the present apparatus, when the second foreign matter detection method described above is employed, the drive mechanism 66 is a light source centered on an axis orthogonal to the optical axis O located in the vicinity of the image sensor 10, for example. The mechanism which rotates the part 60 (the point light source 20 and the collimator lens 30) integrally is shown. At this time, in the case of the reference illumination state of FIG. 1A, the drive mechanism 66 makes the optical axis O ′ of the light source unit 60 parallel to the optical axis O of the image sensor 10 as shown in FIG. Set to state.

  On the other hand, in the case of the third illumination state in FIG. 5A, the drive mechanism 66 sets the optical axis O ′ of the light source unit 60 to be predetermined with respect to the optical axis O of the image sensor 10 as shown in FIG. 10. Set the angle tilted. Thereby, as shown in FIG. 1A, a reference illumination state in which parallel light incident perpendicularly to the light receiving surface 10A of the image sensor 10 of the image sensor package 2 is irradiated from the light source unit 60 as illumination light, As shown in FIG. 5A, switching can be performed between the third illumination state in which parallel light incident obliquely on the light receiving surface 10A of the image sensor 10 is irradiated from the light source unit 60 as irradiation light.

  The drive mechanism 66 may be provided with a mechanism that varies the distance between the light source unit 60 and the image pickup device package 2. Further, the switching of the illumination state of the light source unit 60 by the drive mechanism 66 may be performed manually (manual force), but in the present embodiment, it is performed electrically by driving a motor.

  The control / detection unit 80 performs control of the light source unit 60, control of the image sensor 10 of the image sensor package 2, data processing, and the like. In order to perform these controls and processes, the control / detection unit 80 includes a system control unit 100, an operation unit 82, a display unit 84, a light source control unit 86, an image sensor driving unit 88, an image capturing unit 90, and an image memory 92. , A foreign matter detection unit 94, a foreign matter information storage unit 96, and the like.

  Each component in the control / detection unit 80 is controlled under the control of the system controller 100 connected to each component via the system bus 101 and operates. The user's operation performed on the unit 82 is detected, and various processes are executed according to the operation, and each component unit is caused to execute a required process. The display unit 84 displays guidance information for guiding necessary operations to the user, foreign object detection result information, and the like.

  Control of the light source unit 60 is performed via the light source control unit 86 in accordance with a control signal from the system control unit 100. By switching on / off the supply of power to the light source 62, the light source 62 is controlled. Is switched on / off. In addition, the illumination state of the light source unit 60 is switched as described above by controlling the motor of the drive mechanism 66. The light source 62 may be turned on and off by operating the operation unit 82, or may be interlocked with power on / off of the entire apparatus. The illumination state of the light source unit 60 is switched according to the operation of the operation unit 82 when the user manually performs a series of foreign object inspection operations, and when the series of foreign object inspection operations are automatically performed. Is performed according to an automatic execution program.

  The image sensor 10 of the image sensor package 2 is controlled via the image sensor drive unit 88 in accordance with a control signal from the system control unit 100, and the imaging operation (exposure and accumulated charge of the image sensor 10) is controlled. A drive pulse or the like necessary for discharge is supplied from the image sensor driving unit 88 to the image sensor 10 of the image sensor package 2 via the connection terminal of the mounting unit 50, thereby causing the image sensor 10 to perform an image capturing operation. . Thus, the image on the light receiving surface 10A of the image sensor 10 is photoelectrically converted by the light receiving element of each pixel, and the pixel signal of each pixel is output from the image sensor 10 as an imaging signal.

  The imaging signal output from the imaging element 10 is captured by the image capturing unit 90 via the connection terminal of the mounting unit 50. In the image capturing unit 90, a required analog process (such as noise removal) is performed on the image signal from the image sensor 10, and then the analog signal is converted into a digital signal. The image data is stored in the image memory 92 as image data of the photographed image via the system bus 101.

  Thereby, the captured images P1, P2, and P3 obtained by capturing the image of the light receiving surface 10A of the image sensor 10 in the reference illumination state of FIG. 1, the second illumination state of FIG. 2, and the third illumination state of FIG. can do. Note that the imaging operation of the imaging device 10 and the acquisition of the captured image are performed according to the operation (shutter operation) of the operation unit 82 when the user manually performs a series of foreign substance inspection operations. When this operation is automatically performed, it is performed according to an automatic execution program.

  The foreign object detection unit 94 executes a foreign object detection process in accordance with an instruction from the system control unit 100. The foreign object detection process in the foreign object detection unit 94 reads the captured image stored in the image memory 92, executes the process according to the foreign object detection shown in the first or second foreign object detection method, and Foreign matter present on the light receiving surface 10A of the image pickup device 10 of the package 2 and the surface of the cover glass 12 (front surface 12A and back surface 12B) is detected, and the position, size, and shape of the foreign material are specified. Then, foreign matter information indicating the position, size, and shape of the identified foreign matter is stored in the foreign matter information storage unit 96.

  The foreign substance information stored in the foreign substance information storage unit 96 is written in a memory in the product so that it can be used in the product when the imaging device package 2 to be inspected is incorporated in a product such as a digital camera. Therefore, it is desirable that data communication between the device that writes data in the memory and this device can be performed so that the foreign material information stored in the foreign material information storage unit 96 can be transferred. However, the means for writing the foreign substance information obtained by this apparatus into the memory of a product in which the image sensor package to be inspected is incorporated is not limited to this, and any method may be used.

  FIG. 11 shows a case in which foreign matter detection is automatically performed using the first foreign matter detection method (when foreign matter detection automatic execution processing is performed) for a series of foreign matter detection operations and processing procedures in the foreign matter detection device. This will be described with reference to a flowchart.

  In step S <b> 10, first, the imaging element package 2 to be inspected is mounted on the mounting unit 50. Then, the process proceeds to step S12.

  In step S <b> 12, the user instructs automatic execution of foreign object detection by operating the operation unit 82. As a result, the process proceeds to step S14, and thereafter, a series of operations and processes for foreign object detection are executed by the system control unit 100 in accordance with the automatic execution program.

  In step S14, the light source unit 60 is set to the reference illumination state shown in FIGS. 1A and 8, and the light source 62 is turned on. Then, the process proceeds to step S16.

  In step S <b> 16, the imaging device 10 of the imaging device package 2 is caused to perform an imaging operation, and a captured image P <b> 1 obtained by capturing an image of the light receiving surface 10 </ b> A is acquired and stored in the image memory 92. Then, the process proceeds to step S18. Note that the captured image P1 acquired here corresponds to the captured image P1 shown in FIG.

  In step S18, the light source unit 60 is set to the second illumination state shown in FIGS. Then, the process proceeds to step S20.

  In step S <b> 20, the imaging device 10 of the imaging device package 2 is caused to perform an imaging operation, and a captured image P <b> 2 obtained by capturing an image of the light receiving surface 10 </ b> A is acquired and stored in the image memory 92. Then, the process proceeds to step S22. Note that the captured image P2 acquired here corresponds to the captured image P2 shown in FIG.

  In step S <b> 22, the captured images P <b> 1 and P <b> 2 stored in the image memory 92 are read by the foreign matter detection unit 94, and thereafter, foreign matter detection processing in the foreign matter detection unit 94 is executed. First, a dark region caused by a foreign object is detected in each of the captured images P1 and P2. Then, the process proceeds to step S24.

  In step S24, the dark area detected in the captured image P1 and the dark area detected in the captured image P2 are associated with the dark area caused by the same foreign matter. Then, the process proceeds to step S26.

  In step S26, the above-described enlargement factor K is calculated for each foreign object based on the dark region associated with the captured images P1 and P2. Then, the process proceeds to step S28.

  In step S28, on the basis of the enlargement factor K calculated in step S26, the surface of the light receiving surface 10A of the image sensor 10 and the front surface 12A and the back surface 12B of the cover glass 12 on which foreign matter is present is identified and foreign matter on the surface. The position coordinate range is specified, and the three-dimensional position, size, and shape of each foreign object are specified. Then, the process proceeds to step S30.

  In step S30, foreign object information indicating the position of each foreign object specified in step S28 is stored in the foreign object information storage unit 96, and the foreign object detection automatic execution process is terminated. Note that the foreign substance information may be displayed on the display unit 84.

  The above foreign substance detection automatic execution processing shows the case where the first foreign object detection method is used. However, even when the second foreign object detection method is used, the foreign object detection process is performed in substantially the same manner as the above flowchart. Work and processing are performed. However, in step S18, instead of setting the light source unit 60 to the second illumination state, the third illumination state shown in FIGS. 5A and 10 is set, and the captured image P2 acquired in step S20 is used. The captured image is acquired as a captured image corresponding to the captured image P3 in FIG. In step S26, instead of calculating the enlargement factor K, a shift amount ΔX is calculated. In step S28, the position of each foreign object is specified based on the shift amount.

  Further, when the user manually performs a series of foreign object detection operations, the processing of steps S14 to S20 is performed without instructing automatic execution of movement detection in step S12 in the flowchart of FIG. After performing in accordance with an instruction from 82, the foreign object detection processing by the foreign object detection unit 94 from step S22 to step S30 may be started by an instruction from the operation unit 82.

  According to the above foreign object detection device, when a foreign object exists on any of the light receiving surface 10A of the image sensor 10 of the image sensor package 2, the front surface 12A of the cover glass 12, and the back surface 12B of the cover glass 12, the foreign object is imaged. The three-dimensional position, size, and shape of the foreign object can be specified by specifying not only the position on the light receiving surface 10A of the element 10 but also the distance from the light receiving surface 10A. That is, it is possible to specify the surface where the foreign matter exists among the light receiving surface 10A of the image sensor 10 and the front surface 12A and the back surface 12B of the cover glass 12, and it is possible to specify the position coordinate range of the foreign matter on the surface.

  When the image pickup device package 2 is incorporated into a product, the foreign matter has an adverse effect on the captured image picked up by the image pickup device 10 as a defect area, but the degree of influence is arranged in front of the image pickup device package 2. This depends on not only the state of the optical system (focal length, aperture value, etc.), the size of the foreign material, etc., but also the distance from the light receiving surface 10A at the position where the foreign material exists. Therefore, when the image of the defective area is corrected using the peripheral image or the like, if the distance from the light receiving surface 10A where the foreign substance is present is unknown, the correction is appropriately made according to the state of the optical system. Can not do it.

  On the other hand, by identifying the three-dimensional position, size, and shape of a foreign substance as in the above foreign substance detection apparatus and using the information in a product incorporating the image pickup element package 2, the front of the image pickup element package 2 is used. According to the state of the arranged optical system, it is possible to appropriately grasp the influence of the foreign matter on the captured image, and it is possible to appropriately correct the image of the defect area formed in the captured image by the foreign matter. For example, when the aperture is included in the optical system arranged in front of the image sensor package 2 and the aperture value is small (when the aperture amount of the aperture is large), it is far from the light receiving surface 10A of the image sensor 10. The foreign matter (foreign matter existing on the front surface 12A of the cover glass 12) has almost no influence on the captured image, and when correcting the image of the defective area, the foreign matter can be excluded. On the other hand, when the aperture value is large (when the aperture amount of the aperture is small), even a foreign object that is far from the light receiving surface 10A of the image sensor 10 (a foreign object existing on the front surface 12A of the cover glass 12) is given to the captured image. When the influence is increased and the image of the defective area is corrected, it can be determined that the foreign matter is also taken into consideration. Then, the defect area formed by the foreign substance at that time is accurately grasped in consideration of the three-dimensional position, size, and shape of the foreign substance (and the state of the optical system is taken into consideration), and the defective area The image can be corrected appropriately.

  As described above, in the above embodiment, in the second illumination state in the first foreign object detection method and the third illumination state in the second foreign object detection method, the former is taken when the point light source 20 is arranged at a predetermined position. Based on the image P2, in the latter, the position, size, and shape of the foreign matter (hereinafter referred to as the position of the foreign matter) are specified based on the captured image P3 when the parallel light is irradiated at a predetermined angle. There may be a case where the position of the foreign matter cannot be detected properly due to a cause such as overlapping of dark regions on the light receiving surface 10A due to a plurality of foreign matters existing at different distances from the light receiving surface 10A.

  Therefore, in the former case, the position of the point light source 20 may be changed, and the captured image P2 may be acquired at the position of the point light source 20 where the dark regions due to a plurality of foreign objects do not overlap. Also, a captured image P2 is obtained when the position of the point light source 20 is arranged at each of a plurality of predetermined positions, and a foreign object is obtained based on each captured image P2 (and the captured image P1 in the reference illumination state). May be specified, and the positions of all foreign substances may be specified by integrating them.

  In the latter case, the captured image P2 may be acquired at an angle of parallel light that does not overlap dark regions due to a plurality of foreign substances so that the angle of irradiating parallel light onto the light receiving surface 10A can be changed. Further, a captured image P3 is obtained when the angle at which the parallel light is irradiated is set to each of a plurality of predetermined angles, and each captured image P3 (and the captured image P1 in the reference illumination state) is acquired. The positions and the like of foreign objects may be specified and integrated to specify the positions and the like of all foreign objects.

  In the above embodiment, the parallel light is irradiated perpendicularly to the light receiving surface 10A of the image sensor 10 in the reference illumination state shown in FIGS. 1 (A) and 8. In any of the two foreign matter detection methods, it is not always necessary to irradiate the light receiving surface 10A perpendicularly with parallel light as a reference illumination state. That is, even when parallel light is obliquely applied to the light receiving surface 10A in the reference illumination state, by detecting the range of the XY coordinate value of the dark region of the captured image P1 captured at that time, The foreign matter that has formed the dark region is a position in front of the light receiving surface 10A, and within the region when the range of the dark region on the light receiving surface 10A is extended in the direction of parallel light (with the dark region on the light receiving surface as the range). On the other hand, it can be specified that the foreign object exists in the same direction and in the same direction as the dark region in the parallel light direction.

  In the first foreign object detection method, the distance from the light receiving surface 10A at the position where the foreign object exists is obtained by obtaining the magnification K using the captured image P1 and the captured image P2 captured in the second illumination state. Can be specified. Therefore, the three-dimensional position, size, and shape of the foreign object can be specified from these specified contents.

  In the second foreign object detection method, the photographed image P1 and the photographed image P3 photographed in the third illumination state in which the light receiving surface 10A is irradiated with parallel light at an angle different from the parallel light in the reference illumination state are used. Thus, by obtaining the above-mentioned shift amount ΔX, it is possible to specify in which range the foreign matter exists in the region where the foreign matter specified by the captured image P1 can exist. That is, an area in the direction of parallel light in the reference illumination state with respect to a dark area range on the light receiving surface 10A obtained from the photographed image P1 and a dark area range on the light receiving surface 10A obtained from the photographed image P3. On the other hand, the region where the region that is the direction of the parallel light in the third illumination state intersects can be obtained from the shift amount ΔX, and foreign matter exists in the intersecting region. Therefore, the three-dimensional position, size, and shape of the foreign object can be specified.

  In the above-described embodiment, the image sensor package before being incorporated into the product is the object of foreign matter detection. However, even in the image sensor package after being incorporated into the product, the above-described reference illumination state and second It is possible to illuminate the light receiving surface of the image sensor by switching between the illumination state of the image sensor or by switching between the reference illumination state and the third illumination state. If the foreign object detection process is executed in such a manner that it can be captured, the object can be detected.

  In the above embodiment, the image pickup device package is a target for foreign object detection. However, if the cover glass is arranged in front of the light receiving surface of the image pickup device, the image pickup device package is the same as the above embodiment. A foreign object existing on the light receiving surface or the surface of the cover glass can be detected, and can be a target for detecting a foreign object.

DESCRIPTION OF SYMBOLS 2 ... Image pick-up element package, 10 ... Image pick-up element, 10A ... Light-receiving surface, 12 ... Cover glass, 12A ... Front surface, 12B ... Back surface, 20 ... Point light source, 30 ... Collimator lens, 50 ... Mounting part, 60 ... Light source part, 80 ... Control / detection unit, 100 ... System control unit

Claims (6)

A first illuminating means for irradiating the light receiving surface with parallel light as a first illumination light with respect to a foreign object detection target in which a transparent body is arranged on the front side of the light receiving surface of the image sensor;
Second illuminating means for irradiating the light receiving surface with diffused light diffused from a point light source as second illuminating light;
First photographed image acquisition means for acquiring a first photographed image obtained by photographing the image of the light receiving surface when the first illumination means is irradiated with the first illumination light by the image sensor;
Second captured image acquisition means for acquiring a second captured image obtained by capturing the image of the light receiving surface when the second illumination means is irradiated with the second illumination light by the image sensor;
Based on the first photographed image, the first light is blocked by the foreign matter existing on the light receiving surface and any one of the front surface and the rear surface of the transparent body, and the light reception is shaded by the foreign material. First dark area detecting means for detecting a first dark area on the surface;
A second dark region detecting means for detecting a second dark region on the light receiving surface which is shaded by the foreign matter by blocking the second illumination light based on the second photographed image;
The range of the position coordinates of the first dark area on the light receiving surface detected by the first dark area detecting means, and the second dark area on the light receiving surface detected by the second dark area detecting means. Based on the enlargement ratio with respect to the first dark region, the front surface and the back surface of the transparent body on which the foreign matter exists are specified, and the position coordinate range of the foreign matter on the surface is specified. Foreign matter identification means to
A foreign object detection device for an image sensor comprising:   2. The foreign object detection device for an image sensor according to claim 1, wherein the second illumination unit irradiates the light receiving surface with the light that has passed through the aperture of the diaphragm among the light emitted from a predetermined light source as the second illumination light. . It said first illumination means, the foreign material detecting device of an imaging device according to claim 1 or 2 is irradiated with parallel light perpendicular to the light receiving surface as said first illumination light. The first illumination means converts light that has passed through an aperture of a diaphragm out of light emitted from a predetermined light source into parallel light by a lens, and irradiates the light receiving surface with the parallel light as the first illumination light. Item 4. The foreign object detection device for an image sensor according to any one of Items 1 to 3 . The foreign substance detection target, the foreign material detecting device of an imaging device according to any one of claims 1-4 which is an image pickup element package. A first illumination step of irradiating the light receiving surface with parallel light as first illumination light on the imaging element and a foreign object detection object in which a transparent body is disposed on the front side of the light receiving surface of the image sensor;
A first captured image acquisition step of acquiring a first captured image obtained by capturing the image of the light receiving surface when the first illumination light is irradiated by the first illumination step with the image sensor;
A second illumination step of irradiating the light receiving surface with diffused light diffused from a point light source as second illumination light;
A second captured image acquisition step of acquiring a second captured image obtained by capturing the image of the light receiving surface when the second illumination light is irradiated by the second illumination step with the image sensor;
Based on the first photographed image, the first light is blocked by the foreign matter existing on the light receiving surface and any one of the front surface and the rear surface of the transparent body, and the light reception is shaded by the foreign material. A first dark region detecting step of detecting a first dark region on the surface;
A second dark region detecting step of detecting a second dark region on the light receiving surface which is shaded by the foreign matter by blocking the second illumination light based on the second photographed image;
The range of position coordinates of the first dark region on the light receiving surface detected by the first dark region detecting step and the second dark region on the light receiving surface detected by the second dark region detecting step. Based on the enlargement ratio with respect to the first dark region, the front surface and the back surface of the transparent body on which the foreign matter exists are specified, and the position coordinate range of the foreign matter on the surface is specified. Foreign matter identification process to
A foreign matter detection method for an imaging device comprising:

Images