JP2006121500A - Image processor, method, and imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve disadvantages caused in images due to foreign materials existing in an optical system forming an object image to obtain a clear image immune to the influence of the foreign materials. <P>SOLUTION: A correcting means 14 is constituted so as to correct the local optical quantity of light reduction (signal reduction) appearing on the imaging surface (light receiving surface) of a solid state imaging element 13 due to foreign materials existing on the surface of an optical LPF 12, thus outputting corrected image signals to an analog signal processor circuit. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an imaging apparatus such as a digital camera having an optical member such as a cover glass or an optical LPF (low-pass filter) directly above an imaging element, and an image processing apparatus and method for the imaging apparatus.

  In recent years, an imaging apparatus having a large image sensor and a large optical LPF, such as a single-lens reflex digital still camera, has attracted attention. In such an imaging apparatus, since one pixel is large, a very excellent image can be taken with high resolution.

JP-A-6-98167 JP 2002-44443 A Japanese Patent Laid-Open No. 7-30906

  However, since such a large image pickup device and a large optical LPF are large as elements, defects due to foreign matters such as dust, scratches, and stains are likely to occur in the manufacturing process, and the problem of yield reduction is serious. It is also difficult to provide inexpensive products. Nevertheless, the former imaging element is being improved in the semiconductor process by increasing the degree of cleanliness, and it is becoming possible to steadily produce large-sized ones with few defects at a high yield. On the other hand, the optical element such as the latter optical LPF is not as clean as the semiconductor process in the manufacturing process, and also causes defects due to the inclusion of foreign matters when assembled as a unit including the image pickup element. There are a lot of concerns.

  The present invention has been made in view of the above problems, and it is possible to eliminate the inconvenience caused in the image due to the foreign matter existing in the optical system for forming the subject image and obtain a clean image free from the influence of the foreign matter. It is an object of the present invention to provide an image processing apparatus and method, and an imaging apparatus including the image processing apparatus.

  The image processing apparatus of the present invention is a signal generated in a portion corresponding to the foreign matter in the image signal due to a foreign matter existing in an imaging unit that photoelectrically converts the subject image into an image signal and an optical system that forms the subject image. Correction means for correcting the decrease.

  In one aspect of the image processing apparatus of the present invention, the correction unit changes the correction amount according to the distance between the imaging lens of the optical system and the imaging surface of the imaging unit.

  In one aspect of the image processing apparatus of the present invention, the correction unit changes the correction amount according to the aperture value of the imaging lens of the optical system.

  In one aspect of the image processing apparatus of the present invention, the correction unit includes a position calculation unit that calculates a position of the projection image of the foreign object on the imaging surface of the imaging unit, and a size calculation that calculates the size of the projection image. Means, light intensity calculation means for calculating the light intensity of the projection image, and correction amount calculation means for correcting the signal decrease based on the position and the size of the projection image using the calculated light intensity. Including.

  In one aspect of the image processing apparatus of the present invention, the correction amount calculating means adjusts a correction value by applying a light blocking rate to the light intensity according to a light blocking state of the foreign matter.

  The image pickup apparatus according to the present invention includes an optical system that receives incident light of a subject image, an image pickup unit that photoelectrically converts the subject image into an image signal, and a foreign object existing in the optical system that forms the subject image. Correction means for correcting a signal drop occurring in a portion of the signal corresponding to the foreign substance, signal processing means for processing the corrected image signal, and image display means for displaying an image from the image signal.

  In one aspect of the imaging apparatus of the present invention, the correction unit changes the correction amount according to the distance between the imaging lens of the optical system and the imaging surface of the imaging unit.

  In one aspect of the imaging apparatus of the present invention, the correction unit changes the correction amount according to the aperture value of the imaging lens of the optical system.

  In one aspect of the imaging apparatus of the present invention, the correction unit calculates a position of the projection image of the foreign object on the imaging surface of the imaging unit, and a size calculation unit calculates the size of the projection image. Light intensity calculating means for calculating the light intensity of the projected image, and a correction amount calculating means for correcting the signal decrease based on the position and the size of the projected image using the calculated light intensity. including.

  In one aspect of the imaging apparatus of the present invention, the correction amount calculating means adjusts a correction value by applying a light blocking rate to the light intensity in accordance with a light blocking state of the foreign matter.

  The image processing method of the present invention photoelectrically converts the subject image into an image signal when correcting a signal drop caused in a part corresponding to the foreign matter in the image signal due to a foreign matter existing in an optical system that forms the subject image. Calculating the position of the projected image of the foreign object on the imaging surface of the imaging means to be converted, calculating the size of the projected image, calculating the light intensity of the projected image, and calculating the calculated Correcting the signal drop based on the position and size of the projected image using light intensity.

  In one aspect of the image processing method of the present invention, a correction value is adjusted by applying a light blocking rate to the light intensity in accordance with the light blocking state of the foreign matter.

  According to the present invention, it is possible to eliminate the inconvenience caused in the image due to the foreign matter existing in the optical system for forming the subject image, and to obtain a clean image free from the influence of the foreign matter.

  Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. In the present embodiment, an imaging apparatus such as a digital camera provided with an image processing apparatus is illustrated. FIG. 1 is a block diagram showing a main configuration of the imaging apparatus according to the present embodiment, and FIG. 2 is a block diagram showing a main configuration which is a characteristic part of the imaging apparatus according to the present embodiment.

  The image processing apparatus according to the present embodiment processes an optical system 1 that forms a subject image, an image processing apparatus 2 that performs photoelectric conversion, corrects and outputs a generated image signal, and an analog signal that is an image signal. An analog signal processing circuit 3, an A / D converter 4 for converting an analog signal into a digital signal, a digital signal processing circuit 5 for processing a digital signal, a buffer memory 6 for storing the digital signal, and the entire apparatus (illustrated) In the example, the control circuit 7 for controlling the operations of the image processing device 2, the analog signal processing circuit 3, the A / D converter 4, the digital signal processing circuit 5 and the I / F 8)), and the digital read from the buffer memory An interface (I / F) 8 for transferring signals to the outside (including recording means and the like) and a memory bus 9 are provided.

  The optical system 1 includes an imaging lens 11 that collects an image, and an optical LPF (low-pass filter) 12 that has a coating that cuts infrared rays on the surface. Here, the coating on the surface of the optical LPF 12 is generally called a dichroic surface. In addition to the coating that cuts infrared rays, an anti-reflection coating or the like is deposited, or the imaging element absorbs infrared rays. At the time of manufacture, infrared absorbing glass is pasted so that the infrared light is attenuated, or the optical LPF itself is attached to a three-layer structure with a λ / 4 (1/4 wavelength) plate in between. There is a high possibility that dust will be sandwiched between the respective layers, and it is a member that tends to cause yield defects.

  The image processing apparatus includes a solid-state imaging device 13 that photoelectrically converts an image into an image signal that is an analog signal, and a local image generated on an imaging surface (light-receiving surface) of the solid-state imaging device 13 due to a foreign substance existing on the surface of the optical LPF 12. It comprises correction means 14 that corrects the light quantity reduction (signal reduction) and outputs the corrected image signal to the analog signal processing circuit.

  As shown in FIG. 2, the correction unit 14 calculates the size of the projection image, and the position calculation unit 21 that calculates the position of the projection image of foreign matters such as dust, scratches, and spots on the light receiving surface of the solid-state imaging device 13. Size calculating means 22 for calculating, light intensity calculating means 23 for calculating the light intensity of the projected image, and correction amount calculating means 24 for correcting the decrease in the amount of light based on the position and size of the projected image using the calculated light intensity. And is configured.

Here, the operation principle of the correction means 14 will be described. FIG. 3 is a schematic diagram for explaining a problem that occurs in an image due to foreign matter on the light receiving surface.
As shown in FIG. 3, in the manufacturing process of the optical LPF 12, foreign matter 25 such as dust, scratches, and spots is often mistakenly generated on the surface of the optical LPF 12. The foreign matter may exist between the dichroic and the optical LPF, or may exist between each layer of the dichroic, but there is the highest risk of the foreign matter occurring in the dichroic coating process in which multiple layers are deposited. The light beam 26 that has passed through the imaging lens 11 is blocked by the foreign matter 25, and a projection image 27 corresponding to the foreign matter 25 is formed on the light receiving surface of the solid-state imaging device 13, which becomes a local light amount reduction region. The distance between the light receiving surface of the solid-state imaging device 13 and the surface of the optical LPF 12 is d in terms of air, and the distance between the pupil of the imaging lens 11 and the solid-state imaging device 13 is P ₀ (pupil distance).

FIG. 4 is a schematic diagram for qualitatively explaining the relationship between the components shown in FIG.
4A is basically the same as FIG. 3, but an optical axis 28 is added. As shown in FIG. 4B, when the aperture of the optical system 1 is reduced, the diameter of the projected image 27 of the foreign object 25 is reduced, but a dark shadow with a clear intensity is correspondingly increased. Further, the center of gravity position of the projection image 26 is not changed from FIG. As shown in FIG. 4D, when the aperture is opened, the center of gravity position remains unchanged, the base of the projected image 27 of the foreign object 25 is enlarged, the blur is widened, the intensity is reduced, and a thin shadow is formed.

  On the other hand, as shown in FIG. 4C, when the pupil of the imaging lens 11 is brought closer without changing the aperture, the position of the center of gravity of the projected image 26 is moved away from the optical axis according to the image height (distance from the optical axis). The projected image 26 moves to the position. As shown in FIG. 4E, when the pupil of the imaging lens 11 is moved away, the projected image 26 approaches the optical axis according to the image height. The larger the image height, the larger the amount of change. As described above, the shape of the projection image 26 is changed by changing the aperture, and the position of the center of gravity of the projection image 26 is moved by changing the pupil distance.

5 to 7 are schematic diagrams for quantitatively explaining the relationship between the components shown in FIG.
5 shows the coordinates of the projection image of the foreign matter, FIG. 6 shows the shape of the projection image, and FIG. 7 shows the light intensity of the projection image.

First, the position of the projected image of the foreign matter will be described with reference to FIG.
Here, if the image height that is the distance of the foreign object 25 from the optical axis 28 is h ₀ and the image height of the dust shadow is h, h = h ₀ * P ₀ / (Po−d) due to the similarity of triangles. (1)
It is expressed.

In an imaging apparatus such as a digital camera, since the position of the foreign matter is fixed at the time of manufacture, h0 and d are fixed. Therefore, the image height h representing the position of the projection image 27 of the foreign object 25 is uniquely determined according to the change in the pupil distance Po of the imaging lens 11. If this calculation is performed according to Po, the position of the projection image 27 can be specified. In order to facilitate understanding in the formula (1), the image height from the optical axis 28 is discussed, but actually, the optical axis 28 is the origin, and the position of the foreign object 25 and the projected image 27 is expressed by the x and y coordinates. Just use it and you can use it. That is, assuming that the position of the foreign material 25 is h ₀ (x _0, y ₀ ) and the position of the projection image 27 is h (x, y), x = x ₀ * P ₀ / (P ₀ -d) (2)
_{y = y 0 * P 0 /} (P 0 -d) (3)
Can be expressed.

Next, the size of the projected image of the foreign matter will be described with reference to FIG.
Here, the diameter of the projection image 27 is R, the diameter of the foreign material 25 is p, and the aperture value of the imaging lens 11 is Fno.
The oblique angle of entry of the light beam changes according to the diaphragm of the imaging lens 11, but FIG. 6 shows the outermost light beam 26 that passes through the diaphragm. The aperture value Fno is a representation of the solid angle of the outermost luminous flux 26 that passes through the aperture as a ratio of triangles, so between R, d, and Fno,
R = d / Fno (4)
The relationship holds.

  Originally, the diameter p of the foreign matter 25 is also related, but in reality, it is not manufactured in a quality control state in which a large foreign matter enters the manufacturing process so that the diameter p is affected. The diameter p can be ignored numerically. If the foreign matter 25 is large, the foreign matter 25 can be regarded as a point-like shadow superimposed as an aggregate of points, and therefore the conversion is extremely easy. This is the same even when the foreign object has a complicated shape, and since the shape is an aggregate of points, the shadow of the foreign object may be superimposed and integrated on both x and y axes in accordance with the foreign object shape. .

Next, the light intensity of the projected image of the foreign matter will be described with reference to FIG.
Assuming that the diameter of the foreign material 25 is p, since p is a dot and small, the diameter d / Fno of the projection image 27 is hardly affected, but the light intensity of the projection image 27 is not negligible.

Here, it is based on the idea that the light blocked by the foreign material 25 is spread and thinned to the shape of the shadow spread d / Fno of the foreign material 25. Therefore, the light quantity reduction rate δ is the ratio of the areas of both.
δ = (Fno * p / d) ² (5)
It can be expressed.

While the spot-like foreign matter is small like pollen, the light intensity (light quantity ratio) γ by the projected image 27 of the foreign matter 25 is γ = 1− (Fno), assuming that the light shielding rate is 100% in order to shield 100%. * P / d) ² (6)
It can be expressed.

Here, in the case of a type of optical defect such as peeling of the coating on the surface of the optical LPF 12, if the light shielding state is taken into consideration, the light shielding rate is k (0 ≦ k ≦ 1).
γ = 1−k * (Fno * p / d) ² (7)
It can be expressed. When 100% light is blocked by a foreign object, k = 1 may be set.

Hereinafter, an image processing method using the imaging apparatus (image processing apparatus) according to the present embodiment will be described with reference to FIG.
A correction processing routine is started (step S1). First, the position calculating unit 21 of the correcting unit 14 determines the position (coordinates (x, y)) of the projected image 27 of the foreign object 25 on the light receiving surface of the solid-state imaging device 13 based on the equations (2) and (3). Is calculated (step S2).

  Subsequently, the size calculation unit 22 of the correction unit 14 calculates the size of the projection image 27 (the diameter R of the projection image 27) based on the equation (4) (step S3). Subsequently, the light intensity calculation unit 23 of the correction unit 14 calculates the light amount ratio γ of the projection image 27 based on the equation (7) (step S4).

  Then, the correction amount calculation unit 24 of the correction unit 14 outputs the reciprocal of the light amount ratio γ including the decrease due to light shielding in the equation (7) from the solid-state imaging device 13 over the region of the projection image 27. By multiplying the signal by a gain, the signal decrease due to the projection image 27 is compensated (step S5), and the correction processing routine is ended (step S6).

  In FIG. 8, the image processing method is expressed by software. However, if the content is such a level, correction by hardware is also possible. Even when the foreign matter 25 is large or has a different shape, the shape of the projected image 27 of the foreign matter 25 can be specified by superimposing and integrating with the coordinates (x, y), and the strength of the foreign matter 25 can be specified. And the area of the projected image 27 can be specified. As for the quality of the foreign matter 25, if it is translucent, it can be dealt with by appropriately selecting the k value in the equation (7). Even if the foreign object 25 is located in a place relatively close to the solid-state image sensor 13 such as a cover glass of the package of the solid-state image sensor 13, the expressions (2), (3), (4) and This can be easily handled by simply changing the d value in equation (7).

  In the present embodiment, gain correction is performed according to the light intensity of the projection image 27 in the expression (7). However, when the projection image 27 is small or when the distance between the foreign object 25 and the projection image 27 is small. It is also possible to perform so-called defect correction that complements a corresponding pixel from normal image signals of surrounding pixels.

  In the present embodiment, the image signal is corrected using the correction unit 14 in order to remove the influence of the projection image 27 caused by the foreign matter 25. With this configuration, it is possible to effectively correct the projection image 27 that changes depending on the pupil distance and the diaphragm, that is, the light amount decrease. That is, by storing the coordinates, the light shielding rate, and the distance from the imaging surface in the imaging device before the factory shipment, the foreign matter 25 that is erroneously generated in the optical system 11 at the time of manufacturing the imaging device at the factory. Any optical surface, any material sheet (x, y), or any size can be corrected appropriately and in real time according to the pupil distance and aperture value of the imaging lens 11. be able to.

  As described above, according to the imaging apparatus (image processing apparatus) of the present embodiment, the inconvenience caused in the image due to the foreign matter 25 existing in the optical system 1 that forms the subject image is eliminated, and the influence of the foreign matter 25 is reduced. It is possible to obtain a clean image without any problem.

  A program stored in a RAM, a ROM, or the like of a computer operates in each mechanism constituting the image processing apparatus shown in FIG. 2 and each step constituting the image processing method according to the present invention shown in FIG. Can be realized. This program and a computer-readable storage medium storing the program are included in the embodiment of the present invention.

  Specifically, the program is recorded on a recording medium such as a CD-ROM or provided to a computer via various transmission media. As a recording medium for recording the program, besides a CD-ROM, a flexible disk, a hard disk, a magnetic tape, a magneto-optical disk, a nonvolatile memory card, or the like can be used. On the other hand, as the transmission medium of the program, a communication medium (wired line such as an optical fiber, etc.) in a computer network (LAN, WAN such as the Internet, wireless communication network, etc.) system for propagating and supplying program information as a carrier wave A wireless line or the like.

  In addition, the functions of the above-described embodiments are realized by executing a program supplied by a computer, and the program is used in cooperation with an OS (operating system) or other application software running on the computer. When the functions of the above-described embodiment are realized, or when all or part of the processing of the supplied program is performed by a function expansion board or a function expansion unit of the computer, the function of the above-described embodiment is realized. Such a program is included in the embodiment of the present invention.

  For example, FIG. 9 is a schematic diagram illustrating an internal configuration of a general personal user terminal device. In FIG. 9, reference numeral 1200 denotes a computer PC. The PC 1200 includes a CPU 1201, executes device control software stored in the ROM 1202 or the hard disk (HD) 1211, or supplied from the flexible disk drive (FD) 1212, and collects all devices connected to the system bus 1204. To control.

  The functions stored in the CPU 1201 of the PC 1200, the ROM 1202, or the hard disk (HD) 1211 realize the functions of the respective means such as the means 1 to 5 in the present embodiment and the procedures such as steps 1 to 5.

  A RAM 1203 functions as a main memory, work area, and the like for the CPU 1201. A keyboard controller (KBC) 1205 controls instruction input from a keyboard (KB) 1209, a device not shown, or the like.

  Reference numeral 1206 denotes a CRT controller (CRTC) which controls display on a CRT display (CRT) 1210. A disk controller (DKC) 1207 is a hard disk (boot program (start program: a program that starts execution (operation) of personal computer hardware and software)), a plurality of applications, editing files, user files, a network management program, and the like. HD) 1211 and flexible disk (FD) 1212 are controlled.

  Reference numeral 1208 denotes a network interface card (NIC) that exchanges data bidirectionally with a network printer, another network device, or another PC via the LAN 1220.

It is a block diagram which shows the main structures of the imaging device by this embodiment. It is a block diagram which shows the main structures of the image processing apparatus which is the feature part of the imaging device by this embodiment. It is a schematic diagram for demonstrating the problem which arises in an image with the foreign material on a light-receiving surface. FIG. 4 is a schematic diagram for qualitatively explaining the relationship between each component in FIG. 3. It is a schematic diagram for demonstrating quantitatively the relationship between each component of FIG. It is a schematic diagram for demonstrating quantitatively the relationship between each component of FIG. It is a schematic diagram for demonstrating quantitatively the relationship between each component of FIG. It is a flowchart which shows the image correction method using the imaging device by this embodiment. It is a schematic diagram which shows the internal structure of a general personal user terminal device.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Optical system 2 Image processing apparatus 3 Analog signal processing circuit 4 A / D converter 5 Digital signal processing circuit 6 Buffer memory 7 Control circuit 8 Interface (I / F)
9 Memory bus 11 Imaging lens 12 Optical LPF
13 Solid-state imaging device 14 Correction means 21 Position calculation means 22 Size calculation means 23 Light intensity calculation means 24 Correction amount calculation means 25 Foreign matter 26 Light flux 27 Projected image 28 Optical axis

Claims (12)

Imaging means for photoelectrically converting a subject image into an image signal;
An image processing apparatus comprising: a correction unit that corrects a signal drop caused in a portion corresponding to the foreign substance in the image signal due to a foreign substance existing in the optical system that forms the subject image.   The image processing apparatus according to claim 1, wherein the correction unit changes a correction amount according to a distance between an imaging lens of the optical system and an imaging surface of the imaging unit.   The image processing apparatus according to claim 1, wherein the correction unit changes a correction amount according to an aperture value of the imaging lens of the optical system. The correction means includes
Position calculating means for calculating the position of the projected image of the foreign matter on the imaging surface of the imaging means;
Size calculating means for calculating the size of the projected image;
Light intensity calculating means for calculating the light intensity of the projected image;
The correction amount calculation means which correct | amends the said signal fall based on the said position and said magnitude | size of the said projection image using the calculated said light intensity, The any one of Claims 1-3 characterized by the above-mentioned. An image processing apparatus according to 1.   The image processing apparatus according to claim 4, wherein the correction amount calculating unit adjusts a correction value by applying a light blocking rate to the light intensity according to a light blocking state of the foreign matter. An optical system on which the incident light of the subject image is incident;
Imaging means for photoelectrically converting a subject image into an image signal;
Correction means for correcting a signal decrease caused in a portion corresponding to the foreign matter in the image signal due to the foreign matter existing in the optical system for forming the subject image;
Signal processing means for processing the corrected image signal;
And an image display means for displaying an image from the image signal.   The imaging apparatus according to claim 6, wherein the correction unit changes a correction amount according to a distance between an imaging lens of the optical system and an imaging surface of the imaging unit.   The imaging apparatus according to claim 6, wherein the correction unit changes a correction amount according to an aperture value of an imaging lens of the optical system. The correction means includes
Position calculating means for calculating the position of the projected image of the foreign matter on the imaging surface of the imaging means;
Size calculating means for calculating the size of the projected image;
Light intensity calculating means for calculating the light intensity of the projected image;
The correction amount calculation means for correcting the signal decrease based on the position and the size of the projection image using the calculated light intensity. 9. The imaging device described in 1.   The imaging apparatus according to claim 9, wherein the correction amount calculation unit adjusts a correction value by applying a light blocking rate to the light intensity according to a light blocking state of the foreign matter. When correcting a signal drop caused in a part corresponding to the foreign substance in the image signal due to a foreign substance existing in the optical system that forms the subject image,
Calculating the position of the projected image of the foreign matter on the imaging surface of an imaging means that photoelectrically converts the subject image into an image signal;
Calculating a size of the projected image;
Calculating the light intensity of the projected image;
And correcting the signal decrease based on the position and the size of the projection image using the calculated light intensity.   The image processing method according to claim 11, wherein a correction value is adjusted by applying a light shielding rate to the light intensity in accordance with a light shielding state of the foreign matter.

Images