JP2008131607A - Imaging apparatus, its control method and program, and storage medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To suppress deterioration of a correction precision of a defective pixel even if a contamination adheres to an optical system component of an imaging device. <P>SOLUTION: The imaging apparatus includes: an imaging device in which a plurality of pixels for photoelectrically converting an image of an object are arranged two-dimensionally: a detector which detects a position of a contamination adhering to an optical component disposed on an imaging optical axis, ahead of the imaging device; a corrector which corrects an image signal output from a defective pixel portion of the imaging device using an image signal output from pixels adjacent to the defective pixel portion; and a controller which controls the corrector to correct the image signal output from the defective pixel portion using the image signal output from the pixels adjacent to the defective pixel portion, excluding the pixels affected by shadow of the contamination when it is judged from the result of detection in the detector that the pixels adjacent to the defective pixel portion are affected by the shadow of the contamination. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a technique for correcting pixel defects in an imaging apparatus such as a digital camera.

  2. Description of the Related Art An imaging device such as a digital camera that records and reproduces still images and moving images captured by a solid-state imaging device (hereinafter referred to as imaging device) such as a CCD or CMOS sensor using a memory card having a solid-state memory device as a recording medium is known. .

  In such an image pickup apparatus, external light incident through the optical lens is received by an image pickup element formed by two-dimensionally arranging a large number of light receiving elements (hereinafter referred to as pixels) in a plane. Each pixel receives light from the imaging target and outputs an analog electric signal corresponding to the amount of light received. The output of each pixel of the image sensor is scanned and sequentially converted into a digital signal by an A / D converter, and then supplied to a signal processing circuit. The signal processing circuit converts a digital signal into a display signal and supplies it to a display device or a recording medium. As a result, an image corresponding to the two-dimensionally arranged pixels is displayed on the display screen.

  In an image sensor, it is known that among the pixels arranged two-dimensionally, a local sensitivity failure of a semiconductor occurs during or after the manufacturing process. When this phenomenon occurs, a charge output corresponding to the amount of incident light cannot be obtained, so that there is a problem that the acquired image appears as a so-called defective pixel in which white spots and black spots can be seen regardless of the subject.

  Conventionally, in order to correct image quality degradation caused by such defective pixels by signal processing, the following method has been employed. That is, when the image pickup device is shipped from the factory, the pixel output during a predetermined accumulation time in a predetermined environment is evaluated, and for a pixel determined to be defective, the defect type (black point, white point) and the pixel address ( Information on the coordinates (x, y)) of the defective pixel and the degree (level) of the defect is obtained. Then, the obtained information is stored in a nonvolatile memory provided in the imaging apparatus, and a defective pixel portion is corrected based on this information with respect to the photographed image at the time of actual photographing (for example, Patent Document 1). .

  In addition, since the number of defective pixels increases with the passage of time, there is also an imaging device that is provided with means for detecting defective pixels, additionally stores information on newly generated defective pixels, and corrects defective pixels. (For example, patent document 2).

As a method for correcting a defective pixel, a method is known in which an average value or an interpolation value of pixels of the same color around a correction target pixel is replaced with an image signal of a pixel defective portion. An example of this method will be described with reference to FIG. 4A. FIG. 4A is a diagram schematically showing a part of the pixels of the image sensor, in which pixels of three colors R, G, and B are arranged in a regular combination. Assuming that G21 is a defective pixel, the average value of image signals of adjacent pixels G16, G17, G25, and G26 is obtained, and the defective pixel is corrected by replacing it with the G21 image signal.
JP 2000-23051 A JP 2005-123873 A JP 2003-018440 A

  By the way, in a digital camera, dust, dust, or the like is placed on the surface of an image sensor protective glass or an optical filter (hereinafter collectively referred to as an image sensor optical system component) disposed in front of the image sensor and on the photographing optical axis. Foreign matter may adhere. Then, the adhered foreign matter appears in the photographed image as a shadow. In such a case, when trying to interpolate a defective pixel with surrounding pixels of the same color as described above, if the pixel used for the interpolation is a shadow of a foreign object, the defective pixel cannot be corrected accurately. . Specifically, when pixels used for interpolation are in the shadow of a foreign object, the image signal of these pixels is lowered, and the correction value is pulled in the decreasing direction.

  Therefore, there is known a camera equipped with means for automatically removing foreign substances adhering to the image sensor optical system parts (for example, Patent Document 3), and most foreign substances can be removed by this foreign substance removing operation. It was.

  However, some of the adhered foreign matter is very fine and cannot be removed enough to cover several pixels of the image sensor. Therefore, when the foreign matter is attached to a position that affects the image signal used for correction, the correction value is pulled in a decreasing direction and the correction accuracy is reduced. That is, even if the conventional foreign matter removing means is used, the problem of a reduction in the correction accuracy of defective pixels is not completely solved.

  Therefore, the present invention has been made in view of the above-described problems, and an object of the present invention is to suppress a reduction in the correction accuracy of defective pixels even when foreign matter is attached to the imaging element optical system component. It is to be.

  In order to solve the above-described problems and achieve the object, an imaging apparatus according to the present invention includes an imaging element in which a plurality of pixels that photoelectrically convert a subject image are two-dimensionally arranged, and a front side of the imaging element. Detecting means for detecting the position of the foreign matter attached to the optical member disposed on the photographing optical axis, and outputting an image signal output from the defective pixel portion of the imaging element from a pixel in the vicinity of the defective pixel portion. A correction unit that corrects using the image signal and a detection result of the detection unit, when it is determined that a pixel in the vicinity of the defective pixel portion is affected by the shadow of the foreign object, The correction means is controlled so as to correct the image signal output from the defective pixel portion using an image signal output from a pixel in the vicinity of the defective pixel portion excluding pixels affected by the shadow. Control means, And butterflies.

  The image pickup apparatus control method according to the present invention includes an image pickup device in which a plurality of pixels for photoelectrically converting a subject image are two-dimensionally arranged, and arranged in front of the image pickup device and on a shooting optical axis. Detecting means for detecting the position of the foreign matter adhering to the optical member and an image signal output from the defective pixel portion of the image sensor using an image signal output from a pixel in the vicinity of the defective pixel portion A correction unit, and a method of controlling an imaging apparatus, wherein when it is determined from the detection result of the detection unit that a pixel in the vicinity of the defective pixel unit is affected by the shadow of the foreign matter, The correcting unit corrects an image signal output from the defective pixel portion using an image signal output from a pixel in the vicinity of the defective pixel portion excluding pixels affected by the shadow of the foreign matter. A control process for controlling And wherein the door.

  A program according to the present invention causes a computer to execute the above control method.

  A storage medium according to the present invention stores the above program.

  According to the present invention, it is possible to prevent the correction accuracy of defective pixels from being lowered even when foreign matter is attached to the image sensor optical system component.

  Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a block diagram illustrating a configuration of a digital camera using an image sensor having a plurality of photoelectric conversion units such as a CCD and a CMOS sensor.

  The analog output signal of the image sensor 1 having a plurality of photoelectric conversion units is subjected to noise reduction processing and gain adjustment processing by the CDS / AGC circuit 2, converted into a digital signal by the A / D converter 3, and digital signal processing IC Is input to the engine 4. Note that a plurality of photoelectric conversion units (pixels) that photoelectrically convert a subject image are two-dimensionally arranged in the image sensor 1.

  The engine 4 includes a defective pixel correction unit 5, a defective pixel information storage unit 6, a foreign matter adhesion detection unit 7, a foreign matter information storage unit 8, a switch 9, and a development processing unit 10.

  The image signal input to the engine 4 is input to the defective pixel correction unit 5, and predetermined correction described later is performed on the image signal from the defective pixel of the image sensor 1. The image signal processed by the defective pixel correction unit 5 is input to the development processing unit 10, subjected to development processing such as color interpolation processing, and stored in the memory 19.

  The defective pixel correction unit 5 is effective based on the position data of the defective pixels obtained from the information of the defective pixel information storage unit 6 and the foreign matter information storage unit 8 and the presence or absence of foreign matter in the image signal used for correcting the defective portion image. A correct correction value is determined, and the image signal from the defective pixel portion is corrected.

  For example, FIG. 4A is a diagram schematically showing a part of the pixels of the image sensor, in which pixels of three colors R, G, and B are arranged in a regular combination. If G21 is a defective pixel, the average value of the image signals of G16, G17, G25, and G26, which are adjacent pixels of the same color, is obtained and replaced with the image signal of G21 to correct the image signal from the defective pixel portion. In addition to the above correction method, it may be possible to change the number of pixels of the image signal used for correction, and any method other than the above may be used as long as it can avoid image deterioration due to defective pixels.

  The defective pixel information storage unit 6 stores, for example, information such as the position data of the defective pixels, the defect level, and the defect type acquired when the image pickup device 1 is shipped from the factory. Further, the foreign matter on the surface of the cover glass of the image sensor 1 or the surface of the optical filter 20 affixed to the cover glass surface of the image sensor 1 is detected by the foreign matter adhesion detector 7, and the foreign matter position information, the size of the foreign matter, etc. It is stored in the information storage unit 8. Hereinafter, the surface of the cover glass of the image sensor 1 or the surface of the optical filter 20 attached to the cover glass surface of the image sensor 1 will be referred to as “image sensor surface” for simplicity.

  The image developed by the development processing unit 10 is displayed on a display (LCD) 11 such as a TFT liquid crystal display, or outputted to an external monitor from a video terminal (VIDEO) 12 or the like.

  The developed image can also be stored in an external memory 13, for example, a CF card (Compact Flash (registered trademark) memory card). The engine 4 is connected to an interface 14 such as a USB interface, which is a serial communication device, through which image data can be output to an external computer or the like. Further, the engine 4 drives a timing generator (TG / VDR) 15 to generate a clock signal or the like to be supplied to the image sensor 1.

  Further, a microprocessor (CPU) 18 is connected to the engine 4, various settings are performed in accordance with instructions from the CPU 18, and operations are managed by the CPU 18. The memory 19 is used for storing a program to be supplied to the CPU 18 and temporarily storing a photographed image, and is also used as a work area. The lens (LENS) 16 is an optical system for photographing, and has an actuator for adjusting its aperture and focus. The actuator is controlled according to an instruction from the CPU 18. The foreign matter removing unit 17 has an actuator for removing foreign matter on the surface of the optical filter 20 attached to the image sensor 1, and is driven and controlled by the CPU 18 when the power is turned on, for example. Specifically, for example, a piezoelectric element can be used as the actuator, and the optical filter 20 is vibrated by the piezoelectric element to remove foreign matter.

  Further, the optical filter 20 is assembled by adhering in a highly sealed state after sufficiently removing foreign matter on the surface of the cover glass of the image sensor 1 and on the surface of the optical filter 20.

  Next, the operation of the foreign matter adhesion detection unit 7 provided in the engine 4 will be described with reference to the flowchart of FIG.

  In FIG. 2, when the main power source (not shown) of the camera is activated, the CPU 18 controls the foreign matter removing unit 17 to remove foreign matter on the surface of the optical filter 20 attached to the cover glass of the image sensor 1 for a predetermined time. This is automatically executed (step S201). In the present embodiment, this foreign matter removal operation is executed when the main power supply of the camera is started. However, the operation time is not limited to this, and may be automatically executed periodically, or arbitrarily by the user. It does not matter if it is executed. The details of the foreign matter removing operation may be a method as described in, for example, Japanese Patent Application Laid-Open No. 2003-018440, or any other method that can effectively remove foreign matter on the surface of the image sensor. However, it is not limited to the method.

  The purpose of the foreign matter removal operation is to remove foreign matter attached to the surface of the optical filter 20 attached to the cover glass of the image sensor 1. However, there is a limit to this removal capability, and most of the attached foreign matter can be removed with certainty. However, there are cases in which some of the very fine foreign matter of several pixels of the image sensor cannot be removed. Therefore, after step S202, it is detected how much foreign matter remains on the surface of the image sensor.

  When the foreign matter removing operation is completed, the CPU 18 initializes the stored contents of the foreign matter information storage unit 8 in the engine 4 (step S202). Then, the switch 9 is turned on so that the image signal from the A / D converter 3 is input to the foreign matter adhesion detection unit 7 (step S203).

  Then, a foreign object detection image is taken (step S204). Since a shadow is formed on the photographed image at a place where foreign matter is attached to the surface of the image pickup device 1, relatively bright uniform light that is easy to detect the foreign matter is photographed in step S204. As a light incident method, a dedicated light source may be provided inside the camera as described in JP-A-2002-300442, or light from the outside of the camera may be used. An output signal of the image sensor 1 obtained by photographing the uniform light is input to the engine 4 via the CDS / AGC circuit 2 and the A / D converter 3. The image signals input to the engine 4 are sequentially input to the foreign matter adhesion detection unit 7 for each image signal corresponding to one pixel (step S205).

  The foreign matter adhesion detection unit 7 performs defective pixel determination on the input image signal based on information in the defective pixel information storage unit (step S206), and foreign matter adhesion determination is performed on the image signal that is a defective pixel. Not determined (Yes in step S206). Then, the process proceeds to S209.

  An image signal that is not a defective pixel (determination No in step S206) is compared with a predetermined value (step S207), and when the compared image signal is smaller than the predetermined value, it is determined that foreign matter is attached to the surface of the image sensor. Then, information such as the position data of the pixel affected by the foreign matter and the level of the image signal is written in the foreign matter information storage unit 8 (step S208). On the other hand, when the compared image signal is equal to or greater than the predetermined value, it can be determined that no foreign matter is attached, and therefore writing to the foreign matter information storage unit 8 is not performed.

  Then, every time the detection operation for one pixel is completed, it is confirmed whether the detection operation for all the pixels is completed (step S209), and steps S205 to S209 are repeated until the detection operation is completed.

  Next, the operation of the defective pixel correction unit 5 provided in the engine 4 will be described with reference to the flowchart of FIG.

  In FIG. 3, when normal shooting is performed by the user's hand, the output of the image sensor 1 that receives light from the shooting target is input to the engine 4 via the CDS / AGC circuit 2 and the A / D converter 3. (Step S301). The image signal input to the engine 4 is sequentially input to the defective pixel correction unit 5 for each image signal corresponding to one pixel (step S302).

  The defective pixel correction unit 5 performs defective pixel determination on the input image signal based on information in the defective pixel information storage unit 6 (step S303). If the image signal is not for a defective pixel (determination No in step S303), the process proceeds to step S307 without correcting the image signal. The description of step S307 will be described later.

  If the image signal is from the defective pixel portion (determination Yes in step S303), the foreign object information storage unit 8 determines whether or not the peripheral image signal is a foreign object adhesion target in order to create a correction value for the defective pixel portion. (Step S304). When the surrounding image signal is not affected by the adhesion of foreign matter (determination No in step S304), a correction value is obtained using image signals from pixels around the defective pixel by a normal method (step S305). ). Then, the obtained correction value is replaced with the image signal of the defective pixel portion (step S306).

  In the above correction method, for example, as shown in FIG. 4A, the average value of the image signals of the same color pixels G16, G17, G25, and G26 (hatched portions) adjacent to the defective pixel G21 is obtained, and this average value is replaced with the G21 image signal. Thus, the defective pixel is corrected.

  After the replacement of the image signal (step S306), every time the correction operation for one pixel is completed, it is confirmed whether the defective pixel determination for all pixels and the correction operation of the image signal from the defective pixel portion are completed (step S307). ), Steps S302 to S307 are repeated until this is completed.

  On the other hand, in step S304, when the peripheral image signal is affected by the adhesion of foreign matter (determination Yes in step S304), the following is performed. That is, the image signal from the defective pixel portion is corrected so as not to use the image signal affected by the foreign matter adhesion (steps S308 and S306).

  For example, as shown in FIG. 4B, when the image signal of G17, which is one of the same color pixels adjacent to the defective pixel G21, is affected by the adhesion of foreign matter, the following is performed. That is, an average value of image signals of G16, G25, and G26 (hatched portions) excluding G17 is obtained, and defective pixels are corrected by replacing the average value with the image signal of G21. Alternatively, as shown in FIG. 4C, in addition to G16, G25, and G26 except G17, the average value of the G20 and G30 image signals may be obtained, and the defective pixel may be corrected by replacing the average value with the G21 image signal. . Further, as shown in FIG. 4D, when the image signals of the same color pixels G17 and G26 adjacent to the defective pixel G21 are affected by the adhesion of foreign matter, the following is performed. That is, in addition to G16 and G25 except G17 and G26, the average value of the image signals of G12, G20, and G30 is obtained, and the defective value is corrected by replacing the average value with the image signal of G21.

  Returning to FIG. 3, after the replacement of the image signal (step S <b> 306), every time the correction operation for one pixel is completed, it is confirmed whether the defective pixel determination for all pixels and the correction operation for the image signal from the defective pixel portion are completed. (Step S307). Then, steps S302 to S307 are repeated until this operation ends.

  The above is the correction operation for defective pixels in the present embodiment.

  In the above description, when the pixels around the defective pixel portion are affected by the foreign matter, the signal of the pixel affected by the foreign matter is not used for correction. However, when both the defective pixel portion and the surrounding pixels are affected by the foreign matter (shadowed by the foreign matter), the correction is performed in the same manner as when not affected by the foreign matter. That is, even if a pixel around the defective pixel portion is affected by a foreign substance, correction is performed using the signal of the pixel.

  The reason is as follows.

  The defective pixel portion often has a white spot rather than a normal black spot. Therefore, if the signal of the defective pixel portion is used as it is without correcting the defective pixel portion, the pixel shines white and the defect is easily noticeable. For this reason, if the defective pixel portion is corrected using a signal of a peripheral pixel that is a shadow of a foreign object, the defective pixel portion outputs a dark signal, so that the defect is less noticeable. For this reason, when both the defective pixel portion and the peripheral pixels are affected by foreign matter, correction is performed assuming that there is no foreign matter influence.

  As described above, in the present embodiment, basically, the image signal of the defective pixel of the image sensor is corrected by the image signal from the pixel portion in the vicinity of the defective pixel portion. However, in that case, when the image signal from the above-mentioned neighboring pixel portion is affected by the adhesion of foreign matter, the image signal affected by the adhesion of foreign matter is not used for correction. As a result, it is possible to correct the image signal of the defective pixel portion without causing an extreme deterioration in accuracy. Further, not only the image signal affected by the adhesion of foreign matter is not used for correction, but also the peripheral image signal which is not affected by the adhesion of foreign matter and is not used at the time of normal correction is added and corrected. As a result, the amount of information increases, so that the correction accuracy is improved to the same level as during normal correction.

(Other embodiments)
The object of each embodiment is also achieved by the following method. That is, a storage medium (or recording medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded is supplied to the system or apparatus. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but the present invention includes the following cases. That is, based on the instruction of the program code, an operating system (OS) or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

  Furthermore, the following cases are also included in the present invention. That is, the program code read from the storage medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, a CPU or the like provided in the function expansion card or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the above storage medium, the storage medium stores program codes corresponding to the procedure described above.

It is a block diagram which shows the structure of the digital camera concerning one Embodiment of this invention. It is a flowchart explaining the foreign material adhesion detection operation in the digital camera of one embodiment of the present invention. It is a flowchart explaining the defective pixel correction | amendment operation | movement in the digital camera of one Embodiment of this invention. It is the figure which represented typically a part of pixel of an image sensor for demonstrating the method of correct | amending the image signal from a defective pixel part. It is the figure which represented typically a part of pixel of an image sensor for demonstrating the method of correct | amending the image signal from a defective pixel part. It is the figure which represented typically a part of pixel of an image sensor for demonstrating the method of correct | amending the image signal from a defective pixel part. It is the figure which represented typically a part of pixel of an image sensor for demonstrating the method of correct | amending the image signal from a defective pixel part.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image pick-up element 2 CDS / AGC circuit 3 A / D converter 4 Engine 5 Defective pixel correction part 6 Defective pixel information storage part 7 Foreign substance adhesion detection part 8 Foreign substance information storage part 9 Switch 10 Development processing part 11 Display part 12 Video terminal 13 Flash memory card 14 USB terminal 15 Timing generator 16 Lens 17 Foreign matter removing unit 18 Microprocessor 19 Memory 20 Optical filter

Claims (5)

An image sensor in which a plurality of pixels for photoelectrically converting a subject image are two-dimensionally arranged;
Detecting means for detecting the position of a foreign substance attached to an optical member disposed in front of the image sensor and on a photographing optical axis;
Correction means for correcting an image signal output from a defective pixel portion of the image sensor using an image signal output from a pixel in the vicinity of the defective pixel portion;
When it is determined from the detection result of the detection means that pixels in the vicinity of the defective pixel portion are affected by the shadow of the foreign matter, the defect excluding the pixels affected by the shadow of the foreign matter Control means for controlling the correction means so as to correct the image signal output from the defective pixel portion using an image signal output from a pixel in the vicinity of the pixel portion;
An imaging apparatus comprising:   When the pixel in the vicinity of the defective pixel portion is affected by the shadow of the foreign object, the control means determines a pixel not affected by the shadow of the foreign object in place of the pixel affected by the shadow of the foreign object. 2. The imaging apparatus according to claim 1, wherein the correction unit is controlled to select and use an image signal from the pixel for correcting an image signal output from the defective pixel unit. An image sensor in which a plurality of pixels that photoelectrically convert a subject image are two-dimensionally arranged, and a detection that detects the position of a foreign substance attached to an optical member disposed in front of the image sensor and on the photographing optical axis And a correction unit that corrects an image signal output from a defective pixel portion of the imaging element using an image signal output from a pixel in the vicinity of the defective pixel portion. There,
When it is determined from the detection result of the detection means that pixels in the vicinity of the defective pixel portion are affected by the shadow of the foreign matter, the defect excluding the pixels affected by the shadow of the foreign matter An imaging apparatus comprising: a control step of controlling the correction unit so as to correct an image signal output from the defective pixel unit using an image signal output from a pixel in the vicinity of the pixel unit. Control method.   A program for causing a computer to execute the control method according to claim 3.   A computer-readable storage medium storing the program according to claim 4.

Images