JP2008244868A - Imaging apparatus, and control method, program and storage medium thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To suppress the precision of dust removal correction for image data from being degraded even in the case where the situation of dust deposition changes after dust correction data are acquired. <P>SOLUTION: The imaging apparatus comprises: an image pickup element in which a plurality of pixels each for performing optic/electric conversion on a subject image are arrayed in a two-dimensional manner; a foreign material detection section for detecting information about the position of a foreign material deposited on an optical member disposed on a photographing optical axis in front of the image pickup element; a change detection section for detecting a change in the situation of the foreign material deposited on the optical member based on a picked-up image obtained by performing optic/electric conversion on the subject image by means of the image pickup element and the information about the position of the foreign material being stored in the storage section; and an updating section for updating the information about the position of the foreign material being stored in the storage section based on the information about the change in the deposition situation of the foreign material detected by the change detection section. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a technique for reducing the influence of foreign matter such as dust in an imaging optical path on an image in an imaging apparatus.

  In an imaging device such as a digital camera that captures an image by converting an image signal into an electrical signal, the imaging light beam is received by the imaging device, a photoelectric conversion signal output from the imaging device is converted into image data, and a memory card, etc. Record on a recording medium. As an image sensor, a CCD (Charge Coupled Device), a C-MOS (Complementary Metal Oxide Semiconductor), and the like are known.

  In such an imaging apparatus, an optical low-pass filter and an infrared cut filter are disposed on the subject side of the imaging element. It is known that if foreign matter such as dust adheres to the cover glass of the image sensor or the surface of these filters, the adhering portion becomes a black dot and appears in the photographed image, thereby reducing the quality of the photographed image. .

  In particular, in a single-lens reflex digital camera with interchangeable lenses, mechanical operation parts such as shutters and quick return mirrors are arranged in the vicinity of the image sensor, and foreign matters such as dust generated from these operation parts are captured by the image sensor and May adhere to the low-pass filter. Further, when the lens is exchanged, dust or the like may enter the camera body from the opening of the lens mount and adhere to it.

  Conventionally, in the field of copiers, as a countermeasure against dust that may change constantly, a technique for detecting dust by capturing a uniform luminance pattern having a uniform reflection surface before reading a document is disclosed in Patent Document 1 and Patent Document 2. Is disclosed.

  In Patent Document 3, a transmittance map of the entire image is created from a reference image obtained by photographing a subject with uniform brightness, and the captured image is corrected based on the transmittance map, and shadows of dust, etc. An electronic camera that removes light is disclosed.

  In the techniques disclosed in Patent Literature 1 and Patent Literature 2, dust correction data is acquired before the main scan or simultaneously with the main scan in a copier or scanner, and dust correction processing is performed. Copiers and scanners have a uniform illumination means for a document or film surface at a fixed distance, and have a completely uniform reflective surface or a new infrared illumination means. Transmittance data can be obtained relatively easily. However, in the case of an electronic camera, it is difficult to obtain transmittance data under such completely uniform conditions except for inspection at the time of manufacture.

Further, in the electronic camera, when the dust correction data is acquired before shooting or after shooting, there is a possibility that the release time lag increases or the continuous shooting frame speed decreases.
Japanese Patent Laid-Open No. 10-294870 Japanese Patent Laid-Open No. 11-27475 JP 2004-222231 A JP 2006-203776 A

  Here, as a method for preventing the inconvenience such as an increase in the release time lag and a decrease in the continuous shooting frame speed as described above, the dust correction data is acquired every certain time interval or not every shooting. A method of performing every number of times of photographing can be considered.

  However, this method has a problem that the reliability of the dust information is lowered when the dust adhesion state changes between the acquisition of dust correction data and the next acquisition of dust correction data. . In other words, it is conceivable that the dust disappears after the dust correction data is acquired, and unnecessary dust removal processing is performed even on the portion where the dust is removed, which may deteriorate the image quality.

  Furthermore, in an imaging apparatus having a function of removing dust attached to an optical surface as disclosed in Japanese Patent Application Laid-Open No. 2006-203776 (Patent Document 4), a dust removal operation is executed after obtaining dust correction data. And the possibility of running out of trash further increases.

  Accordingly, the present invention has been made in view of the above-described problems, and an object of the present invention is to suppress a decrease in accuracy of dust removal correction for image data even when the dust adhesion state changes after dust correction data acquisition. Is to do.

  In order to solve the above-described problems and achieve the object, an imaging apparatus according to the present invention includes an imaging element in which a plurality of pixels that photoelectrically convert a subject image are two-dimensionally arranged, and imaging in front of the imaging element. Foreign matter detection means for detecting information on the position of the foreign matter attached to the optical member disposed on the optical axis, storage means for storing at least information on the position of the foreign matter detected by the foreign matter detection means, and the imaging A change for detecting a change in the adhesion state of the foreign matter to the optical member based on a captured image obtained by photoelectrically converting the subject image by an element and information on the position of the foreign matter stored in the storage means An updater that updates information on the position of the foreign matter stored in the storage means based on information on a change in the adhesion state of the foreign matter detected by the detection means and the change detection means Characterized by comprising the, the.

  The image pickup apparatus control method according to the present invention includes an image pickup device in which a plurality of pixels for photoelectrically converting a subject image are two-dimensionally arranged, and an optical element disposed on a photographing optical axis in front of the image pickup device. A method for controlling an imaging apparatus, comprising: a foreign matter detection unit that detects information about a position of a foreign matter attached to a member; and a storage unit that stores at least information about the position of the foreign matter detected by the foreign matter detection unit. Based on the captured image obtained by photoelectrically converting the subject image by the imaging element and information on the position of the foreign matter stored in the storage means, the change in the state of attachment of the foreign matter to the optical member is determined. Based on the change detection step to detect and information on the change in the adhesion state of the foreign matter detected in the change detection step, information on the position of the foreign matter stored in the storage means An updating step of new to, characterized by including the.

  A program according to the present invention causes a computer to execute the above control method.

  A storage medium according to the present invention stores the above program.

  According to the present invention, it is possible to suppress a reduction in accuracy of dust removal correction for image data even when the dust adhesion state changes after dust correction data is acquired.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described in detail with reference to the accompanying drawings.

(First embodiment)
FIG. 1 is a block diagram showing a circuit configuration of an interchangeable lens single-lens reflex digital camera as an imaging apparatus according to the first embodiment of the present invention.

  In FIG. 1, the microcomputer 402 controls the operation of the entire camera, including processing of image data output from the image sensor 418 and display control of the LCD monitor device 417. Note that the image sensor 418 is configured by two-dimensionally arranging a plurality of pixels that photoelectrically convert a subject image.

  The switch (SW1) 405 is turned on when the release button 114 (see FIG. 2) is half-pressed. When the switch (SW1) 405 is turned on, the digital camera according to the present embodiment is in a shooting preparation state. The switch (SW2) 406 is turned on when the release button 114 is pressed to the end (fully pressed state), and when the switch (SW2) 406 is turned on, the digital camera of this embodiment starts a photographing operation.

  The lens control circuit 407 performs communication with the photographing optical system 200 (see FIG. 3) and drive control of the photographing optical system 200 and driving control of the diaphragm blades during AF (autofocus).

  In FIG. 1, an external display control circuit 408 controls an external display device (OLC) 409 and a display device (not shown) in the viewfinder. The switch sense circuit 410 transmits signals of a large number of switches including the electronic dial 411 provided in the camera to the microcomputer 402.

  The strobe light emission dimming control circuit 412 is grounded via the X contact 412a and controls the external strobe. The distance measuring circuit 413 detects the defocus amount with respect to the subject for AF. The photometry circuit 414 measures the luminance of the subject.

  A shutter control circuit 415 controls the shutter and performs appropriate exposure on the image sensor. The LCD monitor device 417 and the backlight illumination device 416 constitute an image display device. The recording device 419 is, for example, a hard disk drive or a semiconductor memory card that can be attached to and detached from the camera body.

  Further, the microcomputer 402 includes an A / D converter 423, an image buffer memory 424, an image processing circuit 425 including a DSP, and a pixel defect position memory 426 that stores that a predetermined pixel in the image sensor itself is defective. Is connected. Also connected is a dust position memory 427 that stores pixel positions in the image sensor where image defects are caused by dust (hereinafter also referred to as foreign matter). Note that the pixel defect position memory 426 and the dust position memory 427 are preferably non-volatile memories. Further, the pixel defect position memory 426 and the dust position memory 427 may be stored using different addresses in the same memory space.

  Reference numeral 428 denotes a nonvolatile memory that stores programs executed by the microcomputer 402.

  Further, reference numeral 418b denotes a vibration device for vibrating an optical member arranged on the photographing optical axis in front of the image sensor 418.

  FIG. 2 is a perspective view showing the external appearance of the digital camera according to the present embodiment, and FIG. 3 is a vertical sectional view of FIG. In FIG. 2, an eyepiece window 111 for finder observation, an AE (automatic exposure) lock button 112, an AF distance measuring point selection button 113, and a release button 114 for instructing a photographing operation are provided on the upper part of the camera body 100. It has been. An electronic dial 411, a shooting mode selection dial 117, and an external display device 409 are also provided. The electronic dial 411 is a multi-function signal input device that is used in combination with other operation buttons to input numerical values to the camera and to switch the shooting mode. The external display device 409 includes a liquid crystal display device, and displays shooting conditions such as shutter speed, aperture, and shooting mode, and other information.

  Also, on the back of the camera main body 100, an LCD monitor device 417 for displaying captured images and various setting screens, a monitor switch 121 for turning on / off the LCD monitor device 417, a cross switch 116, and a menu button 124 is provided.

  The cross placement switch 116 has four buttons arranged vertically and horizontally and a SET button arranged in the center. The user instructs the camera to select and execute menu items displayed on the LCD monitor device 417. Used for.

  The menu button 124 is a button for causing the LCD monitor device 417 to display a menu screen for performing various camera settings. For example, when selecting and setting the shooting mode, after pressing the menu button 124, the user operates the up / down / left / right buttons of the cross switch 116 to select the desired mode, and the desired mode is selected. Setting is completed by pressing the SET button. The menu button 124 and the cross switch 116 are also used for setting a dust detection mode, which will be described later, and a display mode setting in the dust detection mode.

  Since the LCD monitor device 417 of the present embodiment is a transmissive type, an image cannot be visually recognized only by driving the LCD monitor device, and a backlight illumination device 416 is necessarily provided on the back surface thereof as shown in FIG. is there. As described above, the LCD monitor device 417 and the backlight illumination device 416 constitute an image display device.

  As shown in FIG. 3, the photographing optical system 200 can be attached to and detached from the camera body 100 via a lens mount 202. In FIG. 3, 201 denotes a photographing optical axis, and 203 denotes a quick return mirror.

  The quick return mirror 203 is disposed in the photographing optical path, and a position for guiding subject light from the photographing optical system 200 to the finder optical system (referred to as a position shown in FIG. 3, an oblique position) and a position for retracting outside the photographing optical path (retraction) It is possible to move between them.

  In FIG. 3, the subject light guided from the quick return mirror 203 to the finder optical system is imaged on the focus plate 204. Reference numeral 205 denotes a condenser lens for improving the visibility of the finder, and 206 denotes a pentagonal roof prism.

  Reference numerals 209 and 210 respectively denote a rear curtain and a front curtain that constitute a shutter, and an image sensor 418 that is a solid-state image sensor disposed behind the rear curtain 209 and the front curtain 210 is exposed for a necessary time. The subject image formed by the photographing optical system 200 is converted into an electric signal for each pixel by the image sensor 418. The photoelectrically converted photographed image is processed by the A / D converter 423, the image processing circuit 425, etc. It is recorded in the recording device 419 as data.

  The image sensor 418 is held on the printed circuit board 211. Behind this printed board 211, a display board 215, which is another printed board, is arranged. An LCD monitor device 417 and a backlight illumination device 416 are disposed on the opposite surface of the display substrate 215.

  An optical member 418a such as an optical low-pass filter or an infrared cut filter is disposed on the photographing optical axis 201 in front of the image sensor 418. Then, foreign matter such as dust adheres to the surface of the optical member 418a disposed in front of the image sensor 418, and the shadow of the foreign matter is reflected in an image photoelectrically converted by the image sensor 418. The present embodiment relates to a technique for correcting an image in which the shadow of the foreign object is reflected.

  In the digital camera of the present embodiment, a vibration device 418b made of a piezoelectric element or the like is connected to the optical member 418a. The vibration device 418b is driven by an instruction from the microcomputer 402, and the optical member 418a vibrates. As a result, foreign matters such as dust adhering to the optical member 418a are removed. The driving timing of the vibration exciter 418b may be a preset timing or a timing instructed by the user.

  Reference numeral 419 denotes a recording device for recording image data, and 217 denotes a battery (portable power supply). The recording device 419 and the battery 217 are detachable from the camera body.

(Dust detection process)
FIG. 4 is a flowchart for explaining dust detection processing (detection processing of a pixel position where an image defect is caused by dust) in the digital camera according to the present embodiment. This processing is performed by the microcomputer 402 executing a dust detection processing program stored in the memory 428.

  This dust detection processing is processing for detecting the position and size of the shadow of dust attached to the surface of the optical member 418a disposed in front of the image sensor 418 in the image.

  The dust detection process is performed by capturing a dust detection image. When performing dust detection processing, a camera is installed with the photographing optical axis 201 of the photographing optical system (lens) 200 facing a surface having a uniform luminance pattern, such as an exit surface of a surface light source device or a white wall, and preparation for dust detection is performed. Do. Alternatively, a dust detection light unit (a small point light source device mounted instead of a lens) is mounted on the lens mount 202 to prepare for dust detection. The light source of the light unit may be a white LED, for example, and it is desirable to adjust the size of the light emitting surface so that it corresponds to a predetermined aperture value (for example, F64 in this embodiment).

  In this embodiment, a case where a normal photographing optical system is used will be described. However, dust detection may be performed by attaching the light unit to the lens mount 202. Thus, in this embodiment, the dust detection image is an image having a uniform color.

  After the preparation is completed, for example, when the start of dust detection processing is instructed from the cross arrangement switch 116, the microcomputer 402 first sets the aperture. The image forming state of dust near the imaging device (in front of the imaging device) changes depending on the aperture value of the lens, and the position changes depending on the pupil position of the lens. Therefore, in the dust correction data, in addition to the position and size of dust, it is necessary to hold the aperture value at the time of detection and the pupil position of the lens.

  However, it is not always necessary to hold the aperture value in the dust correction data if it is determined in advance that the same aperture value is always used even when different lenses are used at the stage of creating dust correction data. Similarly, by using a light unit or permitting the use of only a specific lens for the pupil position, it is not always necessary to hold the pupil position in the dust correction data. In other words, at the stage of creating dust correction data, if you want to allow multiple lenses to use or change the aperture value to narrow down appropriately, it is necessary to keep the aperture value and lens pupil position at the time of detection in the dust correction data It can be said that there is. Here, the pupil position refers to the distance from the imaging plane (focal plane) of the exit pupil.

  The flowchart will be specifically described below.

  First, when the dust detection process is started, the aperture value of the photographing optical system 200 is designated. Here, for example, F16 is designated (step S21).

  Next, the microcomputer 402 causes the lens control circuit 407 to perform aperture blade control of the photographing optical system 200, and sets the aperture to the aperture value designated in step S21 (step S22). Further, the focus position is set to infinity (step S23).

  When the aperture value and focus position of the photographing optical system 200 are set, photographing in the dust detection mode is executed (step S24). Details of the uniform luminance photographing routine performed in step S24 will be described later with reference to FIG. The captured image data is stored in the buffer memory 424 as dust detection image data.

  When shooting is completed, the aperture value and lens pupil position at the time of shooting are acquired (step S25).

  Next, data corresponding to each pixel of the dust detection image data stored in the image buffer memory 424 is called to the image processing circuit 425 (step S26).

  The image processing circuit 425 performs the process shown in FIG. 7, and acquires the position and size of the pixel where dust is present (step S27).

  Then, the position and size of the pixel where dust is present acquired in step S27 and the aperture value and lens pupil position information acquired in step S25 are registered in the dust position memory 427 (step S28). Here, when the light unit described above is used, lens information cannot be acquired. Therefore, when the lens information cannot be acquired, it is determined that the light unit is used, and predetermined lens pupil position information and a converted aperture value calculated from the light source diameter of the light unit are registered.

  Here, in step S28, the position of the defective pixel (pixel defect) from the time of manufacture recorded in the pixel defect position memory in advance is compared with the position of the read pixel data to check whether the pixel defect is present. Only the area determined not to be due to the pixel defect may be registered in the dust position memory 427.

  An example of the data format of dust correction data stored in the dust position memory 427 is shown in FIG. As shown in FIG. 5, the dust correction data stores lens information and dust position and size information when photographing dust detection image data. This dust correction data is added to the image together with the shooting time information of the image data during normal shooting, and used in dust removal processing described later.

  Specifically, the actual aperture value (F value) at the time of photographing dust detection image data and the lens pupil position at that time are stored as lens information at the time of photographing dust detection image data. Subsequently, the number of detected dust areas (integer value) is stored in the storage area, and subsequently, individual specific dust area parameters are repeatedly stored by the number of dust areas. The parameter of the dust area is a set of three numerical values: a dust radius (for example, 2 bytes), a center x coordinate (for example, 2 bytes) in the effective image area, and a similar y coordinate (for example, 2 bytes).

  When the dust correction data size is limited due to the size of the dust position memory 427 or the like, the data is stored with priority from the head of the dust area obtained in step S27. This is because in the dust area acquisition routine in step S27, the dust areas are sorted in the order of noticeable dust, as will be described later.

(Trash area acquisition routine)
Next, the details of the dust region acquisition routine performed in step S27 will be described with reference to FIGS.

  As shown in FIG. 6, the called dust detection image data is developed on a memory and processed in units of predetermined blocks. This is to cope with peripheral dimming caused by lens and sensor characteristics. Peripheral dimming is a phenomenon in which the luminance at the peripheral portion is lower than that at the central portion of the lens, and it is known that it is reduced to some extent by reducing the aperture of the lens. However, there is a problem that even if the aperture is reduced, if the dust position is determined based on a predetermined threshold value for the photographed image, dust in the peripheral portion cannot be detected accurately depending on the lens. Therefore, the image is divided into blocks to reduce the influence of peripheral dimming.

  When the blocks are simply divided, there is a problem that the detection result of the dust straddling the blocks is shifted when the threshold values are different between the blocks. Therefore, the blocks are overlapped, and pixels determined to be dust in any of the blocks constituting the overlap region are treated as dust regions.

  The dust region determination in the block is performed according to the processing flow shown in FIG. First, the maximum luminance Lmax and average luminance Lave in the block are calculated, and the threshold value T1 in the block is calculated using the following equation.

T1 = Lave × 0.6 + Lmax × 0.4
Next, pixels that do not exceed the threshold value are defined as dust pixels (step S61), and isolated regions constituted by dust pixels are each defined as one dust region di (i = 0, 1,..., N) (step S62). . As shown in FIG. 7, for each dust region, the maximum value Xmax and the minimum value Xmin of the horizontal coordinate of the pixels constituting the dust region, the maximum value Ymax and the minimum value Ymin of the vertical coordinate are obtained, and the dust region di A radius ri representing the size of is calculated by the following equation (step S63).

ri = √ [{(Xmax−Xmin) / 2} ² + {(Ymax−Ymin) / 2} ² ]
The relationship between Xmax, Xmin, Ymax, Ymin and ri is shown in FIG.

  Thereafter, in step S64, an average luminance value for each dust region is calculated.

  In some cases, the data size of the dust correction data is limited due to the limitation due to the size of the dust position memory 427. In order to cope with such a case, the dust position information is sorted according to the size and the average luminance value of the dust region (step S65). In this embodiment, sorting is performed in descending order of ri. When ri is equal, the sort is performed in ascending order of the average luminance value. In this way, it is possible to preferentially register noticeable dust in the dust correction data. The sorted dust area is Di, and the radius of the dust area Di is Ri.

  If there is a dust area larger than a predetermined size, it may be excluded from sorting and placed at the end of the sorted dust area list. This is because a large dust region may lower the image quality if interpolation processing is performed later, and is preferably treated as the lowest priority for correction.

(Uniform luminance shooting routine)
Next, details of the uniform luminance photographing routine performed in step S24 of FIG. 4 will be described using the flowchart shown in FIG. This uniform luminance photographing routine is executed by the microcomputer 402 executing a uniform luminance photographing program stored in the memory 428.

  When this uniform luminance photographing routine is executed, in step S201, the microcomputer 402 operates the quick return mirror 203 shown in FIG. 3, performs so-called mirror up, and retracts the quick return mirror 203 outside the photographing optical path.

  Next, in step S202, charge accumulation in the image sensor 418 is started, and in the next step S203, the shutter front curtain 210 and rear curtain 209 shown in FIG. In step S204, the charge accumulation of the image sensor is terminated. In the next step S205, image data is read from the image sensor 418, and the image data processed by the A / D converter 423 and the image processing circuit 425 is temporarily stored in the buffer memory 424. To do.

  When reading of all image data from the image sensor 418 is completed in the next step S206, the quick return mirror 203 is mirrored down in step S207, the quick return mirror is returned to the oblique position, and a series of imaging operations is completed.

(Normal shooting routine)
Next, the details of the normal shooting routine, which is a mode for shooting a normal subject, will be described using the flowchart shown in FIG. Shooting other than the uniform luminance shooting mode corresponds to this normal shooting routine. The processing is performed by the microcomputer 402 executing a normal photographing program stored in the memory 428.

  When this normal photographing routine is executed, in step S401, the microcomputer 402 operates the quick return mirror 203 shown in FIG. 3, performs so-called mirror-up, and retracts the quick return mirror 203 outside the photographing optical path.

  Next, in step S402, charge accumulation in the image sensor 418 is started, and in the next step S403, exposure is performed by running the shutter front curtain 210 and the rear curtain 209 shown in FIG. In step S404, the charge accumulation of the image sensor 418 is terminated, and in the next step S405, image data is read from the image sensor 418 and processed by the A / D converter 423 and the image processing circuit 425 in the buffer memory 424. Remember.

  When reading of all image data from the image sensor 418 is completed in the next step S406, the quick return mirror 203 is mirrored down in step S407, the quick return mirror is returned to the oblique position, and a series of imaging operations is completed.

  In step S300, dust correction data is updated based on the image data read in step S406. Since the dust area can be known from the dust correction data already stored in the dust position memory 427, it is confirmed whether a dust shadow is reflected in the dust area of the image data read in step S406. If there is no shadow of dust, it is determined that the dust has already disappeared, and the dust information is deleted.

  For example, the case where the image data read in step S406 is an image as shown in FIG. 11 will be described as an example. Assume that the i-th dust region 1100 obtained from dust correction data is an empty portion of the image of FIG. The sky is bright and almost uniform, so if there is any garbage, the shadow of the dust should be clearly reflected. Nevertheless, if there is no dust shadow in the i-th dust area 1100, it is highly likely that the dust has moved or has been removed, so the i-th dust data is extracted from the dust correction data. delete. That is, the i-th dust data is deleted from the dust correction data stored in the dust position memory 427, and the dust correction data in the dust position memory 427 is updated.

  As described above, by updating the dust correction data based on the image data obtained by the normal shooting, the reliability of the dust correction data can be increased. In particular, in the image pickup apparatus according to the present embodiment having a function of removing dust interposed between the photographing optical system 200 and the image pickup element 418, there is a high possibility that dust moves or falls after obtaining a uniform luminance image. . Therefore, it is effective to update the dust correction data based on the normal captured image as in the present embodiment. Further, in the imaging apparatus of this embodiment, dust correction data is automatically updated in the imaging apparatus, so that it is not necessary for the photographer to specify unnecessary dust information, and the photographer is forced to perform troublesome work. There is nothing.

  Note that the dust correction data update routine in step S300 will be described in more detail later.

  In step S408, the dust correction data (data updated in step S300) shown in FIG. 5 is recorded in the recording device 419 in association with the image data together with the camera setting value at the time of shooting.

  Specifically, for example, association can be realized by adding dust correction data to the Exif area, which is the header area of an image file in which camera setting values at the time of shooting are recorded. Alternatively, the dust correction data can be independently recorded as a file, and the association can be realized by recording only the link information to the dust correction data file in the image data. However, if the image file and the dust correction data file are recorded separately, the link relationship may be lost when the image file is moved. Therefore, it is desirable to hold the dust correction data integrally with the image data.

(Dust removal process)
Next, the flow of dust removal processing will be described. The dust removal process is performed not on the digital camera body but on a separately prepared image processing apparatus.

  FIG. 12 is a diagram showing an outline of the system configuration of the image processing apparatus.

  The CPU 1001 controls the operation of the entire system, and executes a program stored in the primary storage unit 1002. The primary storage unit 1002 is mainly a memory, and reads and stores programs stored in the secondary storage unit 1003. The secondary storage unit 1003 corresponds to, for example, a hard disk. Generally, the capacity of the primary storage unit is smaller than the capacity of the secondary storage unit, and programs and data that cannot be stored in the primary storage unit are stored in the secondary storage unit. Data that must be stored for a long time is also stored in the secondary storage unit. In the present embodiment, the program is stored in the secondary storage unit 1003, read into the primary storage unit 1002 when the program is executed, and the CPU 1001 performs an execution process.

  Examples of the input device 1004 include a mouse and keyboard used for system control, as well as a card reader, scanner, film scanner, and the like necessary for inputting image data. Examples of the output device 1005 include a monitor and a printer. Various other configurations of the apparatus configuration method are conceivable, but the description thereof is omitted because it is not the main point of the present invention.

  The image processing apparatus is equipped with an operating system capable of executing a plurality of programs in parallel, and an operator can operate a program operating on the apparatus using a GUI.

  The flow of dust removal processing in the image processing apparatus is shown in FIG.

  First, normal captured image data to which dust correction data is attached from the recording device 419 in the digital camera is taken into the image processing device and stored in the primary storage unit 1002 or the secondary storage unit 1003 (step S90).

  Next, the dust correction data added to the photographed image in step S408 in FIG. 10 is extracted from the normally photographed image data (image to be subjected to dust removal processing) (step S91).

  Next, the coordinate sequence Di (i = 1, 2,... N), the radius sequence Ri (i = 1, 2,..., N), the aperture value f1, and the lens pupil position L1 are extracted from the dust correction data extracted in step S91. Obtain (step S92). Here, Ri is the size of the dust at the coordinate Di calculated in step S65 of FIG.

  In step S93, the aperture value f2 and the lens pupil position L2 at the time of shooting of the normally shot image are acquired, and in step S94, Di is converted by the following equation. Here, d is a distance from the image center to the coordinate Di, and H is a distance between the surface of the image sensor 418 and dust. The coordinate Di ′ after conversion and the radius Ri ′ after conversion are defined by the following equation, for example.

Di ′ (x, y) = (L2 × (L1−H) × d / ((L2−H) × L1)) × Di (x, y)
Ri ′ = (Ri × f1 / f2 + 3) (1)
The unit here is a pixel, and “+3” for Ri ′ is a margin amount.

  In step S95, dust in the area indicated by the coordinates Di 'and the radius Ri' is detected, and interpolation processing is applied as necessary. Details of the interpolation processing will be described later.

  In step S96, it is determined whether or not dust removal processing has been applied to all coordinates. If processing has been completed for all coordinates, the processing is terminated; otherwise, the processing returns to step S95.

(Interpolation routine)
Next, details of dust region interpolation processing will be described. A flowchart showing the flow of the interpolation routine is shown in FIG. First, in step S1201, dust region determination is performed. The dust area is an area that satisfies all of the following conditions.
(1) Next, using the average luminance Yave and the maximum luminance Ymax of the pixels included in the center coordinates Di ′ and radius Ri ′ (Di ′, Ri ′ obtained in Expression (1)) calculated in step S94 in FIG. An area darker than the threshold value T2 obtained by the equation.

T2 = Yave × 0.6 + Ymax × 0.4
(2) A region not in contact with the circle having the center coordinate Di ′ and the radius Ri ′.
(3) A region in which the radius value calculated by the same method as in step S63 in FIG. 8 is not less than l1 pixels and less than l2 pixels with respect to the isolated region constituted by the low-luminance pixels selected in (1).
(4) A region including the center coordinates Di of the circle.

  In this embodiment, l1 is 3 pixels and l2 is 30 pixels. In this way, only an isolated small area can be handled as a dust area. If the lens pupil position cannot be obtained accurately, the condition (4) may have a width. For example, if the region of interest includes coordinates within a range of ± 3 pixels in the X direction and the Y direction from the coordinate Di, a condition such as determining as a dust region can be considered.

  If there is such an area in step S1202, the process advances to step S1203 to perform dust area interpolation. If there is no such area, the process ends. The dust region interpolation processing executed in step S1203 is performed by a known defect region interpolation method. Known defect area interpolation methods include, for example, pattern replacement disclosed in JP-A-2001-223894. In Japanese Patent Laid-Open No. 2001-223894, a defective region is specified using infrared light. In this embodiment, the dust region detected in step S1201 is treated as a defective region, and the dust region is treated as a normal pixel by pattern replacement. Interpolate with. For pixels that are not filled by pattern replacement, p normal pixels are selected in the order closest to the interpolation target pixel from the image data after pattern interpolation, and interpolation is performed using the average color.

  By attaching dust correction data to an image in this way, there is an advantage that it is not necessary to be aware of the correspondence between dust correction image data and captured image data. In addition, since the dust correction data is compact data composed of position, size, and conversion data (aperture value, distance information on the lens pupil position), the captured image data size does not become extremely large. Further, by performing interpolation processing only on the region including the pixel specified by the dust correction data, the probability of erroneous detection can be greatly reduced.

  Next, the dust correction data update routine executed in step S300 of FIG. 10 will be described using FIG.

  When the dust correction data update routine of step S300 in FIG. 10 is executed, dust correction data is first acquired in step S301. Dust correction data already stored in the dust position memory 427 is extracted.

  Next, the coordinate sequence Di (i = 1, 2,... N), the radius sequence Ri (i = 1, 2,..., N), the aperture value f1, and the lens pupil position L1 are extracted from the dust correction data extracted in step S301. Obtain (step S302). Here, Ri is the size of the dust at the coordinate Di calculated in step S65 of FIG.

  In step S303, the aperture value f2 and the lens pupil position L2 at the time of shooting of the normally shot image are acquired, and in step S304, Di is converted by the following equation. Here, d is a distance from the image center to the coordinate Di, and H is a distance between the surface of the image sensor 418 and dust. The coordinate Di ′ after conversion and the radius Ri ′ after conversion are defined by the following equation, for example.

Di ′ (x, y) = (L2 × (L1−H) × d / ((L2−H) × L1)) × Di (x, y)
Ri ′ = (Ri × f1 / f2 + 3) (2)
The unit here is a pixel, and “+3” for Ri ′ is a margin amount.

  In step S305, the output of the range of dust coordinates Di ′ (x, y) and radius Ri ′ converted in step S304 in the image data is read.

In step S306, it is determined whether there is a dust shadow in the area read in step S305. The presence / absence of a dust shadow is determined by determining that there is no dust shadow when all of the following conditions are satisfied, and that there is a dust shadow when one of the following conditions is not satisfied.
(1) The average luminance Yave of the pixels included in the center coordinate Di ′ and the radius Ri ′ (Di ′, Ri ′ obtained by the equation (2)) calculated in step S304 in FIG. 15 is larger than a certain threshold Y0. .

Yave> Y0
(2) All the luminances Y of the pixels included in the center coordinates Di ′ and the radius Ri ′ (Di ′, Ri ′ obtained by the expression (2)) fall within the following range.

K1 × Yave <Y <K2 × Yave Here, K1 and K2 are proportional constants (3), center coordinates Di ′ calculated in step S304 of FIG. 15, radius Ri ′ (Di ′, Ri determined by equation (2)) Using the average luminance Yave and the maximum luminance Ymax of the pixels included in '), there is no region darker than the threshold value T3 obtained by the following equation.

T3 = Yave × 0.6 + Ymax × 0.4
By checking whether or not the average luminance Yave is larger than the threshold Y0 in (1) and checking whether or not the luminance Y is within a certain range in (2), the region defined by the central coordinate Di ′ and the radius Ri ′ is It is checked whether it is a uniform image area. Further, in (3), whether or not there is a dust shadow is determined by checking whether or not there is a dust shadow in the area defined by the center coordinate Di ′ and the radius Ri ′.

  In (1), by examining whether the average luminance Yave is greater than the threshold value Y0, the following effects can be obtained. That is, since the light does not reach the area defined by the center coordinates Di ′ and the radius Ri ′, it is determined that the dust is not taken out and the dust is removed even though the dust is present. Can be prevented.

  Further, in (2), by checking whether the luminance Y is within a certain range, the region defined by the center coordinate Di ′ and the radius Ri ′ does not have a fine pattern, and it is reliably determined whether there is a shadow of dust. It is judged whether it is a pattern that can be performed. For example, if K1 = 0.8 and K2 = 1.2, it is determined whether the average luminance Yave is within a range of ± 20%. As a result, it is possible to prevent erroneous determination of the presence / absence of dust.

  In (3), by confirming again whether or not there is an area darker than the threshold value T3, a final confirmation is made as to whether or not the shadow of dust has really disappeared.

  By narrowing down with AND conditions of a plurality of judgment expressions (1) to (3), the risk of accidentally deleting dust data at that position even though dust remains attached is reduced.

  If there is no dust shadow in the i-th dust area read in step S305, the reliability parameter Ri is incremented (step S307). The reliability Ri is a parameter indicating the reliability of the i-th dust region. The initial value is “0”, and is incremented by 1 every time it is determined that there is no dust in the dust presence / absence determination. When the uniform luminance image is acquired, the reliability Ri is reset to the initial value “0”. In other words, the smaller this value, the more reliable the data.

  Next, it is determined whether or not the reliability Ri is greater than a certain threshold value R0 (step S308). If it is larger than the threshold value R0, it is determined that there is a high possibility that the dust has moved from the area or the dust has been removed, and the dust data in the i-th dust area is deleted.

  As described above, by deleting the dust area that clearly has no dust from the dust correction data, the reliability of the dust correction data can be increased. Thereby, it is possible to prevent unnecessary dust correction from being performed on an area where dust has been removed in subsequent shooting, and to improve the reliability of dust correction processing.

  If it is determined in step S306 that there is a dust shadow, the reliability Ri is smaller than the threshold value R0 in step S308, or the i-th dust area data is deleted in step S309, the i-th is the final n It is determined whether it is the second (step S310). If it is not final yet, i is incremented by one and the process proceeds to determination of the next dust region. In the final case, the dust correction data update routine is terminated.

  In the above description, the dust correction data is updated based on the single image information of the photographed image. However, a change in dust position is detected based on the image information of a plurality of images, and the dust correction data is updated. If the update is performed, the accuracy can be further improved.

  As described above, in the imaging apparatus of the present embodiment, the dust correction data update in step S300 is executed every time normal shooting is performed. Thereby, it is confirmed whether or not dust really exists in the dust area stored in the dust correction data, and the dust correction data is updated as necessary. This increases the reliability of the dust correction data and reduces the risk of erroneous dust correction processing. In addition, the dust correction data update process described above is automatically performed in the camera every time normal shooting is performed, so that a troublesome task for the photographer to specify unnecessary dust correction processing is not required.

  In addition, a method in which the photographer himself photographs a uniform luminance pattern and creates dust correction data is also conceivable. However, in order to increase the reliability of the dust correction data, it is necessary to photograph a uniform luminance pattern before or after the main photographing, which is not realistic because the burden on the photographer is too great. In the imaging apparatus according to the present embodiment, since the dust correction data is updated based on the captured image every time normal shooting is performed, the dust correction data is automatically generated without the photographer being conscious. Can be improved.

  According to the above configuration, every time normal shooting other than shooting for obtaining a uniform luminance image is performed, it is confirmed whether there is a dust shadow based on the obtained subject data, and dust correction is performed. By updating the data, the reliability of the dust correction data can be improved. In particular, in the image pickup apparatus of the present embodiment having a function of removing dust interposed between the photographing optical system and the image pickup device, there is a high possibility that dust will move or fall after obtaining a uniform luminance image. Therefore, it is very effective to update the dust correction data based on the normal captured image as in this embodiment.

  In addition, since a sequence for automatically updating dust correction based on the obtained subject data every time normal shooting is performed, a troublesome task for the photographer to specify unnecessary dust correction data becomes unnecessary. . This makes it possible to automatically update the dust correction data in the imaging apparatus without bothering the photographer.

(Second Embodiment)
In the first embodiment, the dust removal process is executed using an image processing apparatus prepared separately instead of the digital camera body. In the present embodiment, a method for executing dust removal processing in the digital camera body will be described. Since the configuration of the digital camera according to the second embodiment is the same as that of the first embodiment, description thereof will be omitted, and only operations different from those of the first embodiment will be described.

(Dust removal routine)
The dust removal routine in the digital camera according to this embodiment performs the same processing as the processing shown in the flowchart of FIG. This processing is performed by the microcomputer 402 executing a dust removal processing program stored in the memory 428.

  For example, when the start of dust removal processing is instructed from the cross placement switch 116, the microcomputer 402 calls the data corresponding to each pixel of the captured image stored in the image buffer memory 424 to the image processing circuit 425. The image processing circuit 425 performs the processing shown in FIG. 14 and executes dust pixel interpolation processing. Finally, the interpolation processing result is recorded in the recording device 419 as a new image file.

  The details of the processing in FIG. 14 have been described in the first embodiment, and will be omitted.

  Even in the imaging apparatus of the second embodiment, every time normal shooting is performed, the dust correction data update process in step S300 is executed, so that there is really dust in the dust area stored in the dust correction data. Make sure that The dust correction data is updated as necessary. This increases the reliability of the dust correction data and reduces the risk of erroneous dust correction processing. In addition, the dust correction data update process described above is automatically performed in the camera every time normal shooting is performed, so that a troublesome task for the photographer to specify unnecessary dust correction processing is not required.

  As described above, according to the second embodiment, dust can be suitably removed without being aware of the correspondence between dust detection image data and normal photographing data. Furthermore, even when dust correction data is embedded in an image, it is possible to attach the data without greatly increasing the size of the image data file.

(Other embodiments)
The object of each embodiment is also achieved by the following method. That is, a storage medium (or recording medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded is supplied to the system or apparatus. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but the present invention includes the following cases. That is, based on the instruction of the program code, an operating system (OS) or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

  Furthermore, the following cases are also included in the present invention. That is, the program code read from the storage medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, a CPU or the like provided in the function expansion card or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the above storage medium, the storage medium stores program codes corresponding to the procedure described above.

1 is a block diagram illustrating a circuit configuration of a lens interchangeable single-lens reflex digital camera as an imaging apparatus according to a first embodiment of the present invention. 1 is a perspective view illustrating an appearance of a digital camera according to a first embodiment. FIG. 3 is a vertical sectional view of FIG. 2. It is a flowchart for demonstrating a dust detection routine. It is a figure which shows the example of a data format of the dust correction data stored in a dust position memory. It is the figure which showed a mode that the dust area | region acquisition process was performed per block. It is the figure which showed the relationship between Xmax, Xmin, Ymax, Ymin, and ri of a dust region. It is a flowchart for demonstrating a garbage area | region acquisition routine. It is a flowchart for demonstrating a uniform luminance imaging | photography routine. It is a flowchart for demonstrating a normal imaging | photography routine. It is a figure which shows the example of the image data for demonstrating the concept of dust correction data update. It is the figure which showed the outline of the system configuration | structure of an image processing apparatus. It is a flowchart for demonstrating a dust removal routine. It is a flowchart for demonstrating an interpolation routine. It is a flowchart for demonstrating a dust correction data update routine.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Camera body 200 Shooting optical system 202 Lens mount 418 Image sensor

Claims (8)

An image sensor in which a plurality of pixels for photoelectrically converting a subject image are two-dimensionally arranged;
Foreign matter detection means for detecting information on the position of the foreign matter attached to the optical member disposed on the photographing optical axis in front of the imaging element;
Storage means for storing information on at least the position of the foreign matter detected by the foreign matter detection means;
Based on a captured image obtained by photoelectrically converting the subject image by the imaging element and information on the position of the foreign matter stored in the storage unit, a change in the state of attachment of the foreign matter to the optical member is detected. Change detecting means for
Updating means for updating the information on the position of the foreign matter stored in the storage means based on the information on the change in the adhesion state of the foreign matter detected by the change detection means;
An imaging apparatus comprising:   The imaging apparatus according to claim 1, wherein at least information related to the position of the foreign matter stored in the storage unit is attached to the captured image and stored in a storage medium.   The change detecting means detects whether or not the foreign substance exists at a position on the screen of the captured image corresponding to the information on the position of the foreign substance stored in the storage means, and detects that no foreign substance exists. In this case, the updating unit deletes the information on the position of the foreign object detected to be absent from the information on the position of the foreign object stored in the storage unit. The imaging device described.   The imaging apparatus according to claim 1, wherein the change detection unit detects a change in an adhesion state of the foreign matter to the optical member based on a plurality of the captured images.   The imaging apparatus according to claim 1, further comprising a removing unit that removes the foreign matter from the optical member. An image sensor in which a plurality of pixels that photoelectrically convert a subject image are two-dimensionally arranged, and a foreign object that detects information related to the position of a foreign object attached to an optical member disposed on a photographing optical axis in front of the image sensor A method for controlling an imaging apparatus comprising: a detection unit; and a storage unit that stores information on at least the position of the foreign matter detected by the foreign matter detection unit,
Based on a captured image obtained by photoelectrically converting the subject image by the imaging element and information on the position of the foreign matter stored in the storage unit, a change in the state of attachment of the foreign matter to the optical member is detected. A change detection step to
An update step of updating information on the position of the foreign matter stored in the storage unit based on information on a change in the adhesion state of the foreign matter detected in the change detection step;
An image pickup apparatus control method comprising:   A program for causing a computer to execute the control method according to claim 6.   A computer-readable storage medium storing the program according to claim 7.

Images