JP4960852B2 - Image processing apparatus, control method thereof, interchangeable lens single-lens reflex digital camera, and program - Google Patents

Description

  The present invention relates to a technique for reducing the influence of a shadow caused by a foreign substance adhering to a surface such as an optical low-pass filter, which appears in a photographed image photographed by an imaging device using an imaging device such as a CCD or CMOS sensor.

  2. Description of the Related Art In recent years, many image pickup apparatuses that generate image signals using an image pickup device such as a CCD and record the data as data, such as digital cameras and digital video cameras, have come into circulation. In a digital camera, a photosensitive film that has been used as a conventional recording medium is no longer necessary, and instead of this, a data image is recorded on a data recording medium such as a semiconductor memory card or a hard disk device. Since these data recording media can be written and erased any number of times unlike films, the cost for consumables can be reduced, which is very convenient.

  Usually, a digital camera is equipped with an LCD (Liquid Crystal Display) monitor device capable of displaying captured images at any time and a detachable mass storage device.

  Using a digital camera equipped with these two devices not only eliminates the need for a film as a recording medium that has been used as a consumable item in the past, but also allows the captured image to be displayed on the LCD monitor device and immediately confirmed on the spot. . Therefore, unsatisfactory image data can be erased on the spot or re-photographed as needed, and the efficiency of photography is dramatically increased compared to film cameras using film. I can say.

  Due to such convenience and technological innovations such as an increase in the number of pixels in the image sensor, the range of use of digital cameras has been expanded. In recent years, there are many digital cameras capable of exchanging lenses, such as single-lens reflex cameras.

  However, in the single-lens reflex digital camera as described above, dust, dust, etc. on the surface of an optical system (hereinafter collectively referred to as an image sensor optical system component) such as an image sensor protective glass and an optical filter fixed to the image sensor. Of foreign matter (hereinafter simply referred to as dust) may adhere. When dust adheres to the image sensor optical system component in this way, there is a problem that the quality of the photographed image is deteriorated, for example, light is blocked by the dust and the portion is not photographed.

  Not only digital cameras but also cameras using silver halide film have a problem of being captured if there is dust on the film, but in the case of film, the film moves from frame to frame, so it is the same for all frames It is very rare for trash to appear.

  However, since the image pickup device of the digital camera does not move and the image is taken with one image pickup device, once dust adheres to the image pickup device optical system parts, the same dust appears in many frames (photographed images). . Particularly in an interchangeable lens digital camera, there is a problem that dust easily enters the camera when the lens is replaced.

  Therefore, the photographer must always pay attention to the adhesion of dust to the image sensor optical system parts, and has spent a lot of labor on checking and cleaning the dust. In particular, since the image sensor is disposed at a relatively deep location inside the camera, it is not easy to clean or check for dust.

  Furthermore, in an interchangeable lens digital camera, not only does dust intrude due to the attachment / detachment of the lens, but also dust on the image sensor optical system components due to scraping by driving a focal plane shutter placed immediately before the image sensor. Easy to adhere.

Therefore, there is a method to make dust inconspicuous by taking a picture of a white wall etc. with the lens aperture closed and preparing an image showing only dust on the image sensor in combination with a normal shot image. It is known (see Patent Document 1). However, with this method, it is necessary to always be aware of the association between the image captured for dust detection and the actual captured image group associated therewith, which is cumbersome. Therefore, after capturing the position of dust by shooting a white wall etc., hold it on the digital camera, and attach a list of dust positions and sizes (dust information) to the image data during normal shooting Can be considered. For example, by using a separately prepared image processing apparatus, the dust position on the photographed data is analyzed from the attached dust position, and the analyzed area is interpolated with surrounding pixels, thereby making it less noticeable.
JP 2004-222231 A

  On the other hand, applications for correcting lens characteristics, particularly distortion aberration, lateral chromatic aberration, and the like have appeared by using information on a photographing lens at the time of photographing. By performing such lens aberration correction processing, a more uniform and high-quality image can be obtained. However, since such processing deforms the image, there is a problem that the coordinates recorded in the dust information and the actual position of the dust image are shifted, and the dust removal processing cannot be executed correctly.

  In order to avoid such a problem, it is conceivable to perform dust removal processing before applying aberration correction processing according to lens characteristics. However, such a processing order may not be possible due to the division of roles between the camera and the application and restrictions on the image processing order within the application.

  For example, when a lens aberration correction process is performed on a photographed image in the camera and a dust removal process is executed by an image processing apparatus outside the camera, the dust removal process is performed after the lens aberration correction process. It will be.

  Therefore, the present invention has been made in view of the above-described problems, and an object of the present invention is to perform subsequent dust removal processing with high accuracy even when lens aberration correction is performed on a captured image first. It is to be.

  In order to solve the above-described problems and achieve the object, an image processing apparatus according to the present invention includes position information of a foreign object in an imaging screen extracted based on an image signal for foreign object detection obtained from an imaging unit. , And foreign object information acquisition means for acquiring lens information of the first photographing lens used when acquiring the image signal as foreign object information, and lens information for acquiring lens information of the second photographing lens at the time of photographing the subject Acquisition means; storage means for storing aberration information including at least aberration information of the second photographing lens; position information of the foreign matter; lens information of the first photographing lens; and lenses of the second photographing lens. Based on the information, the first conversion means for converting the position information of the foreign matter in the imaging screen when the second photographing lens is photographed, and the difference converted by the first conversion means. The position information of the foreign matter in the imaging screen after the subject image photographed at the time of photographing the subject is corrected by the image processing based on the lens information of the second photographing lens and the aberration information. And a second conversion means for converting to a second conversion means.

  Further, another image processing apparatus according to the present invention acquires the position and size information of the foreign matter in the imaging screen extracted based on the image signal for foreign matter detection obtained from the imaging means, and the image signal. Foreign matter information acquisition means for acquiring the lens information of the first photographing lens used when performing the processing as foreign matter information, lens information acquisition means for acquiring lens information of the second photographing lens at the time of subject photographing, and at least the first Storage means for storing aberration information including aberration information of the second photographing lens, and position and size information of the foreign matter based on the lens information of the first photographing lens and the lens information of the second photographing lens. The first conversion means for converting the position and size information of the foreign matter in the imaging screen when the second photographing lens is used for photographing, and the first conversion means for converting the information. The object position and size information is further displayed in the imaging screen after the subject image photographed at the time of photographing the subject is subjected to aberration correction by image processing based on the lens information and the aberration information of the second photographing lens. And a second conversion means for converting into the position and size information of the foreign matter.

  In addition to the above means, the interchangeable lens single-lens reflex digital camera according to the present invention acquires the image signal for detecting the foreign material in the foreign material detection mode, and extracts the position information of the foreign material in the imaging screen from the image signal. It has the extraction means to perform.

  Further, the control method of the image processing apparatus according to the present invention obtains the position information of the foreign matter in the imaging screen extracted based on the image signal for foreign matter detection obtained from the imaging means, and the image signal. Foreign matter information acquisition step for acquiring lens information of the first photographing lens used in the process as foreign matter information, lens information acquisition step for obtaining lens information of the second photographing lens at the time of subject photographing, and positional information of the foreign matter Is converted into position information of the foreign matter in the imaging screen when the second photographing lens is photographed based on the lens information of the first photographing lens and the lens information of the second photographing lens. The position information of the foreign matter converted in the conversion step and the first conversion step further includes lens information of the second photographing lens and at least aberration information of the second photographing lens. Based on the aberration information stored in the storage means for storing information, a second subject image is converted into foreign object position information in the imaging screen after aberration correction is performed by image processing on the subject image captured during the subject photographing. A conversion step.

  In addition, another image processing apparatus control method according to the present invention includes the position and size information of the foreign matter in the imaging screen extracted based on the image signal for foreign matter detection obtained from the imaging means, and the image. A foreign matter information acquisition step of acquiring lens information of the first photographing lens used when acquiring the signal as foreign matter information, a lens information acquisition step of acquiring lens information of the second photographing lens at the time of subject photographing, Based on the lens information of the first photographing lens and the lens information of the second photographing lens, the position and size information of the foreign matter is determined based on the second photographing lens. A first conversion step for converting the information into position and size information, and the position and size information of the foreign matter converted in the first conversion step, and further lens information of the second photographing lens. After the subject image photographed at the time of photographing the subject is corrected for aberration by image processing based on the aberration information stored in the storage means for storing aberration information including at least aberration information of the second photographing lens And a second conversion step of converting into foreign object position and size information in the imaging screen.

  According to the present invention, even when lens aberration correction is performed on a captured image first, it is possible to perform subsequent dust removal processing with high accuracy.

DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.
(Configuration of imaging device)
FIG. 1 is a block diagram illustrating a circuit configuration of a lens interchangeable single-lens reflex digital camera (hereinafter simply referred to as a camera) as an imaging apparatus according to an embodiment of the present invention.

  In FIG. 1, a microcomputer 402 controls the operation of the entire camera, including processing of image data output from an image sensor (CCD in this embodiment) 418 that photoelectrically converts a subject image and display control of an LCD monitor device 417. To do.

  The switch (SW1) 405 is turned on when the release button 114 (see FIG. 2) is half-pressed. When the switch (SW1) 405 is turned on, the camera according to the present embodiment is in a shooting preparation state. The switch (SW2) 406 is turned on when the release button 114 is pressed to the end (fully pressed state), and when the switch (SW2) 406 is turned on, the camera of this embodiment starts a photographing operation.

  The lens control circuit 407 performs communication control with the photographic lens 200 (see FIG. 3) and drive control of the photographic lens 200 and drive control of the diaphragm blades during AF (autofocus).

  In FIG. 1, an external display control circuit 408 controls an external display device (OLC) 409 and a display device (not shown) in the viewfinder. The switch sense circuit 410 transmits signals of a large number of switches including the electronic dial 411 provided in the camera to the microcomputer 402.

  The strobe light emission dimming control circuit 412 is grounded via the X contact 412a and controls the external strobe. The distance measuring circuit 413 detects the defocus amount with respect to the subject for AF. The photometry circuit 414 measures the luminance of the subject.

  A shutter control circuit 415 controls the shutter and performs appropriate exposure on the image sensor. The LCD monitor device 417 and the backlight illumination device 416 constitute an image display device. The recording device 419 is, for example, a hard disk drive or a semiconductor memory card that can be attached to and detached from the camera body.

  Further, the microcomputer 402 includes an A / D converter 423, an image buffer memory 424, an image processing circuit 425 including a DSP, and a pixel defect position memory 426 that stores that a predetermined pixel in the image sensor itself is defective. Is connected. Also connected is a dust position memory 427 that stores the pixel position in the image pickup device causing the image defect due to dust. Note that the pixel defect position memory 426 and the dust position memory 427 are preferably non-volatile memories. Further, the pixel defect position memory 426 and the dust position memory 427 may be stored using different addresses in the same memory space.

  Reference numeral 428 denotes a nonvolatile memory that stores programs executed by the microcomputer 402.

  The lens characteristic data supply unit 430 stores a correction coefficient for correcting aberration corresponding to the characteristics of various photographing lenses in a memory (not shown), and supplies the correction coefficient to the microcomputer 402. In the aberration correction processing according to the photographic lens in the present embodiment, a correction function is determined in advance, and the correction amount is calculated by applying the correction coefficient supplied from the lens characteristic data supply unit 430 to the correction function. The image is deformed according to the correction amount. Details of the aberration correction processing of the photographing lens will be described later. The storage destination of the correction coefficient is not limited to the memory included in the lens characteristic data supply unit 430 and may be stored in the recording device 419. Whether or not to apply aberration correction processing of the taking lens can be selected on a menu screen described later.

  FIG. 2 is a perspective view showing the appearance of the camera according to the present embodiment, and FIG. 3 is a vertical sectional view of FIG.

In FIG. 2, an eyepiece window 111 for finder observation, an AE (automatic exposure) lock button 112, an AF distance measuring point selection button 113, and a release button 114 for performing a photographing operation are provided on the upper part of the camera body 100. ing. An electronic dial 411, a shooting mode selection dial 117, and an external display device 409 are also provided. The electronic dial 411 is a multi-function signal input device that is used in combination with other operation buttons to input numerical values to the camera and to switch the shooting mode. The external display device 409 includes a liquid crystal display device, and displays shooting conditions such as shutter speed, aperture, and shooting mode, and other information.
Also, on the back of the camera main body 100, an LCD monitor device 417 for displaying captured images and various setting screens, a monitor switch 121 for turning on / off the LCD monitor device 417, a cross switch 116, and a menu button 124 is provided.

  The cross placement switch 116 has four buttons arranged vertically and horizontally and a SET button arranged in the center. The user instructs the camera to select and execute menu items displayed on the LCD monitor device 417. Used for.

  The menu button 124 is a button for causing the LCD monitor device 417 to display a menu screen for performing various camera settings. For example, when selecting and setting the shooting mode, after pressing the menu button 124, the user operates the up / down / left / right buttons of the cross switch 116 to select the desired mode, and the desired mode is selected. Setting is completed by pressing the SET button. The menu button 124 and the cross-shaped switch 116 are also used for setting a dust detection mode, which will be described later, and for setting a display mode and an identification mark in the dust detection mode.

  Since the LCD monitor device 417 of the present embodiment is a transmissive type, an image cannot be visually recognized only by driving the LCD monitor device, and a backlight illumination device 416 is necessarily provided on the back surface thereof as shown in FIG. is there. As described above, the LCD monitor device 417 and the backlight illumination device 416 constitute an image display device.

  As shown in FIG. 3, the taking lens 200 that is an imaging optical system can be attached to and detached from the camera body 100 via a lens mount 202. In FIG. 3, 201 denotes a photographing optical axis, and 203 denotes a quick return mirror.

  The quick return mirror 203 is disposed in the photographing optical path, and a position for guiding the subject light from the photographing lens 200 to the finder optical system (referred to as a position shown in FIG. 3, an oblique position) and a position for retracting outside the photographing optical path (retraction position). Is called).

  In FIG. 3, the subject light guided from the quick return mirror 203 to the finder optical system is imaged on the focus plate 204. Reference numeral 205 denotes a condenser lens for improving the visibility of the finder, and 206 denotes a pentagonal roof prism.

  Reference numerals 209 and 210 respectively denote a rear curtain and a front curtain that constitute a shutter, and an image sensor 418 that is a solid-state image sensor disposed behind the rear curtain 209 and the front curtain 210 is exposed for a necessary time. A captured image converted into an electrical signal for each pixel by the image sensor is processed by the A / D converter 423, the image processing circuit 425, and the like, and recorded in the recording device 419 as image data. An optical element 418a such as an optical low-pass filter is disposed in front of the image sensor 418, and foreign matters such as dust adhering to the surface of the optical element 418a are reflected in the imaging screen of the image sensor 418. .

  The image sensor 418 is held on the printed circuit board 211. Behind this printed board 211, a display board 215, which is another printed board, is arranged. An LCD monitor device 417 and a backlight illumination device 416 are disposed on the opposite surface of the display substrate 215.

  Reference numeral 419 denotes a recording device for recording image data, and 217 denotes a battery (portable power supply). The recording device 419 and the battery 217 are detachable from the camera body.

(Lens aberration correction routine)
A distortion aberration correction process will be described as an example of the lens aberration correction process in the present embodiment.

  In order to perform the lens aberration correction process, identification information for identifying a lens, focal length information for photographing for specifying a zoom position, and optical information for photographing such as an F value as an aperture value are necessary. In the present embodiment, these pieces of information are stored in a MakerNote tag of Exif information, which is header information of photographed image data at the time of subject photographing. By storing in this way, it is possible to manage the captured image and the optical information at the time of shooting one-on-one. In order to further improve the correction accuracy, useful information such as subject distance information may be stored in addition to the above information.

  FIG. 4 is a diagram showing the relationship between the image height before correction of distortion and the image height after correction. Here, the image height is a distance from the center of the image to the pixel (i, j). As shown in the figure, the conversion from the image height before distortion aberration correction to the image height after distortion aberration correction is forward converted, and the conversion from image height after distortion aberration correction to image height before distortion aberration correction is inversely converted. . In the figure, the lens characteristic data is a curve determined by optical information at the time of shooting such as focal length information at the time of shooting and F value. The distortion aberration correction process is performed using a distortion aberration correction lookup table that is a table of correspondence between the image height before distortion aberration correction and the image height after distortion aberration correction. Accordingly, the distortion correction lookup table is prepared corresponding to various optical information at the time of shooting, and these are stored in the lens characteristic data supply unit 430.

  In general, image processing involving pixel conversion is generally performed by inverse conversion. That is, the pixel before conversion is obtained using the inverse conversion table with the converted pixel as a reference, and the image data (pixel value) in the pixel before the conversion is set as image data (pixel value) in the pixel after conversion. It is a replacement process. Image processing by inverse transformation is desirable in this way because in the case of a bitmap that is a digital image, the pixel before processing does not necessarily correspond to the pixel after processing.

  FIG. 5 is a flowchart showing the flow of distortion correction processing in this embodiment.

  In step S1801, optical information (including aberration information) is acquired from the Exif information of the processing target image (lens information acquisition), and a distortion aberration correction lookup table corresponding to the information is acquired from the lens characteristic data supply unit 430. .

  In step S1802, the pixel pitch is acquired from the Exif information of the processing target image. The pixel pitch is a distance expressed between adjacent pixels in millimeters.

  In step S1803, an image size of an output image that is an image after distortion correction is obtained. A method for calculating the image size of the output image will be described later.

  Next, in step S1804, using the image size (width fw pixels, height fh pixels) obtained in step S1803, the lookup table size fs is calculated by the following equation. fs is the distance in pixels from the center of the output image to the vertex.

fs = √ (((fw / 2) * (fw / 2)) + ((fh / 2) * (fh / 2)))
In step S1805, the distortion correction lookup table obtained in step S1801 is spline interpolated and converted into a distortion aberration correction lookup table corresponding to the lookup table size fs obtained in step S1804. Here, the input value of the distortion aberration correction lookup table is the distance in pixels from the center of the output image (image after distortion correction), and the output value is from the center of the input image (image before distortion aberration correction). The distance in pixels. At this time, in order to convert the distance in pixels to the distance in millimeters, the pixel pitch obtained in step S1802 is used.

  In step S1806, a pixel value calculation process, which will be described later, is executed for each pixel constituting the image having the image size obtained in step S1803. When the pixel values are calculated for all the pixels, the process ends.

  FIG. 6 is an explanatory diagram for explaining image processing in which distortion is corrected by the above-described inverse transformation.

  Distortion may occur when a barrel-shaped distortion or a pincushion-type distortion occurs.For example, if all pixels of a rectangular image with a barrel-shaped distortion are corrected to an undistorted image, The output image is an amoeba-shaped image with four vertices sharpened at an acute angle. In the figure, this shape is represented by a solid line as an image after distortion correction. As a final output image, a rectangular image represented by a dotted line is cut out.

Here, when it is desired to obtain the pixel value of the coordinate P (i, j) in the image after distortion correction, first, the distance d from the center of this coordinate is obtained. Then, an output value d ′ with respect to the input value d is obtained from the distortion aberration correction lookup table. That is, the image height before distortion aberration correction is obtained from the image height after distortion aberration correction. Then, a coordinate P ′ (i ′, j ′) having a distance d ′ in the direction of θ = tan ⁻¹ (j / i) is obtained in the image before distortion correction correction. Here, P ′ (i ′, j ′) is not necessarily an integer coordinate. Therefore, the pixel value of P ′ (i ′, j ′) is calculated by bilinear interpolation using the pixel values in the integer coordinates of the surrounding four points including the coordinates. When the inverse transformation process is performed in this way, the coordinate value at the coordinate P (i, j) after distortion correction can be obtained as the coordinate value of the coordinate P ′ (i ′, j ′) before distortion correction. . If this operation is repeated for all coordinates within the rectangle surrounded by the dotted line of the image after distortion correction, all pixel values of the output image can be acquired.

  Here, a preferable method of determining the size of the output image that is a rectangle surrounded by a dotted line in FIG. 6 will be described.

  FIG. 7 is a diagram for explaining the relationship by taking out the first quadrant of the image before the distortion correction (dashed line) and the image after the distortion correction (dotted line). Note that the maximum value in the x-axis direction of the image before distortion correction is w / 2, and the maximum value in the y-axis direction is h / 2.

  First, in the x-axis direction, the y-coordinate before distortion correction is fixed to h / 2, and what kind of curve is drawn when the x-coordinate is changed from 0 to w / 2. Here, the distortion correction look-up table is prepared for the inverse conversion processing as the input value is the distance in pixels from the center of the output image and the output value is the distance in pixels from the center of the input image. Suppose that it is remade for forward conversion processing. By using the distortion aberration correction look-up table for forward conversion processing that has been recreated in this way, coordinates after repeated conversion from (0, h / 2) to (w / 2, h / 2) are obtained.

  Next, similarly, in the y-axis direction, the x coordinate before distortion correction is fixed to w / 2, and what curve is drawn when the y coordinate is changed from 0 to h / 2. That is, the coordinates after repeated conversion from (w / 2, 0) to (w / 2, h / 2) are obtained using a distortion aberration correction lookup table for forward conversion processing.

  A combination of the coordinates thus obtained is an outline after distortion correction, which is represented by a solid line in the figure. This shape is the same shape as the first quadrant of the amoeba shape in which the four vertices are sharpened as described with reference to FIG.

  In the outline thus obtained, a rectangle inscribed therein, that is, w ′ / 2 which is the minimum value in the x-axis direction and h ′ / 2 which is the minimum value in the y-axis direction, are respectively expressed in the x-axis and y-axis. Is defined as the size of the output image (first quadrant).

  In this example, a rectangular image with a barrel-shaped distortion is corrected. However, when a rectangular image with a pincushion-shaped distortion is corrected, the output outer shape is an amoeba shape. The shape is similar to an ellipse. Even in this case, the size of the output image can be determined by performing the same processing and obtaining the inscribed rectangle.

  According to the size of the output image that is a rectangle defined in this way, if the inverse transformation process described with reference to FIG. 6 is repeated for all coordinates, all pixel values of the output image can be acquired. Become.

(Dust detection process)
FIG. 8 is a flowchart for explaining dust detection processing (detection processing of a pixel position where an image defect is caused by dust) in the foreign matter detection mode of the camera according to the present embodiment. This processing is performed by the microcomputer 402 executing a dust detection processing program stored in the memory 428.

  The dust detection processing (foreign matter information acquisition processing) is performed by capturing a dust detection image (foreign matter detection image). When performing dust detection processing, a camera is installed with the photographing optical axis 201 of the photographing lens 200 facing a surface having a uniform color, such as an emission surface of a surface light source device or a white wall, and preparation for dust detection is performed. Alternatively, a dust detection light unit (a small point light source device mounted instead of a lens) is mounted on the lens mount 202 to prepare for dust detection. The light source of the light unit is, for example, a white LED, and it is desirable to adjust the size of the light emitting surface so that it corresponds to a predetermined aperture value (for example, F16 in the present embodiment).

  In this embodiment, a case where a normal photographing lens is used will be described. However, dust detection may be performed by attaching the light unit to the lens mount 202. Thus, in this embodiment, the dust detection image is an image having a uniform color.

  After the preparation is completed, for example, when the start of dust detection processing is instructed from the cross arrangement switch 116, the microcomputer 402 first sets the aperture. The image formation state of dust near the image sensor changes depending on the aperture value of the lens, and the position changes depending on the pupil position of the lens. Accordingly, the dust correction data (foreign matter information) needs to hold the aperture value and the pupil position of the lens in addition to the dust position and size.

  However, it is not always necessary to hold the aperture value in the dust correction data if it is determined in advance that the same aperture value is always used even when different lenses are used at the stage of creating dust correction data. Similarly, by using a light unit or permitting the use of only a specific lens for the pupil position, it is not always necessary to hold the pupil position in the dust correction data. In other words, at the stage of creating dust correction data, if you want to allow multiple lenses to use or change the aperture value to narrow down appropriately, it is necessary to keep the aperture value and lens pupil position at the time of detection in the dust correction data It can be said that there is. Here, the pupil position refers to the distance from the imaging plane (focal plane) of the exit pupil.

  Here, for example, F16 is designated (step S21).

  Next, the microcomputer 402 causes the lens control circuit 407 to perform aperture blade control of the photographing lens 200, and sets the aperture to the aperture value designated in step S21 (step S22). Further, the focus position is set to infinity (step S23).

  When the aperture value and focus position of the photographing lens are set, photographing in the dust detection mode is executed (step S24). Details of the imaging processing routine performed in step S24 will be described later with reference to FIG. The captured image data is stored in the buffer memory 424.

  When shooting is completed, the aperture value and lens pupil position at the time of shooting are acquired (step S25). Data corresponding to each pixel of the photographed image stored in the image buffer memory 424 is called to the image processing circuit 425 (step S26). The image processing circuit 425 performs the processing shown in FIG. 10, and acquires the position and size of the pixel where dust is present (step S27). The position and size of the pixel where the dust exists in step S27 and the aperture value and lens pupil position information acquired in step S25 are registered in the dust position memory 427 (step S28). Here, when the light unit described above is used, lens information cannot be acquired. Therefore, when the lens information cannot be acquired, it is determined that the light unit is used, and predetermined lens pupil position information and a converted aperture value calculated from the light source diameter of the light unit are registered.

  Here, in step S28, the position of the defective pixel (pixel defect) from the time of manufacture recorded in the pixel defect position memory in advance is compared with the position of the read pixel data to check whether the pixel defect is present. Only the area determined not to be due to the pixel defect may be registered in the dust position memory 427.

An example of the data format of dust correction data stored in the dust position memory 427 is shown in FIG. As shown in FIG. 9, the dust correction data stores lens information (lens characteristic information) and dust position and size information when a detection image is captured (when an image signal is acquired). This dust correction data is added to the image together with the shooting time information of the image data during normal shooting, and used in dust removal processing described later.
Specifically, the actual aperture value (F value) at the time of detection image capture and the lens pupil position at that time are stored as lens information at the time of detection image capture. The number of detected dust areas (integer value) is stored in the subsequent storage area, and subsequently, individual specific dust area parameters are repeatedly stored for the number of dust areas. The parameter of the dust area is a set of three numerical values: a dust radius (for example, 2 bytes), a center x coordinate (for example, 2 bytes) in the effective image area, and a similar y coordinate (for example, 2 bytes).

  When the dust correction data size is limited due to the size of the dust position memory 427 or the like, the data is stored with priority from the head of the dust area obtained in step S27. This is because in the dust area acquisition routine in step S27, the dust areas are sorted in the order of noticeable dust, as will be described later.

(Trash area acquisition routine)
Next, details of the dust region acquisition routine performed in step S27 will be described with reference to FIGS.

  As shown in FIG. 11, the called image data is developed on a memory and processed in units of predetermined blocks. This is to cope with peripheral dimming caused by lens and sensor characteristics. Peripheral dimming is a phenomenon in which the luminance at the peripheral portion is lower than that at the central portion of the lens, and it is known that it is reduced to some extent by reducing the aperture of the lens. However, there is a problem that even if the aperture is reduced, if the dust position is determined based on a predetermined threshold value for the photographed image, dust in the peripheral portion cannot be detected accurately depending on the lens. Therefore, the image is divided into blocks to reduce the influence of peripheral dimming.

  When the blocks are simply divided, there is a problem that the detection result of the dust straddling the blocks is shifted when the threshold values are different between the blocks. Therefore, the blocks are overlapped, and pixels determined to be dust in any of the blocks constituting the overlap region are treated as dust regions.

  The determination of the dust area in the block is performed according to the process flow shown in FIG. First, the maximum luminance Lmax and average luminance Lave in the block are calculated, and the threshold value T1 in the block is calculated using the following equation.

T1 = Lave × 0.6 + Lmax × 0.4
Next, pixels that do not exceed the threshold value are defined as dust pixels (step S61), and isolated regions constituted by dust pixels are each defined as one dust region di (i = 0, 1,..., N) (step S62). . As shown in FIG. 12, for each dust region, the maximum value Xmax and the minimum value Xmin of the horizontal coordinate of the pixels constituting the dust region, the maximum value Ymax and the minimum value Ymin of the vertical coordinate are obtained, and the dust region di A radius ri representing the size of is calculated by the following equation (step S63).

ri = √ [{(Xmax−Xmin) / 2} ² + {(Ymax−Ymin) / 2} ² ]
FIG. 12 shows the relationship between Xmax, Xmin, Ymax, Ymin and ri.

  Thereafter, in step S64, an average luminance value for each dust region is calculated.

  In some cases, the data size of the dust correction data is limited due to the limitation due to the size of the dust position memory 427. In order to cope with such a case, the dust position information is sorted according to the size and the average luminance value of the dust region (step S65). In this embodiment, sorting is performed in descending order of ri. When ri is equal, the sort is performed in ascending order of the average luminance value. In this way, it is possible to preferentially register noticeable dust in the dust correction data. The sorted dust area is Di, and the radius of the dust area Di is Ri.

  If there is a dust area larger than a predetermined size, it may be excluded from sorting and placed at the end of the sorted dust area list. This is because a large dust region may lower the image quality if interpolation processing is performed later, and it is desirable to treat it as the lowest priority for editing.

(Imaging processing routine)
Next, details of the imaging processing routine performed in step S24 of FIG. 8 will be described using the flowchart shown in FIG. The processing is performed by the microcomputer 402 executing an imaging processing program stored in the memory 428.

  When this imaging processing routine is executed, in step S201, the microcomputer 402 operates the quick return mirror 203 shown in FIG. 3, performs so-called mirror up, and retracts the quick return mirror 203 outside the imaging optical path.

  Next, charge accumulation in the image sensor is started in step S202, and in the next step S203, exposure is performed by running the shutter front curtain 210 and rear curtain 209 shown in FIG. In step S204, the charge accumulation of the image sensor is terminated, and in the next step S205, the image signal is read from the image sensor, and the image data processed by the A / D converter 423 and the image processing circuit 425 is temporarily stored in the buffer memory 424. .

  When reading of all image signals from the image sensor is completed in the next step S206, the quick return mirror 203 is mirrored down in step S207, the quick return mirror is returned to the oblique position, and a series of imaging operations is completed.

  In step S208, it is determined whether normal shooting (photographing of the subject image) or dust detection image shooting is performed. In normal shooting, the process proceeds to step S209, and the dust correction data shown in FIG. The data is recorded in the recording device 419 in association with the data.

  Specifically, for example, association can be realized by adding dust correction data to the Exif area, which is the header area of an image file in which camera setting values at the time of shooting are recorded. Alternatively, the dust correction data can be independently recorded as a file, and the association can be realized by recording only the link information to the dust correction data file in the image data. However, if the image file and the dust correction data file are recorded separately, the link relationship may be lost when the image file is moved. Therefore, it is desirable to hold the dust correction data integrally with the image data.

(Conversion of dust correction data)
When both the lens aberration correction process and the dust removal process are performed on the captured image, there are cases where the lens aberration correction process is performed first and then the dust removal process is performed next, and vice versa. If dust removal processing is performed first and then lens aberration correction processing is performed, dust correction data can be used without considering the influence of the aberration correction processing. Therefore, it is easier to perform dust removal processing before performing distortion correction processing. However, there are many cases where an imaging device that performs shooting and an image processing device that allows a user to process / view captured image data are different, and there is a problem of division of roles in what type of image processing is performed by each device. is there. In addition, there is a problem that the area to be subjected to dust removal processing is not necessarily an area that can be appropriately corrected in relation to the subject image. For example, when trying to correct dust existing across a subject with a significant contrast difference, there is a risk that the difference in contrast will be eliminated by the process. In other words, there is a case where it is not desired to perform dust removal processing, and processing that requires such determination is complicated and time consuming, so it is advantageous to perform it as late as possible.

  Therefore, in the present embodiment, first, lens aberration correction processing is performed, and then dust removal processing is performed. When processing is performed in this order, dust correction data must be converted according to lens aberration.

  In the dust correction data format, as described with reference to FIG. 9, the dust area is represented by a set of three numerical values: a dust radius, a center x coordinate, and a center y coordinate.

  First, the coordinate conversion involves obtaining the coordinate point after the distortion correction from the coordinate point before the distortion correction, so that the process by forward conversion is performed. Here, the distortion correction look-up table is prepared for the inverse conversion processing as the input value is the distance in pixels from the center of the output image and the output value is the distance in pixels from the center of the input image. Suppose that it is remade for forward conversion processing. Then, the coordinate A (Xa, Ya) described in the dust correction data is converted into A ′ (Xa ′, Ya ′) using the reconstructed distortion aberration correction lookup table for forward conversion processing.

  Specifically, the image height da, which is the distance from the center of the coordinate A, is input to the reconstructed distortion aberration correction look-up table, and the image height da ′ after distortion aberration correction is obtained. Then, a coordinate A ″ (Xa ″, Ya ″) having a distance da ′ in the direction of θ = tan−1 (Ya / Xa) is obtained. Then, an integer coordinate closest to A ″ is determined as A ′. And

  Next, radius conversion is performed. When distortion correction is performed, if the image is converted in a direction in which the image height is reduced, the dust area after conversion is smaller than that before conversion, and if the image is converted in a direction in which the image height is increased, the reverse operation is performed. In addition, the dust area after the conversion becomes larger than before the conversion. This depends on whether the distortion aberration that occurs is a pincushion mold or a barrel mold. Radial conversion pays attention to this relationship and uses the following conversion formula.

ra ′ = ra + α (da′−da)
Here, ra ′ is a radius after conversion, and ra is a radius before conversion. da ′ is the image height after conversion of the above-described center coordinates A (Xa, Ya), and da is the image height before conversion. α is an experimentally determined constant that allows for a margin. By using such a conversion formula, it is possible to appropriately convert the radius according to the distortion aberration type and coordinate position.

  By performing such conversion on all dust data described in the dust correction data, it is possible to obtain converted dust correction data.

  Here, both the position and the size of the dust are converted, but it is often unnecessary to convert the size so strictly as compared to the position. Therefore, if the accuracy is not required, the conversion is not performed. The size at the time of dust detection may be used.

(Process for removing dust)
Next, the flow of processing for removing dust (foreign matter removal processing) will be described. The processing for removing dust is performed not on the camera body but on a separately prepared image processing apparatus. For simplicity, the case where lens aberration correction is not performed will be described first. A description of the case where dust removal processing is performed after lens aberration correction will be described later.

  FIG. 14 is a diagram showing an outline of the system configuration of the image processing apparatus.

  The CPU 1001 controls the operation of the entire system, and executes a program stored in the primary storage unit 1002. The primary storage unit 1002 is mainly a memory, and reads and stores programs stored in the secondary storage unit 1003. The secondary storage unit 1003 corresponds to, for example, a hard disk. Generally, the capacity of the primary storage unit is smaller than the capacity of the secondary storage unit, and programs and data that cannot be stored in the primary storage unit are stored in the secondary storage unit. Data that must be stored for a long time is also stored in the secondary storage unit. In the present embodiment, the program is stored in the secondary storage unit 1003, read into the primary storage unit 1002 when the program is executed, and the CPU 1001 performs an execution process.

  Examples of the input device 1004 include a mouse and keyboard used for system control, as well as a card reader, scanner, film scanner, and the like necessary for inputting image data. Examples of the output device 1005 include a monitor and a printer. Various other configurations of the apparatus configuration method are conceivable, but the description thereof is omitted because it is not the main point of the present invention.

  The image processing apparatus is equipped with an operating system capable of executing a plurality of programs in parallel, and an operator can operate a program operating on the apparatus using a GUI. The image processing apparatus according to the present embodiment can execute two processes as an image editing function that is a process for removing dust. One is copy stamp processing and the other is repair processing. The copy stamp process is a function of combining a part of an area on an image designated by the user with a cursor or the like into another area designated separately. The repair process is a function of detecting an isolated area that meets a predetermined condition in a specified area and interpolating the isolated area with surrounding pixels. This repair processing includes a manual repair function that is performed on an area designated by a cursor or the like by the user, and an automatic repair function that automatically executes processing using dust correction data. This automatic repair function has been simply referred to as dust removal processing until now, and details of the specific processing will be described later.

  FIG. 15 is a diagram showing a GUI (Graphical User Interface) of an image editing program in the image processing apparatus. The window includes a close button 1100 and a title bar 1101. When the close button is pressed, the program is terminated. When an image to be edited is specified by dragging and dropping the file into the image display area 1102 and the image to be edited is determined, the file name is displayed in the title bar 1101 and then the object image is displayed in the image display area 1102 as Fit. To do.

  There are two display states of the image to be edited, Fit display and pixel-size display, which can be switched with a display mode button 1108. In this GUI, the processing position is specified by clicking on the image. At the time of Fit display, the coordinates on the processing image corresponding to the clicked position are calculated according to the display magnification, and the processing is applied to the coordinates. Shall. In this GUI, the processing range is specified by a radius. This radius is a radius on the image to be edited, and may be different from the radius on the Fit display image depending on the display magnification.

  When an automatic repair process execution button 1103 is pressed, an automatic dust removal process described later is executed, and the processed image is displayed in the image display area 1102. The automatic repair process execution button 1103 is enabled only when the image is not edited, and is disabled when the image is edited by executing the copy stamp process, the repair process, and the automatic repair process.

  A radius slider 1106 is a slider for designating an application range of copy stamp processing and repair processing.

  When the repair processing mode button 1104 is pressed, the repair processing mode is set. When the left click in the image is performed in the repair processing mode, the repair processing described later is applied to a region having the left clicked coordinate as the center and the number of pixels specified by the radius slider 1106 as the radius. After applying the repair process, the repair process mode is exited. Further, when the right click is pressed on the image display area 1102 in the repair mode, or when any button on the GUI is pressed, the repair mode is exited.

  When the copy stamp processing mode button 1105 is pressed, the copy stamp mode is set. When the left click is made in the image in the copy stamp mode, the left clicked coordinate is set as the center coordinate of the copy source area. If the image is further left-clicked with the center coordinates of the copy source area set, the left-clicked coordinates are the center coordinates of the copy destination area, and the radius specified by the radius slider 1106 at this point is the copy radius. The copy stamp process is executed to leave the copy stamp mode with the center coordinates of the copy source area not set. When the right click is pressed on the image display area 1102 in the copy stamp mode, or when any button on the GUI is pressed, the center coordinates of the copy source area are set to the unset state and the copy stamp mode is set. Exit.

  When the save button 1107 is pressed, the processed image is saved.

  In this image editing program, both the original image and the processed image are held as shown in FIG. That is, when a process is executed by the user's GUI operation, the previous and subsequent images are held, and at the same time, the editing process designated by the operation is recorded. Specifically, the editing process specified by the GUI is sequentially applied to the processed image, and the applied editing process is registered as an editing history. One editing process registered in the editing history is called an editing entry.

An example of the edit entry is shown in FIG. The edit entry in this embodiment includes a process ID for distinguishing between the copy stamp process and the repair process, a center and radius indicating the application area of the process, and from a copy source coordinate required for the copy stamp process to a copy destination coordinate. Relative coordinates and differential image data described later are held. When the automatic repair process is executed, the repair process is executed according to the dust correction data, and an edit entry is added to the edit history every time the repair process is applied.
With such an implementation, it is possible to completely discard the editing history and restore the original image, or cancel the previous editing process.

  For example, the process of canceling the immediately preceding editing process can be realized by overwriting the processed image once with the original image and re-executing the editing process until immediately before the editing entry to be canceled. However, it may take time to re-execute the editing process, such as when the number of entries is very large. Therefore, every time the editing operation is executed, the difference between the image data before and after the execution of the editing process is taken and held in the editing entry. If the difference image is retained, when one edit entry is canceled, the edit processing described in the edit entry is not performed again from the beginning until the previous edit entry, but only the difference image is reflected and the previous one is reflected. You can return to the edit entry. Note that an editing history that is a collection of editing entries is stored in the primary storage unit 1002 or the secondary storage unit 1003 of the image processing apparatus.

Next, details of the repair process and the automatic repair process will be described. Since the copy stamp process is a well-known technique, a detailed description of the process is omitted.
The repair process is a process for detecting an isolated area in a specified area and interpolating the isolated area. The repair process is realized by applying an interpolation routine to be described later to a region expressed by the center coordinates and radius specified by the GUI.

  In the automatic repair process, dust correction data is extracted from the Exif information of the captured image data that is image data to be subjected to the dust removal process, and the repair process is automatically executed according to the dust correction data. FIG. 18 shows a basic process flow of the automatic repair process.

  First, captured image data to which dust correction data is added as Exif information is taken into the image processing device from the recording device 419 in the camera or removed from the camera, and stored in the primary storage unit 1002 or the secondary storage unit 1003 ( Step S90).

  Next, dust correction data described as Exif information is extracted from the captured image data (step S91).

  Next, the coordinate sequence Di (i = 1, 2,... N), the radius sequence Ri (i = 1, 2,..., N), the aperture value f1, and the lens pupil position L1 are extracted from the dust correction data extracted in step S91. Obtain (step S92). Here, Ri is the size of the dust at the coordinate Di calculated in step S65 of FIG. In step S93, the aperture value f2 and the lens pupil position L2 at the time of photographing of the photographed image are acquired, and in step S94, Di is converted by the following equation. Here, H is a distance between the surface of the image sensor 418 and dust. The coordinate Di ′ after conversion and the radius Ri ′ after conversion are defined by the following equation, for example.

Di ′ (x, y) = (L2 × (L1−H) / ((L2−H) × L1)) × Di (x, y)
Ri ′ = (Ri × f1 / f2 + 3) × 2
The unit here is a pixel, and “+3” for Ri ′ is a margin amount. The reason why the area is doubled is that an area outside the dust area is necessary in order to detect the dust area using the average luminance.

  In step S95, the interpolation processing counter i is initialized to 0, and i is incremented in step S96.

  In step S97, an interpolation routine is executed on the area represented by the i-th coordinate Di 'and radius Ri' to remove dust in the area. In step S98, it is determined whether dust removal processing has been applied to all coordinates. If processing has been completed for all coordinates, the processing is terminated; otherwise, the processing returns to step S96.

(Interpolation routine)
Here, an interpolation routine executed in the repair process and the automatic repair process will be described.

FIG. 19 is a flowchart showing the flow of the interpolation routine. First, in step S1201, dust region determination is performed. Here, it is assumed that the center coordinate of the region to be repaired is P and the radius is R. The dust area is an area that satisfies all of the following conditions.
(I) An area darker than the threshold value T2 obtained by the following equation using the average brightness Yave and the maximum brightness Ymax of the pixels included in the repair processing target area.

T2 = Yave × 0.6 + Ymax × 0.4
(Ii) A region that does not contact the circle represented by the center coordinate P and the radius R described above.
(Iii) A region in which the radius value calculated by the same method as step S63 in FIG. 10 is not less than l1 pixels and less than l2 pixels with respect to the isolated region constituted by the low-luminance pixels selected in (i).

In the automatic repair process, in addition to the above conditions, an area satisfying the following condition is set as a dust area.
(Iv) A region including the center coordinate P of the circle.

  In this embodiment, l1 is 3 pixels and l2 is 30 pixels. In this way, only an isolated small area can be handled as a dust area.

  If there is such an area in step S1202, the process advances to step S1203 to perform dust area interpolation. If there is no such area, the process ends. The dust region interpolation processing executed in step S1203 is performed by a known defect region interpolation method. Known defect area interpolation methods include, for example, pattern replacement disclosed in JP-A-2001-223894. In Japanese Patent Laid-Open No. 2001-223894, a defective region is specified using infrared light. In this embodiment, the dust region detected in step S1201 is treated as a defective region, and the dust region is treated as a normal pixel by pattern replacement. Interpolate with. For pixels that are not filled by pattern replacement, p normal pixels and q farthest in the order closest to the interpolation target pixel are selected from the image data after pattern interpolation, and interpolation is performed using the average color.

  With the method described above, dust can be removed by an image processing apparatus outside the camera using dust correction data acquired by the camera.

  Next, the dust removal processing when performing lens aberration correction will be described in terms of what processing is added to the case where the above-described lens aberration correction is not performed.

When the procedure of performing dust removal processing after performing lens aberration correction is adopted, the necessary conversion processing for dust correction data is summarized into the following two.
(1) Conversion processing based on a difference in pupil position between a lens (first photographing lens) at the time of dust detection image photographing and a lens (second photographing lens) at normal photographing (when photographing a subject) (FIG. 18 in step S94).
(2) Conversion processing based on coordinate conversion by lens aberration correction (for example, conversion processing performed using the above-described distortion aberration correction lookup table for forward conversion processing).

  Here, the conversion process (1) has been described as the process immediately before the dust removal process in FIG. 18, but when performing the lens aberration correction process, the conversion process may be performed at any time before the dust removal process is executed. However, it must be performed before the conversion process (2).

  Further, as described above, in this embodiment, the dust removal process is performed by the image processing apparatus. However, the lens aberration correction can be performed in the camera body as long as the dust removal process is performed. You may carry out with a processing apparatus. The first conversion means for performing the conversion process (1) and the second conversion means for performing the conversion process (2) are the microcomputer 402 in the case of being performed in the camera body, and in the image processing apparatus. When performing, it becomes CPU1001.

  Therefore, when the camera and the image processing apparatus are viewed as a system, a series of conversion processing and correction processing may be performed according to the following procedure.

  Step A: Acquisition of dust detection images and creation of dust correction data. The dust correction data at this time is based on the lens information when the dust detection image is acquired.

  Step B: Acquisition of a subject image. At this time, information on the lens that has captured the subject image is also acquired.

  Step C: The dust correction data conversion process (1) is performed.

  Step D: A lens aberration correction process is performed on the subject image.

  Step E: The dust correction data conversion process (2) is performed.

  Step F: Dust removal processing (automatic repair processing) is performed based on the converted dust correction data.

  However, step C and step D may be reversed, and step D and step E may be reversed as long as the order of performing (2) is performed after performing (1) above. Also, after the acquisition of the subject image in step B, the dust correction data is described in the Exif information and is rewritten every time conversion processing is performed.

  Next, as for the division of roles between the camera body and the image processing apparatus, there is a problem of how far the steps are performed by the camera body and where the steps are performed by the image processing apparatus. Therefore, three examples are shown below.

Example 1
In the first embodiment, steps up to step E are performed by the camera body, and step F is performed by the image processing apparatus. Such a configuration is advantageous when the processing load of the image processing apparatus is light and many photographed images are desired to be handled at the same time at the stage of appreciation after shooting and printing instructions performed using the image processing apparatus.

  Specifically, in the camera body, the microcomputer 402 performs lens aberration correction processing of the subject image by controlling the image processing circuit 425. Then, the microcomputer 402 performs conversion processing based on coordinate conversion by lens aberration correction on the dust correction data, and describes this dust correction data in the Exif information of the subject image on which the lens aberration correction processing has been executed.

  At this stage, the subject image data subjected to the lens aberration correction process having the dust correction data subjected to the conversion process based on the coordinate conversion by the lens aberration correction as Exif information is recorded in the external recording device 419. .

  Next, the CPU 1001 of the image processing apparatus captures subject image data recorded in the external recording device 419 via the input device 1004. Then, the CPU 1001 executes dust removal processing. In the dust removal process described with reference to FIG. 17, the dust correction data conversion process (1) is performed in steps S92 to S94. However, in the first embodiment, since the dust correction data added to the subject image data received by the image processing apparatus has already been subjected to this conversion process, in the dust removal process in the image processing apparatus, these This step is omitted. That is, there is no need to convert dust correction data in the image processing apparatus, and interpolation processing can be performed on the subject image by directly applying the coordinates and radius of the added dust correction data.

(Example 2)
In Embodiment 2, Steps A, B, and D are performed by the camera body, and Steps C, E, and F are performed by the image processing apparatus with Step C and Step D being reversed.

  With this configuration, preparations for dust removal processing that is not necessarily executed by the image processing apparatus depending on the subject image and dust adhesion state can be minimized on the camera side. Accordingly, the processing load on the camera side is reduced, which is convenient for continuous shooting performance of the camera.

  Specifically, in the camera body, the microcomputer 402 performs lens aberration correction processing of the subject image by controlling the image processing circuit 425. At this time, as the Exif information, dust correction data obtained from the dust detection image is described together with lens information when the dust detection image is acquired. Of course, the lens information of the subject image is also described.

  Next, the CPU 1001 of the image processing apparatus captures subject image data recorded in the external recording device 419 via the input device 1004. Then, the CPU 1001 reads the Exif information added to the subject image data. Based on the Exif information, first, conversion processing based on the difference between the lens at the time of dust detection image shooting and the lens at the time of normal shooting is performed, and then the dust correction data is subjected to conversion processing based on coordinate conversion by lens aberration correction. The conversion process based on the coordinate conversion by the lens aberration correction is performed immediately after step S94 in the dust removal process described with reference to FIG. Based on the dust correction data thus converted, dust removal processing is executed. It should be noted that a distortion aberration correction lookup table required when the dust correction data is subjected to conversion processing based on coordinate conversion by lens aberration correction is stored in the secondary storage unit 1003. This distortion aberration correction lookup table is read into the primary storage unit 1002 and used when converting dust correction data.

(Example 3)
In the third embodiment, Step A and Step B are performed by the camera body, and Steps C to F are performed by the image processing apparatus.

  With this configuration, the processing load on the camera is further reduced as compared with the configuration of the second embodiment, so that it is possible to obtain an effect that the photo opportunity is stronger.

  Specifically, the camera body only acquires dust correction data by shooting dust detection images and performs normal shooting. In the Exif information of the subject image obtained by normal shooting, lens information at the time of acquiring a dust detection image and lens information at the time of normal shooting are described together with dust correction data.

  Next, the CPU 1001 of the image processing apparatus captures subject image data recorded in the external recording device 419 via the input device 1004. The CPU 1001 performs lens aberration correction processing on the subject image. Note that the distortion aberration correction lookup table is stored in the secondary storage unit 1003, and is read into the primary storage unit 1002 and used during lens aberration correction processing. Then, the Exif information added to the subject image data is read. Based on the Exif information, first, conversion processing based on the difference between the lens at the time of dust detection image shooting and the lens at the time of normal shooting is performed, and then the dust correction data is subjected to conversion processing based on coordinate conversion by lens aberration correction. The conversion process based on the coordinate conversion by the lens aberration correction is performed immediately after step S94 in the dust removal process described with reference to FIG. Based on the dust correction data thus converted, dust removal processing is executed.

  The correspondence with the "image processing apparatus" in the claims is that the apparatus including the first conversion means for performing the conversion process of (1) and the second conversion means for performing the conversion process of (2) is described in the claims. It can be said that it is an “image processing apparatus”. That is, in the first embodiment, the interchangeable lens single-lens reflex digital camera is the “image processing apparatus” in claims, and in the second and third embodiments, the image processing apparatus is directly equivalent to the “image processing apparatus” in the claims. To do.

(Other embodiments)
The object of each embodiment is also achieved by the following method. That is, a storage medium (or recording medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded is supplied to the system or apparatus. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but the present invention includes the following cases. That is, based on the instruction of the program code, an operating system (OS) running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

  Furthermore, the following cases are also included in the present invention. That is, the program code read from the storage medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU or the like provided in the function expansion card or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the above storage medium, the storage medium stores program codes corresponding to the procedure described above.

1 is a block diagram illustrating a circuit configuration of a lens interchangeable single-lens reflex digital camera as an imaging apparatus according to an embodiment of the present invention. 1 is an external perspective view of a digital camera according to an embodiment of the present invention. It is a vertical sectional view showing an internal structure of a digital camera according to an embodiment of the present invention. It is a figure which shows the relationship between the image height before distortion aberration correction, and the image height after correction | amendment. It is a flowchart which shows the flow of the distortion aberration correction process which concerns on embodiment of this invention. It is explanatory drawing for demonstrating the image process which correct | amends a distortion aberration by reverse conversion. It is a figure explaining the relationship between the image before distortion aberration correction, and the image after distortion aberration correction. It is a flowchart explaining the dust detection process which concerns on embodiment of this invention. It is a figure which shows the data format example of the dust correction data which concerns on embodiment of this invention. It is a flowchart explaining the detail of the dust area | region acquisition routine performed by step S27 of FIG. It is a figure which shows the processing unit of the dust area | region determination process performed by step S62 of FIG. It is a figure which shows the outline | summary of dust area size calculation performed by step S63 of FIG. It is a flowchart explaining the detail of the imaging process routine performed by step S24 of FIG. It is a figure which shows the internal structure of the image editing program which concerns on embodiment of this invention. It is a figure which shows the example of GUI in the image processing apparatus which concerns on embodiment of this invention. 1 is a diagram illustrating an outline of a system configuration of an image processing apparatus according to an embodiment of the present invention. It is a figure which shows the data structure of the edit history of the image editing program which concerns on embodiment of this invention. It is a flowchart explaining the basic flow of the automatic repair process which concerns on embodiment of this invention. It is a flowchart explaining the detail of the interpolation routine which concerns on embodiment of this invention.

Claims (11)

The position information of the foreign matter in the imaging screen extracted based on the image signal for foreign matter detection obtained from the imaging means, and the lens information of the first photographing lens used when acquiring the image signal are used as the foreign matter. Foreign matter information acquisition means for acquiring information;
Lens information acquisition means for acquiring lens information of the second photographic lens at the time of subject shooting;
Storage means for storing aberration information including aberration information of at least the second photographing lens;
Based on the lens information of the first photographing lens and the lens information of the second photographing lens, the position information of the foreign matter is converted into position information of the foreign matter in the imaging screen when photographed with the second photographing lens. First converting means for converting;
Based on the position information of the foreign matter converted by the first converting means and the lens information and aberration information of the second photographing lens, the subject image photographed at the time of photographing the subject is corrected by the image processing. Second conversion means for converting the position information of the foreign matter in the imaging screen after being
An image processing apparatus comprising: The position and size information of the foreign matter in the imaging screen extracted based on the foreign matter detection image signal obtained from the imaging means, and the lens information of the first photographing lens used when acquiring the image signal Foreign matter information acquisition means for acquiring
Lens information acquisition means for acquiring lens information of the second photographic lens at the time of subject shooting;
Storage means for storing aberration information including aberration information of at least the second photographing lens;
Based on the lens information of the first photographing lens and the lens information of the second photographing lens, the position and size information of the foreign matter is determined based on the second photographing lens. First conversion means for converting into position and size information;
Based on the position and size information of the foreign matter converted by the first conversion means, and based on the lens information and aberration information of the second photographing lens, the subject image photographed at the time of photographing the subject is subjected to image processing. Second conversion means for converting the position and size information of the foreign matter in the imaging screen after aberration correction by
An image processing apparatus comprising:   2. The lens information of the first photographing lens and the lens information of the second photographing lens include at least information on an aperture value and a pupil position at the time of obtaining the image signal or photographing the subject. Or the image processing apparatus of 2.   4. The apparatus according to claim 1, further comprising foreign matter removal processing means for performing foreign matter removal processing based on foreign matter information converted by the second conversion means with respect to the subject image corrected for aberration. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 4, further comprising an aberration correction processing unit that corrects an aberration of a subject image captured at the time of photographing the subject based on aberration information stored in the storage unit. An image processing apparatus according to any one of claims 1 to 3,
An extraction means for acquiring an image signal for foreign object detection in the foreign object detection mode, and extracting position information of the foreign object in the imaging screen from the image signal;
An interchangeable lens single-lens reflex digital camera characterized by comprising:   The interchangeable-lens single-lens camera according to claim 6, further comprising an aberration correction processing unit that corrects an aberration of a subject image captured at the time of photographing the subject based on aberration information stored in the storage unit. Ref digital camera.   The apparatus further comprises foreign matter removal processing means for performing foreign matter removal processing on the subject image corrected for aberration by the image processing means based on foreign matter information converted by the second conversion means. The interchangeable lens single-lens reflex digital camera of 7. The position information of the foreign matter in the imaging screen extracted based on the image signal for foreign matter detection obtained from the imaging means, and the lens information of the first photographing lens used when acquiring the image signal are used as the foreign matter. Foreign matter information acquisition process to be acquired as information,
A lens information acquisition step of acquiring lens information of the second photographic lens at the time of subject shooting;
Based on the lens information of the first photographing lens and the lens information of the second photographing lens, the position information of the foreign matter is converted into position information of the foreign matter in the imaging screen when photographed with the second photographing lens. A first conversion step to convert;
The foreign object position information converted in the first conversion step is further stored in storage means for storing lens information of the second photographing lens and aberration information including at least aberration information of the second photographing lens. A second conversion step of converting, based on the aberration information, a subject image photographed at the time of subject photographing into position information of a foreign object in an imaging screen after aberration correction is performed by image processing;
An image processing apparatus control method comprising: The position and size information of the foreign matter in the imaging screen extracted based on the foreign matter detection image signal obtained from the imaging means, and the lens information of the first photographing lens used when acquiring the image signal A foreign substance information acquisition step of acquiring
A lens information acquisition step of acquiring lens information of the second photographic lens at the time of subject shooting;
Based on the lens information of the first photographing lens and the lens information of the second photographing lens, the position and size information of the foreign matter is determined based on the second photographing lens. A first conversion step for converting to position and size information;
A memory for storing the position information and the size information of the foreign matter converted in the first conversion step, lens information of the second photographing lens, and aberration information including at least aberration information of the second photographing lens. Based on the aberration information stored in the means, a second conversion step of converting the object image taken at the time of shooting the object into position information and size information in the imaging screen after the aberration correction is performed by image processing When,
An image processing apparatus control method comprising:   The program for making a computer perform the control method of Claim 9 or 10.

Images