JP4466015B2 - Image processing apparatus and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus that removes the influence of dust or the like in image data taken with an electronic camera or the like.

  Conventionally, in order to correct the influence of dust mixed in an optical system during the manufacture of a video camera, a technique for photographing a white pattern for each aperture value and recording correction information in advance is disclosed in Japanese Patent Laid-Open No. 9-51459. Is disclosed. Further, as a countermeasure against dust that may change constantly in the field of copying machines, a technique for detecting dust by taking in white reference data having a uniform reflecting surface before reading a document is disclosed in Japanese Patent Laid-Open No. 10-294870 and other special techniques. This is disclosed in Japanese Utility Model Publication No. 11-27475. Furthermore, in the scanner field, as a substitute for this white reference data, there is a method of providing an infrared light sensor, obtaining transmittance data simultaneously with visible light data, and obtaining a transmittance attenuation signal due to film defects. U.S. Pat. No. 6,195,161.

JP-A-9-51459 Japanese Patent Laid-Open No. 10-294870 Japanese Patent Laid-Open No. 11-27475 USP 6,195,161

  However, the conventional camera is only intended for fixed dust adhered to optical components at the time of manufacture, and dust that changes with the frequency of use or the passage of time has not been taken into account. In an interchangeable lens type single-lens reflex camera that has begun to spread today, the reflection of dust that changes with time tends to be a big problem, especially because the optical components at the front of the image sensor are exposed.

  On the other hand, in a copier or a scanner, dust data is acquired before or at the same time as the main scan to deal with dust having a temporal change. However, the structure is different from the camera, and it has a uniform illumination means for the original or film surface at a fixed distance, and it has a completely uniform reflection surface or a new infrared illumination means. By doing so, it is relatively easy to obtain transmittance data. However, it is usually difficult for electronic cameras to obtain such complete uniform surface transmittance data except for inspection during manufacturing.

  Further, the copying machine and the scanner are basically fixed optical systems, and there is no need to consider that dust changes due to changes in the optical system. On the other hand, conventional video cameras do not cope with changes in optical conditions other than the aperture value.

  The present invention provides an image processing apparatus and an image processing program that can appropriately remove the influence of dust and the like from image data captured by an electronic camera or the like.

The invention of claim 1 is applied to an image processing apparatus, and an image acquisition means for acquiring an image photographed by an image sensor, and a value of a pixel of interest and a plurality of pixels within a predetermined range including the pixel of interest in the acquired image. A relative ratio calculating means for calculating a relative ratio with the average value of the pixel values, a defect information generating means for generating defect information in the image based on the calculated relative ratio, and an in-image based on the defect information. The correction means corrects the defect by multiplying the value of the corresponding pixel by the inverse value of the relative ratio.
According to a second aspect of the present invention, in the image processing apparatus according to the first aspect, the image acquisition means acquires a plurality of images photographed by the image sensor, and the defect information creation means uses the acquired plurality of images. Thus, defect information in any one of a plurality of images is created.
According to a third aspect of the present invention, in the image processing apparatus according to the first aspect, the image acquisition means acquires a plurality of images taken by the image sensor, and the defect information creation means uses the acquired plurality of images. Thus, defect information corresponding to the entire image of a plurality of images is created.
The invention of claim 4 is applied to an image processing apparatus, and obtains a reference image taken through an optical system, and within the acquired reference image, a value of a pixel of interest and a predetermined range including the pixel of interest. An image acquisition means comprising: a relative ratio calculation means for calculating a relative ratio of the average value of the plurality of pixels; and a defect information creation means for creating defect information in the reference image based on the calculated relative ratio. Further includes a correction unit that acquires a correction target image captured through the optical system and corrects a defect in the correction target image based on defect information in the reference image, and the correction unit has a relative ratio of the reference image. The reciprocal value is corrected by multiplying the value of the corresponding pixel of the image to be corrected.
According to a fifth aspect of the present invention, in the image processing apparatus according to the fourth aspect, the correction means captures the reference image and the correction target image through an optical system having substantially the same aperture condition and pupil position. In this case, the generated defect information is used as it is, and the values of the pixels constituting the correction target image are corrected.
A sixth aspect of the present invention is the image processing apparatus according to the fourth aspect, further comprising defect information converting means for converting defect information according to at least one of an aperture value and a pupil position, which are optical conditions of the optical system. And the correction means uses the converted defect information to correct the correction target when the reference image and the correction target image are photographed through optical systems having different optical conditions at least one of the aperture value and the pupil position. It is characterized by correcting the values of the pixels constituting the image.
According to a seventh aspect of the present invention, in the image processing apparatus according to the first or fourth aspect, the relative ratio calculating means sets the calculated relative ratio to 1 when the calculated relative ratio is included in a predetermined range with 1 therebetween. It is characterized by doing.
According to an eighth aspect of the present invention, in the image processing apparatus according to the seventh aspect, the relative ratio calculating means associates a predetermined range in which the calculated relative ratio is set to 1 with a standard deviation value of the calculated relative ratio. It is what.
According to a ninth aspect of the present invention, in the image processing apparatus according to the fourth aspect, the image acquisition means acquires a reference image captured within a predetermined time before and after the correction target image is captured. .
According to a tenth aspect of the present invention, in the image processing apparatus according to the ninth aspect, the image acquisition means acquires a reference image captured at a time that is close to a second time within the imaging time of the correction target image. Is.
The invention of claim 11 is applied to an image processing apparatus, and an image acquisition means for acquiring an image photographed using an imaging device capable of splitting into a plurality of colors, and a luminance signal is generated from the signals of the plurality of colors of the image. Relative for calculating a relative ratio between the luminance signal value generated by the pixel of interest and the average value of the luminance signal generated by a plurality of pixels within a predetermined range including the pixel of interest in the acquired image in the acquired image A ratio calculating unit; a defect information generating unit that generates defect information in the image based on the calculated relative ratio; and a correcting unit that corrects the value of each color component of the defective pixel in the image based on the defect information. The correction means is characterized in that the reciprocal value of the relative ratio is multiplied and corrected for each value of each color component of the corresponding pixel.
According to a twelfth aspect of the present invention, in the image processing device according to the eleventh aspect, the image acquisition means acquires a plurality of images taken by the image sensor, and the luminance signal generation means determines the luminance signal for the acquired plurality of images. And the defect information creating means creates defect information in any one of the plurality of images using the luminance signals of the plurality of images generated.
According to a thirteenth aspect of the present invention, in the image processing apparatus according to the eleventh aspect, the image acquisition means acquires a plurality of images taken by the image sensor, and the luminance signal generation means determines the luminance signal for the acquired plurality of images. The defect information creating means creates defect information corresponding to the entire images of the plurality of images using the plurality of acquired images.
The invention of claim 14 is applied to an image processing program, and causes a computer to execute the function of the image processing apparatus according to any one of claims 1 to 13 .

  Since the present invention is configured as described above, the following effects can be obtained. It is possible to appropriately acquire defect information such as the influence of dust contained in an image photographed by an electronic camera or the like by an easy method. In particular, even if the reference image includes factors such as gradation that are not perfectly uniform, it can be appropriately eliminated and converted into defect information. Moreover, the influence of dust etc. can be appropriately removed using this defect information.

-First embodiment-
(Configuration of electronic camera and personal computer)
FIG. 1 is a diagram illustrating a configuration of an interchangeable lens type single-lens reflex electronic still camera (hereinafter referred to as an electronic camera). The electronic camera 1 has a variable optical system 3 including a camera body 2 and a mount type interchangeable lens. The variable optical system 3 has a lens 4 and a diaphragm 5 inside. Although the lens 4 is composed of a plurality of optical lens groups, in the figure, it is represented by a single lens, and the position of the lens 4 is referred to as a main pupil position (hereinafter simply referred to as the pupil position). The variable optical system 3 may be a zoom lens. The pupil position is a value determined by the lens type and the zoom position of the zoom lens. May vary depending on focal length.

  The camera body 2 includes a shutter 6, an optical component 7 such as an optical filter or a cover glass, and an image sensor 8. The variable optical system 3 can be attached to and detached from the mount portion 9 of the camera body 2. Further, the variable optical system 3 transmits optical parameters such as information on the pupil position and information on the aperture value to the control unit 17 (FIG. 2) of the electronic camera 1 via the mount unit 9. The aperture value changes from F2.8 to F22, for example.

Reference numeral 10 denotes dust adhering to the surface of the optical component 7 in front of the image sensor 8. As a result of an experiment for changing the aperture value and pupil position of the variable optical system 3 and evaluating the change of dust shadows reflected in the photographed image, the following two facts were found.
(1) The size of dust shadow and the light transmittance change depending on the aperture value.
(2) The dust position shifts depending on the pupil position of the lens.
From these two experimental facts, it can be seen that even if the dust is attached to the fixed position, the way the dust is reflected changes every time the photographing condition (aperture value and pupil position) of the lens changes. A technique for removing the influence of dust on such a variable optical system will be described below.

  FIG. 2 is a block diagram of the electronic camera 1, a PC (personal computer) 31, and a peripheral device. The PC 31 functions as an image processing apparatus, acquires image data from the electronic camera 1, and performs dust influence removal processing described later.

  The electronic camera 1 includes a variable optical system 3, an optical component 7, a shutter 6 (not shown in FIG. 2), an image sensor 8, an analog signal processing unit 12, an A / D conversion unit 13, a timing control unit 14, and an image processing unit 15. , An operation unit 16, a control unit 17, a memory 18, a compression / decompression unit 19, a display image generation unit 20, a monitor 21, a memory card interface unit 22, and an external interface unit 23.

  The imaging element 8 images a subject through the variable optical system 3 and outputs an image signal (imaging signal) corresponding to the captured subject image. The imaging element 8 has a rectangular imaging region composed of a plurality of pixels, and sequentially outputs image signals, which are analog signals corresponding to the charges accumulated in the pixels, to the analog signal processing unit 12 in units of pixels. Output. The image sensor 8 is composed of, for example, a single-plate color CCD. The analog signal processing unit 12 includes a CDS (correlated double sampling) circuit, an AGC (auto gain control) circuit, and the like, and performs predetermined analog processing on the input image signal. The A / D conversion unit 13 converts the analog signal processed by the analog signal processing unit 12 into a digital signal. The timing control unit 14 is controlled by the control unit 17 and controls the timing of each operation of the image sensor 8, the analog signal processing unit 12, the A / D conversion unit 13, and the image processing unit 15.

  The memory card interface unit 22 interfaces with a memory card (card-like removable memory) 30. The external interface unit 23 interfaces with an external device such as the PC 31 via a predetermined cable or wireless transmission path. The operation unit 16 corresponds to a release button, a mode switching selection button, or the like. The monitor 21 displays various menus and displays a subject image captured by the image sensor 8 and a reproduced image based on image data stored in a memory card. The output of the operation unit 16 is connected to the control unit 17, and the output of the display image generation unit 20 is connected to the monitor 21. The image processing unit 15 is composed of, for example, a one-chip microprocessor dedicated to image processing.

  The A / D conversion unit 13, the image processing unit 15, the control unit 17, the memory 18, the compression / decompression unit 19, the display image generation unit 20, the memory card interface unit 22, and the external interface unit 23 are mutually connected via a bus 24. It is connected to the.

  A monitor 32, a printer 33, and the like are connected to the PC 31, and an application program recorded on the CD-ROM 34 is installed in advance. In addition to a CPU, a memory, and a hard disk (not shown), the PC 31 includes a memory card interface unit (not shown) that interfaces with the memory card 30, an electronic camera 1, etc. via a predetermined cable or wireless transmission path. An external interface unit (not shown) that interfaces with an external device is provided.

  In the electronic camera 1 configured as shown in FIG. 1, when the shooting mode is selected by the operator via the operation unit 16 and the release button is pressed, the control unit 17 causes the image sensor 8 to operate via the timing control unit 14. The timing control is performed on the analog signal processing unit 12 and the A / D conversion unit 13. The imaging element 8 generates an image signal corresponding to the optical image formed in the imaging area by the variable optical system 3. The image signal is subjected to predetermined analog signal processing in the analog signal processing unit 12, and is output to the A / D conversion unit 13 as an image signal after analog processing. The A / D conversion unit 13 digitizes the analog-processed image signal and supplies it to the image processing unit 15 as image data.

  In the electronic camera 1 of the present embodiment, the image sensor 8 has a case where the most representative R (red), G (green), and B (blue) color filters of a single-plate color image sensor are arranged in a Bayer array. For example, it is assumed that the image data supplied to the image processing unit 15 is represented in the RGB color system. Each pixel constituting the image data has color information of any one of RGB color components. Here, one photoelectric conversion element constituting the image sensor 8 is referred to as a pixel, and one unit of image data corresponding to this pixel is also referred to as a pixel. An image is also a concept composed of a plurality of pixels.

  The image processing unit 15 performs image processing such as interpolation, gradation conversion, and contour enhancement on such image data. The image data that has undergone such image processing is subjected to predetermined compression processing by the compression / decompression unit 19 as necessary, and is recorded in the memory card 30 via the memory card interface unit 22. Image data for which image processing has been completed may be recorded on the memory card 30 without being subjected to compression processing.

  The image data for which image processing has been completed is provided to the PC 31 via the memory card 30. It may be provided to the PC 31 via the external interface 23 and a predetermined cable or wireless transmission path. It is assumed that the image data for which image processing has been completed has undergone interpolation processing, and color information of all RGB color components is present in each pixel.

(Dust removal process)
Next, processing for removing the influence of dust in each captured image data will be described. In the first embodiment, it is assumed that a reference image for obtaining dust information for each optical shooting condition is shot with the electronic camera 1 every time. However, the reference image is not a completely uniform white reference data, but a blue sky, a nearly uniform wall surface, a gray chart, a plain paper surface, or the like is photographed and used instead. The reference data in this case may include lens peripheral light reduction, subject gradation, image sensor shading, and the like. It is assumed that the reference data can be acquired in a situation where it can be easily photographed at a location that is actually familiar, and does not require strict uniformity, and is converted into a uniform one by an algorithm on the image processing side.

(Operation on the electronic camera side)
FIG. 3 is a diagram for describing a photographing procedure on the electronic camera 1 side in the first embodiment. 1) Normal photographing 101 is performed at the pupil position P1 aperture value A1, and the correction target image data 1 is output. 2) Subsequently, uniform plane imaging 102 is performed at the same pupil position P1 aperture value A1, and the reference image data 1 is output. 3) Next, normal imaging 103 is performed at a different pupil position P2 and aperture value A2, and correction target image data 2 is output. 4) Subsequently, the uniform plane imaging 104 is performed at the same pupil position P2 aperture value A2 as in the normal imaging 103, and the reference image data 2 is output. That is, first, the electronic camera 1 is photographed toward a subject to be photographed (normal photographing), and immediately after that, the electronic camera 1 is photographed toward the sky or a wall surface (uniform surface photographing). Alternatively, the camera may be in the same state as in normal shooting, and a white paper or uniform color paper may be held over a few cm to 10 cm in front of the lens. In this way, a shooting operation is performed in which normal shooting and uniform plane shooting are paired. Here, outputting image data means recording on the memory card 30 or directly outputting to the PC 31 via the external interface 23.

  Since there is a possibility that the state of dust is changed in the electronic camera, in this embodiment, photographing of a uniform surface is performed with the same optical condition immediately after photographing of the correction target image. However, actually, it does not have to be so strictly in time. If the same shooting conditions can be reproduced optically for the pupil position and aperture value, a lot of dust can be used without much change even with uniform plane data shot after about a day. Therefore, the uniform surface data can be substituted if it is taken within a time difference that reproduces the same optical conditions and sufficiently reflects dust information during normal photography. Note that the order of normal imaging and uniform plane imaging may be interchanged so that uniform plane imaging is performed first and normal imaging is continued.

(Image processing device side operation)
Image data captured by the electronic camera 1 is provided to the PC 31 after predetermined image processing. The PC 31 performs dust removal processing using a pair of correction target image data and reference image data. The PC 31 may be said to be an image processing apparatus that performs dust removal processing. Both the reference image data and the correction target image data are input to the PC 31 in a state where the RGB interpolation processing of the Bayer array is completed. The reference image data and the correction target image data described below are data captured under optical conditions of the same pupil position and aperture value. FIG. 6 is a flowchart showing the flow of processing performed by the PC 31.

<Processing for reference image data>
1) Generation of Luminance Surface In step S11 in FIG. 6, a luminance surface is generated. For each pixel [i, j] of the reference image data, a luminance signal is generated from the RGB signal using the following equation (1). [i, j] indicates the position of the pixel.
Y [i, j] = (R [i, j] + 2 * G [i, j] + B [i, j]) / 4 ... (1)
Although it is possible to individually analyze each of the R, G, and B surfaces, basically, the influence of dust shadows only causes signal attenuation and is not related to color components. Therefore, here, conversion to a luminance component that can reduce the influence of random noise while effectively using all information is performed. In addition, by doing so, it is only necessary to analyze the luminance component single surface from the RGB3 surface, and the speed can be increased. The luminance component generation ratio is not limited to the above, and may be R: G: B = 0.3: 0.6: 0.1 or the like.

2) Transmission map generation (Gain map extraction)
In step S12, a transmission map (gain map extraction) including the following processing is performed.
2-1) Local normalization processing (gain extraction processing)
As described above, the reference image data is not necessarily completely uniform. Therefore, the generated luminance surface is not completely uniform. A local pixel value normalization process is performed on such a luminance plane, and the transmittance signal T [i, j] of each pixel is calculated using the following equation (2). To do. That is, the relative ratio between the value of the pixel of interest [i, j] and the average pixel value in the local range including this pixel is determined for each pixel. As a result, nonuniformities such as gradation and shading included in the uniform surface data are eliminated without problems in terms of algorithm, and only a decrease in transmittance due to dust shadows can be extracted. The transmittance of the entire image obtained in this way is called a transmittance map (gain map). The transmittance map indicates defect information of the reference image. The pixel value is a value of a color signal (color information) or a luminance signal (luminance information) of a color component in each pixel. For example, when represented by 1 byte, it takes a value from 0 to 255.

  Here, the range (2a + 1) × (2b + 1) pixels for which the local average is taken is larger than the dust diameter. Ideally, if the area is about 3 times larger than the dust shadow, accurate transmittance data can be obtained. a represents the number of pixels extending left and right around the pixel of interest [i, j], and b represents the number of pixels extending up and down around the pixel of interest [i, j]. For example, if the pixel pitch of the image sensor 8 is 12 um and the distance between the imaging surface and the dust adhesion surface is 1.5 mm, when the aperture value is F22, the diameter of the huge dust is about 15 pixels and the aperture value is F4. The diameter of huge garbage is about 40 pixels. Therefore, it is preferable to set a = 40, b = 40, and set the local average range as an 81 × 81 pixel range. This is an example, and may be a pixel range based on other numbers of pixels.

  The dust shadow greatly depends on the aperture value, and the small dust disappears as soon as the aperture is opened. However, the large dust may occupy a large area even though the shadow is thin even when the aperture is opened. Depending on the pixel pitch width of the image sensor, there may be a case where a round dust shadow is formed over several tens of pixels on the open side, and in that case, it becomes necessary to take a local average over a very wide range. Therefore, in order to increase the processing speed, there is no problem even if the processing is representatively performed with thinned pixels.

  The process of calculating the relative ratio in the range of (2a + 1) x (2b + 1) pixels is referred to as a local normalization process (gain extraction process). A filter that performs a relativization operation in the range of (2a + 1) x (2b + 1) pixels may be called a gain extraction kernel. FIG. 4 is a diagram illustrating a state in which the local normalization process is performed on the luminance surface. FIG. 4A is a diagram illustrating luminance signals of pixels arranged in a certain horizontal direction in the luminance plane. Reference numerals 41 and 42 indicate that the luminance signal is reduced due to dust. FIG. 4B is a diagram obtained by performing the above-described local normalization processing on the luminance signal of FIG. That is, pixel value normalization processing is performed in a local range. Reference numerals 43 and 44 correspond to the reference numerals 41 and 42 in FIG. 4A, and indicate the transmittance of the places where dust exists. In this way, nonuniformities such as gradation and shading included in the uniform surface data are eliminated, and only a decrease in transmittance due to dust shadows can be extracted. Thereby, the position of the dust and the degree of transmittance can be known at the same time.

2-2) Low-pass processing of transmittance map The low-pass processing of the transmittance map may be selectable, but it is preferable to include this processing because it is effective for the most part. Since the transmittance signal T [i, j] includes random noise associated with the quantum fluctuation of the luminance signal, there is a region where the influence of dust shadows remains slightly at a level where the transmittance is close to 1. Because of the randomness, when the threshold determination in 2-4) following is performed, dust shadows may be extracted from the spots. In order to prevent this, if the dust shadows are grouped by the low-pass filter according to the following equation (3), the appearance will be slightly improved.
T [i, j] = {4 * T [i, j]
+ 2 * (T [i-1, j] + T [i + 1, j] + T [i, j-1] + T [i, j + 1])
+ 1 * (T [i-1, j-1] + T [i-1, j + 1] + T [i + 1, j-1] + T [i + 1, j + 1])} / 16 ... (3)

2-3) Statistical analysis of transmittance map For the entire image of the transmittance map obtained by the above-described local normalization processing, the average value M is obtained by the following equation (4), and the standard deviation σ is obtained by the following equation (5). The statistical analysis obtained by Nx and Ny represent the total number of pixels in the x and y directions.

2-4) Threshold judgment Basically, the area ratio of dust signals in the transmittance map is very small, and the statistical analysis result in 2-3) shows that random noise (quantum fluctuations in the transmittance signal) (Shot noise) is being evaluated. Reference numeral 46 obtained by enlarging the reference numeral 45 in FIG. 4 shows a state in which there is this fine random noise. When a histogram of the transmittance map is taken, a normal distribution with a standard deviation σ centering on an average value M (M is a value close to 1) is obtained. FIG. 5 is a diagram showing a histogram of the transmittance map. Since it is considered that the fluctuation range is not subjected to a change in transmittance due to dust shadows, the transmittance may be forcibly set to 1. That is, threshold determination is performed under the following conditions (6) and (7).
if | T [i, j] -M | ≦ 3σ then T [i, j] = 1 ... (6)
else T [i, j] = T [i, j] ... (7)

  Since the normally distributed random data is 99.7% if the range of ± 3σ is collected, the influence of random noise can be excluded almost accurately. The transmittance deviating from ± 3σ is an abnormal signal that can hardly be explained by statistical errors, and is considered to represent a phenomenon caused by a decrease in transmittance due to dust shadows. This abnormal portion is usually a value smaller than 1 in the case of dust shadows.

  However, there are some which show a value larger than 1 although the ratio is small. This is not the effect of dust shadows, but is a phenomenon seen when defects caused by striae (non-uniform refractive index) of optical low-pass filters cause interference fringes that make incident light stronger or weaker. is there. Thereby, this method can also be used for detecting defects of optical members other than dust contained in the optical path. Further, the influence of pixel defects in the image sensor can also be determined by this method. Although dust is more likely to remain near the image sensor 8 without being blurred, it can be accurately determined even when dust on the photographing lens is reflected with considerable blur.

In order to remove only the influence of dust shadows, the threshold value may be determined according to the following conditions (8), (9), and (10).
if | T [i, j] -M | ≦ 3σ then T [i, j] = 1 ... (8)
else if T [i, j]> 1 T [i, j] = 1 ... (9)
else T [i, j] = T [i, j] ... (10)
Since the average value M used for determination always takes a value close to 1, it may be replaced with 1.

  In this way, two types of defect information, map information (determined based on whether T = 1) or not, and transmittance information representing the degree of defect, can be obtained at the same time. Note that the transmittance map described above may be referred to as a gain map because it represents a local relative gain.

  Usually, a defect such as dust is detected by a differential filter for edge detection. However, when dust in the middle of the optical path is targeted, it appears as a dust shadow with very low contrast with the surroundings due to optical blurring. In such a case, the differential filter is very insensitive and can hardly be detected. However, as described above, when a determination method using the statistical property of transmittance is used, it is possible to detect dust with very high sensitivity, and it is possible to correct the influence of foreign matter in the target optical path.

<Processing for correction target image>
3) Gain correction In step S13, gain correction is performed. The correction target image data is corrected using the transmittance map obtained as described above. As shown in equations (11), (12), and (13), gain correction is performed by multiplying the R, G, and B values of the correction target image data by the reciprocal of the transmittance signal.
R [i, j] = R [ij] / T [i, j] ... (11)
G [i, j] = G [ij] / T [i, j] ... (12)
B [i, j] = B [ij] / T [i, j] ... (13)

  As a result, a decrease in luminance due to dust shadows can be neatly corrected. In addition, it is possible to prevent unnecessary correction since the transmittance map is subjected to threshold determination in places where there is no need for correction. That is, since the influence of random noise is removed from the transmittance T of the dust-free portion, there is no concern that the noise of the RGB signal is amplified.

  As described above, in the first embodiment, an image captured at an arbitrary time can be corrected appropriately even with an ordinary electronic camera that does not include a special mechanism for preventing dust. Since photographing of the uniform surface of the reference image does not require strict uniformity, it can be realized relatively easily. In addition, detection and correction can be performed with much higher sensitivity than conventional dust detection methods.

-Second Embodiment-
In the second embodiment, a reference image for obtaining dust information is photographed only once, and a dust removal method using a plurality of images having different optical photographing conditions is also shown using this reference image. Since the configuration of the electronic camera 1 and the PC 31 as the image processing apparatus is the same as that of the first embodiment, description thereof is omitted.

(Operation on the electronic camera side)
FIG. 7 is a diagram for describing a shooting procedure on the electronic camera 1 side in the second embodiment. 1) Uniform plane photography 201 is performed at the pupil position P0 and the aperture value A0, and the reference image data 0 is output. 2) The normal photographing 202 is performed at the pupil position P1 and the aperture value A1, and the correction target image data 1 is output. 3) The normal photographing 203 is performed at the pupil position P2 and the aperture value A2, and the correction target image data 2 is output. 4) The normal photographing 204 is performed at the pupil position P3 and the aperture value A3, and the correction target image data 3 is output. That is, first, the electronic camera 1 is photographed with a uniform surface facing the sky or a wall surface (uniform surface photographing), and then the electronic camera 1 is photographed as needed toward the subject to be photographed (normal photographing).

  Here, it is assumed that the aperture value A0 of the reference image is taken in a state in which the aperture is most narrowed in a variable range prepared in the variable optical system 3. The most narrowed aperture value is, for example, about F22 for a standard lens. On the other hand, it is assumed that the aperture value of the correction target image is the same as that of the reference image or closer to the open side.

  Uniform surface photography can be omitted as long as the state of dust adhesion does not change. Although the number of insertions of uniform plane photography has never been too many, data that is usually only once a day can be effective garbage data. The determination as to whether or not to perform uniform plane imaging is left to the photographer. However, if the previously performed uniform plane imaging is too far in time, the reference data obtained by the uniform plane imaging may be unreliable. Therefore, only the reference image data for uniform plane imaging within a predetermined time from normal imaging may be used. Further, it is not always necessary to perform uniform plane imaging first. You may use the reference image data of the uniform surface photography performed later. When there are a plurality of uniform plane imaging before and after the normal imaging, the reference image data of the uniform plane imaging that is closest in time may be used. Alternatively, if you are concerned about the possibility of newly attached dust, you may select one from the second closest one before and after photographing.

(Image processing device side operation)
In the second embodiment, it is assumed that data that can identify the pupil position and the aperture value is embedded in the reference image data and the correction target image data input to the PC 31 that is the image processing apparatus. The pupil position data may be calculated by using a conversion table from the recording data of the lens type, zoom position, and focus position embedded in the photographing data. FIG. 12 is a flowchart showing the flow of processing performed by the PC 31.

<Processing for reference image>
1) Luminance surface generation and transmittance map generation The luminance surface generation in step S21 and the transmittance map generation in step S22 are performed in the same manner as in the first embodiment.

2) Pupil position conversion of transmittance map In step S23, the pupil position of the transmittance map is converted. When the pupil positions of the reference image and the corrected image are different from each other, the pupil position of the reference image is converted into a dust position predicted to appear when viewed from the pupil position of the correction target image. FIG. 8 is a diagram illustrating how the dust shadow position changes as the pupil position changes. FIG. 8A is a diagram illustrating the relationship between the pupil position, dust, and the imaging surface of the imaging element 8. FIG. 8B is a diagram illustrating a state in which dust shadows are moving on the imaging surface with changes in the pupil position.

ただし、距離lは光学部品の厚みを空気中の光路長に換算した値である。 As is clear from FIG. 8, when the pupil position is different, the position of dust reflected in the image is shifted in the radial direction from the optical axis 51, that is, the center of the image. Here, the amount Δr of dust deviated in the radial direction from the optical axis 51 in the image is estimated. Assuming that the pupil position of the reference image is P0, the pupil position of the correction target image is P0 ′, and dust is attached at a distance l from the imaging surface, Δr can be calculated by the following equation (14).

However, the distance l is a value obtained by converting the thickness of the optical component into an optical path length in the air.

ずれ量Δrは、光軸５１から距離が遠くになるに従い大きくなる。実際の画像の周辺部では、瞳位置の値によっては数十画素に及ぶこともある。 The transmittance map T [i, j] of the reference image is displaced to [r ′, θ] by the following equation (15) on the polar coordinates [r, θ], and the transmittance map T on the coordinates [i, j] 'Convert to [i, j].

The shift amount Δr increases as the distance from the optical axis 51 increases. In the periphery of an actual image, it may reach several tens of pixels depending on the value of the pupil position.

3) F value conversion of transmittance map In step S24, F value conversion of the transmittance map is performed. When the aperture values of the reference image and the corrected image are different from each other, the dust diameter and the transmittance of the reference image are converted into an F value into the dust diameter and the transmittance of the correction target image at the more open aperture value. FIG. 9 is a diagram illustrating how the size of the dust shadow changes when the F value that is the aperture value changes. FIG. 9A shows a case where the F value is large, and FIG. 9B shows a case where the F value is small. As is apparent from FIG. 9, when the definition formula of the F value (F = focal length / effective aperture of the lens) is applied to the distance l from the imaging surface having a similar relationship to the dust adhesion position and the dust spread Γ, the following formula ( 16) is established.

  When l is divided by the pixel pitch a [mm / pixel] of the image sensor, the dust diameter can be expressed by the number of pixels. In this way, it can be predicted that the dust in the point image spreads to the size of the width Γ when the stop has the F value.

  On the other hand, the distribution function of the point image is considered to spread the dust shadow by applying light to the point image dust evenly from each incident angle of the lens opened within the aperture value. You can assume that it has a function. Therefore, the F value conversion can be performed to accurately predict the dust diameter and the transmittance by applying a uniform low-pass filter process represented by a filter width Γ pixel. As the low-pass filter, a circular non-separable filter having a diameter of Γ is considered to be normal, but there is no problem with a vertical Γ and horizontal Γ square separated filter in consideration of speeding up the processing.

  For example, when l = 0.5 mm and a = 5 μm / pixel, the transmission map of F22 is converted to F16, F11, F8, F5.6, and F4. The original filter coefficient is represented in a format as shown in FIG. Using the one-dimensional filter coefficient of FIG. Note that the one-dimensional filter coefficient of the aperture value F16 has a coefficient of 0.5 at both ends and seven coefficients. This is because the even-width spread is filtered in an odd-width range that spreads evenly in the horizontal and vertical directions around the pixel of interest. FIG. 11 is a diagram showing the aperture value F16 filter as a two-dimensional filter.

  By performing the above conversion process, the transmittance map of the reference image is converted into a transmittance map in the pupil position and F value states of the correction target image. That is, the transmittance map equivalent to the transmittance map generated under the optical condition where the correction target image is captured is generated as the transmittance map of the reference image.

<Processing for correction target image>
3) Gain correction In step S25, gain correction is performed using the converted transmittance map. Similar to the first embodiment, for each of the R, G, and B values of the correction target image data, as shown by the equations (17), (18), and (19), the pupil position and the transmission after the F value conversion are performed. Gain correction is performed by multiplying the reciprocal of the rate signal.
R [i, j] = R [ij] / T '[i, j] ... (17)
G [i, j] = G [ij] / T '[i, j] ... (18)
B [i, j] = B [ij] / T '[i, j] ... (19)

  FIG. 13 is a diagram illustrating how the transmittance of medium-sized dust is converted by F-number conversion. The horizontal axis indicates the pixel position, and the vertical axis indicates the transmittance.

  In this way, in a variable optical system, it is not necessary to shoot a reference image under other optical conditions by only shooting a single reference image with a minimum aperture value. In other words, effective correction can be achieved by converting dust data to and from one reference image. Therefore, the burden on the user of the electronic camera is greatly reduced. Further, as in the first embodiment, it is possible to maintain dust detection performance with extremely high sensitivity without requiring strictness of uniform image shooting.

-Third embodiment-
In the third embodiment, a method for detecting and removing dust in a correction target image in a state where there is no uniform plane reference image will be described. The basic principle is that if a flat portion (region in which the image is partially uniform) in the correction target image is found, the dust transmittance map generation processing (gain map) performed for the reference image of the first embodiment is performed. Extraction) can be applied in the same way. Since the configuration of the electronic camera 1 and the PC 31 as the image processing apparatus is the same as that of the first embodiment, description thereof is omitted.

(Operation on the electronic camera side)
FIG. 14 is a diagram for describing a photographing procedure on the electronic camera 1 side in the third embodiment. 1) Normal photographing 301 is performed at the pupil position P1 and the aperture value A1, and the correction target image data 1 is output. 2) Normal photographing 302 is performed at the pupil position P2 and the aperture value A2, and the correction target image data 2 is output. 3) Normal photographing 303 is performed at the pupil position P3 and the aperture value A3, and the correction target image data 3 is output. 4) The normal photographing 304 is performed at the pupil position P4 and the aperture value A4, and the correction target image data 3 is output. That is, in the third embodiment, the uniform plane imaging that is performed in the first embodiment and the second embodiment is not performed.

(Image processing device side operation)
<Processing for correction target image>
FIG. 15 is a diagram illustrating a flowchart of processing performed by the PC 31 that is the image processing apparatus. In step S31, a luminance plane is generated. In step S32, an edge map is generated by gamma correction of the luminance plane, edge extraction filter processing, and threshold determination. In step S33, a dark portion is added to the edge map. In step S34, the edge map is enlarged. In step S35, conversion to a flat map is performed. In step S36, a self gain extraction process is performed. In step S37, self-gain correction processing is performed. Details of each step will be described below.

1) Generation of luminance plane (step S31)
The RGB signal is converted into the luminance signal Y for each pixel [i, j] of the correction target image data. The conversion method is the same as the method performed on the reference image of the first embodiment.

2) Generation of edge map (step S32)
An edge extraction filter is applied to the luminance surface to separate a flat portion and an edge portion in the correction target image. Reflection of dust contained in the optical path in the image appears as dust shadow with very low contrast, and is often not detected by an edge extraction filter as in the prior art. If this fact is used in reverse, it can be assumed in many places that the edge portion extracted by the edge extraction filter is basically an edge in the image, not dust. In order to further distinguish between edges and dust in the image, first, gradation correction processing is performed on the luminance plane.

2-1) Gamma correction of luminance plane Assume that the above-described luminance plane is generated by inputting a correction target image with a linear gradation. At this time, the input signal is Y (0 ≦ Y ≦ Ymax) and the output signal is Y ′ (0 ≦ Y ′ ≦ Y′max). For example, gradation conversion as shown in the following equation (20) is performed. Note that γ = 0.4, 0.5, 0.6 and the like.

  This conversion is a process of increasing the halftone contrast on the low luminance side and decreasing the contrast on the high luminance side. In other words, dust shadows are not noticeable in dark places and stand out in bright places, so the contrast of dust shadows is reduced by this conversion, and one normal image edge is mainly distributed in a halftone, so the contrast is relatively increased. Accordingly, the degree of separation between the dust shadow and the normal edge contrast is improved. Further, in order to convert shot noise caused by the quantum fluctuation of Y ′ so that it can be handled uniformly over all gradations, it is best to set γ = 0.5 in view of the error propagation law. The above equation (20) is a power function. Further, when γ = 0.5, the power function is a square root function.

  When gamma correction processing for final output is applied to the input image, this processing may be skipped if gamma correction processing close to the above conversion is performed. Further, if the above process is performed after performing inverse gamma correction to return to the linear gradation, a more separating function can be obtained.

2-2) Edge Extraction Filter Processing Next, the edge extraction filter according to FIG. 16 and the following equation (21) is applied to the luminance plane subjected to gamma correction. Let YH [i, j] be the edge extraction component of each pixel.
YH [i, j] = {| Y '[i-1, j] -Y' [i, j] | + | Y '[i + 1, j] -Y' [i, j] |
+ | Y '[i, j-1] -Y' [i, j] | + | Y '[i, j + 1] -Y' [i, j] |
+ | Y '[i-1, j-1] -Y' [i, j] | + | Y '[i + 1, j + 1] -Y' [i, j] |
+ | Y '[i-1, j + 1] -Y' [i, j] | + | Y '[i + 1, j-1] -Y' [i, j] |
+ | Y '[i-2, j-1] -Y' [i, j] | + | Y '[i + 2, j + 1] -Y' [i, j] |
+ | Y '[i-2, j + 1] -Y' [i, j] | + | Y '[i + 2, j-1] -Y' [i, j] |
+ | Y '[i-1, j-2] -Y' [i, j] | + | Y '[i + 1, j + 2] -Y' [i, j] |
+ | Y '[i-1, j + 2] -Y' [i, j] | + | Y '[i + 1, j-2] -Y' [i, j] |
+ | Y '[i-3, j] -Y' [i, j] | + | Y '[i + 3, j] -Y' [i, j] |
+ | Y '[i, j-3] -Y' [i, j] | + | Y '[i, j + 3] -Y' [i, j] |
+ | Y '[i-3, j-3] -Y' [i, j] | + | Y '[i + 3, j + 3] -Y' [i, j] |
+ | Y '[i-3, j + 3] -Y' [i, j] | + | Y '[i + 3, j-3] -Y' [i, j] |} / 24 ... (twenty one)

  Here, the filter is designed to collect absolute value differences having a plurality of correlation distances from all directions without omission so that all the original edge portions of the image are extracted as much as possible.

2-3) Determination of threshold value The edge extraction component YH is determined as a threshold value by the following equations (22) and (23), the edge portion or the flat portion is classified, and the result is output to the edge map EDGE [i, j]. . The threshold value Th1 takes a value of about 1 to 5 for 255 gradations. The dust shadow existing on the edge portion is an area that does not require dust removal because the dust shadow is basically buried in the vibration signal of the edge portion and is not noticeable.
if YH [i, j]> Th1 EDGE [i, j] = 1 (edge part) ... (22)
else EDGE [i, j] = 0 (flat part) ... (23)

  As described above, gradation conversion is performed in 2-1) to change the weight between gradations, and threshold determination is performed with a constant threshold Th1 over all gradations in 2-3). However, almost the same effect can be obtained by performing edge extraction while maintaining the linear gradation, and setting a threshold value according to the luminance level and performing threshold determination.

3) Add dark part to edge map (step S33)
The edge map represents a region where the gain map should not be extracted. In addition to the edge region, a region that is dangerous to extract a gain map is a dark region (dark portion). In the dark area, the S / N is poor, so the reliability is low even if the relative gain is extracted. Further, dust shadows existing on the dark part are hardly noticeable, so there is no need to remove dust. Therefore, a dark region is also added to the edge map by the following equation (24). The threshold value Th2 is set to a value of about 20 or less for 255 linear gradations. This operation can be easily understood by describing this operation as “EDGE '= EDGE + DARK”.
if Y [i, j] ≦ Th2 EDGE [i, j] = 1 ... (24)

4) Edge map enlargement process (step S34)
Similar to the first embodiment, a transmittance map is generated by comparing the relative ratio between the center value and the average value of (2a + 1) x (2b + 1) pixels in the flat portion. Therefore, (2a + 1) x (2b + 1) pixel enlargement processing of the edge portion is performed by the following equation (25) so that the edge portion does not enter this kernel in advance. m = 1,2, ..., a, n = 1,2, ..., b.
if EDGE [i, j] = 1 EDGE [i ± m, j ± n] = 1 ... (25)

5) Conversion to flat map (step S35)
The edge map EDGE [i, j] is converted into a flat map FLAT [i, j] by the following equations (26) and (27). Achieved by bit inversion. The flat region indicated by the flat map represents a region that may be self-extracted in the correction target image by the gain map extraction kernel formed of (2a + 1) x (2b + 1) pixels.
if EDGE [i, j] = 0 FLAT [i, j] = 1 (flat part) ... (26)
else FLAT [i, j] = 0 (edge part) ... (27)

6) Self-gain extraction (step S36)
The processing procedure for generating the transmittance map from the reference image data in the first embodiment is performed only for the area of FLAT [i, j] = 1.

6-1) Local normalization processing (gain extraction processing)
T [i, j] in the region of FLAT [i, j] = 1 is generated from the relative ratio in (2a + 1) x (2b + 1) pixels. All areas where FLAT [i, j] = 0 are set to T [i, j] = 1.
6-2) Statistical analysis of transmittance map
A statistical analysis is performed on T [i, j] in the region of FLAT [i, j] = 1 in the same manner as in the first embodiment to calculate an average value m and a standard deviation σ.
6-3) Threshold judgment
The threshold value of T [i, j] in the region of FLAT [i, j] = 1 is determined in the same manner as in the first embodiment, and the value of m ± 3σ is T [i, j] = Set to 1.

7) Self-gain correction (step S37)
As in the first embodiment, self-gain correction is performed by multiplying the R, G, B values of the correction target image by the reciprocal of the self-extracted transmittance signal T [i, j].

  In this way, even if there is no uniform plane reference image data, dust in the correction target image itself can be self-extracted and corrected. That is, a region that satisfies a predetermined condition that can guarantee flatness as described above is extracted from one captured image. The extracted same area is set as a reference image and a correction target image. In the third embodiment, there is no need to take into account the influence of the variable optical system. In particular, it is effective in removing small dust shadows that occupy a huge number.

-Fourth embodiment-
In the fourth embodiment, as in the second embodiment, only one reference image is taken and information on the dust position is used, but the transmittance map is not from the reference image, but the third embodiment. In this manner, a method of self-extraction from the correction target image itself is employed. In the second embodiment, the pupil position conversion of the transmittance map is performed. However, when the value of the pupil position is an approximate value and not accurate, an error may occur in the pupil position conversion. On the other hand, in the third embodiment, large dust is extracted as an edge in the edge map extraction, and correction may not be performed. The fourth embodiment addresses such problems of the second embodiment and the third embodiment. In other words, while using the method of generating a highly reliable transmittance map according to the third embodiment, it is supplemented with highly reliable dust position information acquired by the same method as in the second embodiment. . Note that the configurations of the electronic camera 1 and the PC 31 as the image processing apparatus are the same as those in the first embodiment, and a description thereof will be omitted.

(Operation on the electronic camera side)
The shooting procedure is the same as in the second embodiment.

(Image processing device side operation)
FIG. 17 is a diagram illustrating a flowchart of processing performed by the PC 31 that is the image processing apparatus.

<Processing for reference image>
1) The generation of the luminance surface in step S41 is the same as in the first and second embodiments. 2) The generation of the transmittance map (gain map extraction) in step S42 is the same as that in the first and second embodiments. 3) The pupil position conversion of the transmittance map in step S43 is the same as in the second embodiment. 4) The F value conversion of the transmittance map in step S44 is the same as that in the second embodiment.

5) Determination of threshold value of transmittance map In step S45, the threshold value of the transmittance map is determined. When the F value conversion of the transmittance map is performed, a number of pixels close to the transmittance 1 in which dust shadows are almost disappearing are generated by the low-pass filter process. In order to distinguish these from clear dust shadows, threshold determination is performed again using the following equations (28) and (29). Here, the standard deviation value σ calculated in the process 2) “transmission map generation” is used again. The pupil position and the transmittance map after F-number conversion are represented by T ′ [i, j].
if | T '[i, j] -1 | ≦ 3σ then T' [i, j] = 1 ... (28)
else T '[i, j] = T' [i, j] ... (29)

6) Conversion to dust map In step S46, the transmittance map is binarized by the following equations (30) and (31) and converted to a dust map dmap [i, j].
if T '[i, j] <1 dmap [i, j] = 1 ... (30)
else dmap [i, j] = 0 ... (31)
Here, the determination of Expression (30) may be made such that T ′ [i, j] <0.95 with a little more margin.

7) Dust Map Enlargement Process In step S47, the dust map is enlarged by the error assumed in the pupil position conversion by the following equation (32), so that the area within the allowable error includes dust. Make a garbage map. Here, for example, an error of ± 3 pixels is expected. m = 1,2,3 and n = 1,2,3.
if dmap [i, j] = 1 dmap [i ± m, j ± n] = 1 ... (32)

<Processing for correction target image>
1) The generation of the luminance surface in step S51 is the same as that in the third embodiment. 2) The generation of the edge map in step S52 is the same as in the third embodiment. 3) The process of adding a dark part to the edge map in step S53 is the same as that in the third embodiment. This operation can be easily understood by describing this operation as “EDGE '= EDGE + DARK”.

4) Excluding the dust region from the edge map In step S54, the dust region is excluded from the edge map. Most of the dust shadows are not extracted because the contrast is low. However, some of the dust shadows are large dust and have high contrast, and the edges may be extracted. In particular, it occurs in some dust shadows in the correction target image photographed with narrowing down. In order to prevent these dust shadows from deviating from the designation of the gain extraction region as an edge region, the dust position is forcibly excluded from the edge portion by the following equation (33) using the dust map information obtained in step S46. Process. Here, in order to prevent the edge region from being cut off too much, the dust map before the dust map enlargement process in step S47 is used. This operation can be easily understood by describing this operation as “EDGE '' = EDGE + DARK-DUST”.
if dmap [i, j] = 1 EDGE [i, j] = 0 ... (33)

4 ′) Edge map peripheral assimilation process (correction of dust cut-out portion) (S60)
Since it is unnatural that the edge map is hollowed out (excluded) only in the dust part, peripheral assimilation processing in the edge map is performed. For example, if the background is a uniform image such as a blue sky, no problem occurs even if the dust map information in step S46 is used to cut out an edge portion of large dust from the edge map. Rather, it needs to be hollowed out. However, if the background is an image having a pattern or structure, and the portion is removed from the edge map by assuming that there is dust from the dust map information in step S46, an unnatural correction is made due to the relationship with the actual pattern or structure around. Processing will be performed. Therefore, when it is determined that a pixel determined not to be an edge portion has a large number of edge pixels in its peripheral pixels, the pixel is set as an edge portion again.

Edge map peripheral assimilation processing is performed by the following processing. Specifically, for example, the surrounding eight pixels for the pixel of interest as shown in FIG. 19 (black circle pixels in FIG. 19, FIG. 19 shows only the fourth quadrant for the pixel of interest [i, j] = [0,0]. In the case where more than four pixels are edges, the pixel of interest is also an edge. Exceeding 4 means that the majority of pixels in the peripheral 8 pixels are edges. That is, when a majority of edge pixels exist in the peripheral pixels, the target pixel is also an edge. In the following processing example, the eighth pixel in the horizontal and vertical directions is seen, but it is not necessarily limited to the eighth pixel. What is necessary is just to look at a pixel several to ten or more pixels ahead. Further, the following processing may be performed for all the pixels, or may be limited to only the pixels that are edge-extracted with dmap = 1.
Data copy
tmp [i, j] = EDGE [i, j] Peripheral assimilation for all pixels [i, j]
if tmp [i, j] = 0 {
sum = tmp [i-8, j] + tmp [i + 8, j] + tmp [i, j-8] + tmp [i, j + 8]
+ tmp [i-8, j-8] + tmp [i + 8, j + 8] + tmp [i-8, j + 8] + tmp [i + 8, j-8]
if sum> 4 EDGE [i, j] = 1
}

  5) The edge map enlargement process in step S55 is the same as in the third embodiment. 6) The conversion to the flat map in step S56 is the same as in the third embodiment.

7) Identification of self gain extraction region In step S57, the self gain extraction region is identified. It is most reasonable from the viewpoint of preventing erroneous correction of the correction target image to perform dust removal only on a flat part and an area specified as a dust area. Therefore, area information that satisfies both conditions is obtained by the following equation (34) and substituted into the flat map. That is, FLAT = 1 only when 1 is set in both FLAT and dmap, and FLAT = 0 otherwise.
FLAT [i, j] = FLAT [i, j] * dmap [i, j] ... (34)

8) Self-Gain Extraction Unlike the third embodiment, the self-gain extraction in step S58 performs only the local normalization process (gain extraction process). Subsequent statistical analysis of the transmittance map and dust region limitation processing by threshold processing are unnecessary because they are already limited to the vicinity of dust by the processing of 7). The local normalization process (gain extraction process) is the same as that in the third embodiment. As described above, by performing the dust search by expanding the gain extraction area around the dust by the error of the pupil position conversion accuracy by the process of step S47, dust can be extracted without omission.

  Here, low-pass processing is performed on the self-extracted transmittance map. A low pass process similar to that of the first embodiment is performed on all self-extracted regions of T [i, j] to remove the fluctuation component of the pixel [i, j] included in T [i, j]. . In the present embodiment, the low-pass process is an important process because the self-gain correction is performed by extracting the self-gain without limiting the threshold value process by statistical analysis only to the local region of the dust position. That is, the pixel value and the transmittance value T [i, j] fluctuate in the same direction before the low-pass processing, so if self-gain correction described later is performed without performing the low-pass processing, the image is entirely covered only in the local region. Cheap. Therefore, if the fluctuation component of T [i, j] is removed, the fluctuation component of the pixel value is not lost, and the continuity of graininess with the peripheral portion can be maintained. This is particularly effective when there is a lot of randomness such as a high-sensitivity image. Here, the low-pass filter may be designed to be a little stronger than the first embodiment, and a large dust portion (T [i, j] value that is easily affected by the low-pass filter is considerably smaller than 1). However, the low-pass filter processing may be removed only for (1).

9) Self-Gain Correction The self-gain correction in step S59 is the same as in the third embodiment. Even if the pupil position conversion accuracy of the transmittance map of the reference image is poor, the dust transmittance information is extracted from the correction target image itself, so that clean correction without any deviation can be performed. The self-gain extraction is performed only in the self-gain extraction area specified in step S57. Therefore, the correction process is performed only within this range, and the processing load is reduced.

  As described above, in the fourth embodiment, by effectively using the dust map information of the reference image, it is possible to self-extract all dust from large dust to small dust in the correction target image without omission. Become. Further, when the pupil position conversion accuracy of the transmittance map of the reference image is poor, it can be used as an alternative means of the second embodiment. Further, as in the second embodiment, the load on the photographer of the reference image photographing can be very small.

  In the image processing apparatuses according to the first to fourth embodiments described above, black stains due to the influence of dust or the like generated in an image taken with an electronic camera at an arbitrary use time and an arbitrary use condition, etc. It is possible to appropriately correct the defect and reproduce a high-quality image.

  In the first, second, and fourth embodiments, when creating the transmittance map, a reference image that the photographer thinks is nearly uniform is photographed, and the photographed reference image is localized. A transmittance map was created by applying standard processing. However, there may be a small pattern or the like in the reference image that the photographer thinks is almost uniform. This can be dealt with by basically shooting the subject with a blur. For example, the paper may be photographed at a position closer than the shortest photographing distance of the lens. Even if there is a small pattern, it is almost uniform enough to achieve the purpose if it blurs to an image that changes slowly over a wide range than the gain extraction kernel of (2a + 1) x (2b + 1) size Can be a reference image.

  In the fourth embodiment, the self-gain extraction area is specified in step S57, and correction is performed in a limited range in step S58. The method of limiting the correction range to the peripheral range (neighboring region) including the dust region can be applied in the first to third embodiments. In the first to third embodiments, dust may be identified from the obtained transmittance map and its peripheral area may be obtained.

  In the third embodiment, the process of acquiring one photographed image, extracting a flat portion from one photographed image, and generating a dust map has been described. However, when there is large dust on the flat part, the part may not be extracted as a flat part. In the fourth embodiment, an example of dealing with this by acquiring a reference image has been shown. However, even if a reference image is not acquired, such a large portion of dust can also be obtained by using the correlation of a plurality of images. It can be recognized as a flat part for which defect information is to be created. For example, in a plurality of captured images of different subjects, if there is an image that is always detected by edge extraction at the same position, it may be an image due to dust. Accordingly, when an AND is performed on a plurality of captured images for the edge map processed in the third embodiment, the ANDed portion is excluded from the edge map. In this way, the ANDed portion can be added to the flat portion, and a transmittance map can be created for large dust. Note that the AND is not necessarily limited to the edge map. Any data may be used as long as it is data generated from a photographed image and can recognize dust on the photographing optical path by taking an AND between a plurality of photographed images. For example, if the transmittance map by the gain extraction kernel is forcibly calculated for the entire image regardless of whether it is a flat part, and there is an image extracted with the same transmittance at the same position between multiple images Also, it is conceivable to take an AND that leaves the defect information as a transmittance map and excludes it from the defect information.

  Further, if the transmittance map in the flat portion obtained in the third embodiment is ORed for a plurality of captured images, a transmittance map of the entire captured screen can be obtained. The flat part obtained in the third embodiment differs in the position of the flat part on the photographing screen every time the subject is different. If the OR of these flat portions is taken, the entire photographing screen may be obtained. Therefore, it is possible to obtain a transmittance map of the entire photographing screen from a plurality of photographed images, that is, a plurality of correction target images, without photographing a reference image for obtaining dust information. The transmittance map of the entire photographing screen can be used in common for a plurality of correction target images.

  As described above, when ANDing an edge map or ORing a transmittance map for a plurality of captured images, the pupil position and F value (aperture value) may differ for each captured image. In this case, as described in the second embodiment, the pupil position conversion and the F value conversion may be used in the state of the image signal or in the state of the transmittance map.

  In the above embodiment, an example of the RGB color system of the Bayer array has been described, but it goes without saying that it does not depend on the color filter arrangement method at all as long as interpolation processing is finally performed. The same applies to other color systems (for example, complementary color systems).

  In the above embodiment, an example of an interchangeable lens type single-lens reflex electronic still camera has been described. However, the present invention is not necessarily limited to this content. The present invention can also be applied to a non-interchangeable lens type camera. What is necessary is just to acquire a pupil position and an aperture value by a well-known method suitably.

  In the above-described embodiment, an example in which image data captured by the electronic still camera 1 is processed has been described. However, the present invention is not necessarily limited to this content. The present invention can also be applied to image data taken by a video camera that handles moving images. It can also be applied to image data taken with a camera-equipped mobile phone. Furthermore, it can be applied to a copying machine, a scanner, and the like. That is, the present invention can be applied to any image data captured using an image sensor.

  In the above-described embodiment, an example in which image data captured by the electronic camera 1 is processed by the PC (personal computer) 31 to remove the influence of dust has been described. However, the present invention is not necessarily limited to this content. Such a program may be provided on the electronic camera 1. Further, such a program may be provided in a printer, a projection device, or the like. That is, the present invention can be applied to any apparatus that handles image data.

  The program executed on the PC 31 can be provided through a recording medium such as a CD-ROM or a data signal such as the Internet. FIG. 18 is a diagram showing this state. The PC 31 receives a program via the CD-ROM 34. Further, the PC 31 has a connection function with the communication line 401. A computer 402 is a server computer that provides the program, and stores the program in a recording medium such as a hard disk 403. The communication line 401 is a communication line such as the Internet or personal computer communication, or a dedicated communication line. The computer 402 reads the program using the hard disk 403 and transmits the program to the PC 31 via the communication line 401. That is, the program is transmitted as a data signal on a carrier wave via the communication line 401. Thus, the program can be supplied as a computer-readable computer program product in various forms such as a recording medium and a carrier wave.

  Although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other embodiments conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention.

It is a figure which shows the structure of the electronic camera of an interchangeable lens system. 1 is a block diagram of an electronic camera, a personal computer (PC), and a peripheral device. It is a figure explaining the imaging | photography procedure on the electronic camera side in 1st Embodiment. It is a figure which shows a mode that the local normalization process was performed with respect to the luminance surface. It is a figure which shows the histogram of a transmittance | permeability map. It is a flowchart which shows the flow of the process performed by PC in 1st Embodiment. It is a figure explaining the imaging | photography procedure on the electronic camera side in 2nd Embodiment. It is a figure which shows a mode that the position of a dust shadow changes when a pupil position changes. It is a figure which shows a mode that the magnitude | size of a dust shadow changes when F value which is an aperture value changes. It is a figure which shows the one-dimensional filter coefficient with respect to each aperture value. It is the figure which represented the filter converted into the transmittance | permeability map in aperture value F16 with the two-dimensional filter. It is a flowchart which shows the flow of the process performed by PC in 2nd Embodiment. It is a figure which shows a mode that the transmittance | permeability is converted by F value conversion about medium grade garbage. It is a figure explaining the imaging | photography procedure on the electronic camera side in 3rd Embodiment. It is a figure which shows the flowchart of the process performed by PC in 3rd Embodiment. It is a figure which shows an edge extraction filter. It is a figure which shows the flowchart of the process performed by PC in 4th Embodiment. It is a figure which shows a mode that a program is provided through data signals, such as recording media, such as CD-ROM, and the internet. It is a figure which shows the mode of the edge assimilation process of an edge map.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Electronic camera 2 Camera main body 3 Variable optical system 4 Lens 5 Aperture 6 Shutter 7 Optical component 8 Imaging element 9 Mount part 12 Analog signal processing part 13 A / D conversion part 14 Timing control part 15 Image processing part 16 Operation part 17 Control part DESCRIPTION OF SYMBOLS 18 Memory 19 Compression / decompression part 20 Display image generation part 21, 32 Monitor 22 Memory card interface part 23 External interface part 24 Bus 30 Memory card 31 PC (personal computer)
33 Printer 34 CD-ROM

Claims (14)

Image acquisition means for acquiring an image captured by the image sensor;
A relative ratio calculating means for calculating a relative ratio between the value of the pixel of interest and the average value of a plurality of pixels within a predetermined range including the pixel of interest in the acquired image;
Defect information creating means for creating defect information in the image based on the calculated relative ratio;
Correction means for correcting defects in the image based on the defect information,
The image processing apparatus according to claim 1, wherein the correction unit corrects an inverse value of the relative ratio by multiplying a value of a corresponding pixel. The image processing apparatus according to claim 1.
The image acquisition means acquires a plurality of images taken by the image sensor,
The defect information creating means creates defect information in any one of the plurality of images using the plurality of acquired images. The image processing apparatus according to claim 1.
The image acquisition means acquires a plurality of images taken by the image sensor,
The defect information creating means creates defect information corresponding to the entire images of the plurality of images using the plurality of acquired images. Image acquisition means for acquiring a reference image photographed through the optical system;
A relative ratio calculating means for calculating a relative ratio between the value of the target pixel and the average value of a plurality of pixels within a predetermined range including the target pixel in the acquired reference image;
A defect information creating means for creating defect information in the reference image based on the calculated relative ratio;
The image acquisition means acquires a correction target image photographed through the optical system,
Based on the defect information in the reference image, further comprising correction means for correcting defects in the correction target image,
The image processing apparatus, wherein the correction unit corrects an inverse value of a relative ratio of the reference image by multiplying a value of a corresponding pixel of the correction target image. The image processing apparatus according to claim 4.
The correction means uses the generated defect information as it is when the reference image and the correction target image are photographed through an optical system having substantially the same optical conditions in terms of aperture value and pupil position. An image processing apparatus for correcting a value of a pixel constituting the correction target image. The image processing apparatus according to claim 4.
A defect information converting means for converting the defect information according to at least one of an aperture value and a pupil position, which are optical conditions of the optical system,
The correction means uses the converted defect information when the reference image and the correction target image are photographed through an optical system having different optical conditions at least one of an aperture value and a pupil position. An image processing apparatus for correcting a value of a pixel constituting the correction target image. The image processing apparatus according to claim 1 or 4,
The image processing apparatus according to claim 1, wherein the relative ratio calculation unit sets the calculated relative ratio to 1 when the calculated relative ratio is included in a predetermined range including 1. The image processing apparatus according to claim 7.
The image processing apparatus according to claim 1, wherein the relative ratio calculation unit associates the predetermined range in which the calculated relative ratio is set to 1 with a standard deviation value of the calculated relative ratio. The image processing apparatus according to claim 4.
The image processing apparatus is characterized in that the image acquisition means acquires a reference image captured within a predetermined time before and after the correction target image is captured. The image processing apparatus according to claim 9.
The image processing device is characterized in that the image acquisition means acquires a reference image captured at a time that is close to a second time within the imaging time of the correction target image. Image acquisition means for acquiring an image photographed using an imaging device capable of splitting into a plurality of colors;
Luminance signal generating means for generating a luminance signal from signals of a plurality of colors of the image;
Relative ratio calculation means for calculating a relative ratio between the value of the generated luminance signal of the target pixel and the average value of the generated luminance signal of a plurality of pixels within a predetermined range including the target pixel in the acquired image. When,
Defect information creating means for creating defect information in the image based on the calculated relative ratio;
Correction means for correcting the value of each color component of the defective pixel in the image based on the defect information;
The image processing apparatus according to claim 1, wherein the correction unit corrects the reciprocal value of the relative ratio by multiplying the value of each color component of the corresponding pixel. The image processing apparatus according to claim 11.
The image acquisition means acquires a plurality of images taken by the image sensor,
The luminance signal generation means generates the luminance signal for the acquired plurality of images,
The defect information creating means creates defect information in any one of the plurality of images by using luminance signals of the plurality of generated images. The image processing apparatus according to claim 11.
The image acquisition means acquires a plurality of images taken by the image sensor,
The luminance signal generation means generates the luminance signal for the acquired plurality of images,
The defect information creating means creates defect information corresponding to the entire images of the plurality of images using the plurality of acquired images. The image processing program for executing the functions of the image processing apparatus according to a computer in any one of claims 1 to 13.

Images