JP2005301524A - Image processing program, method and apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing program or the like that allows fully restoring an image picked up with an imaging device by specifying the dust part of the image. <P>SOLUTION: The image processing program includes an image selection step (S2) for selecting, from images taken in the same photographic mode by the same imaging device as that which took an image to be processed, one or more images as reference images in the order from the image picked up on the closest date to the date on which the image to be processed is picked up; a dust specifying step (S3) for adding the brightness of the image to be processed and that of the reference image together for each matching pair of pixels and specifying a group of pixels whose added brightness is not more than a predetermined value as the dust part of the image; a restoration step (S4) for setting a plurality of areas in such a way as to include the dust part of the image specified in the image to be processed and for restoring the dust part of the image using pixel values in an area where the distribution of pixel values for the part of the image other than the dust part is smallest; and a confirmation step (S5) for confirming processing results in reference to the unprocessed image. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing program, an image processing method, and an image processing device for removing an image portion of dust contained in an image.

  An imaging apparatus such as a digital camera generally includes an imaging optical system and an imaging element (image sensor) that photoelectrically converts a subject image formed by the imaging optical system. In such an imaging apparatus, dust or dirt (hereinafter collectively referred to as “dust”) may adhere to the lens located closest to the subject in the lens group constituting the imaging optical system. is there.

  In particular, in a so-called single-lens reflex camera type imaging device in which a lens can be exchanged, dust may adhere to the lens located closest to the imaging element or the imaging surface of the imaging element.

  Such dust impairs the quality of the captured image, but the dust on the lens is blurred on the imaging surface, whereas the dust adhering to the imaging surface of the image sensor is a clear black spot, etc. As it appears in the image, it is easily noticeable. Accordingly, the photographer tries to remove such dust with a blower as much as possible in advance, but it is difficult to remove it completely by hand.

  For such dust that has adhered to the imaging surface, technology for physically removing it using, for example, ultrasonic vibration has been put into practical use, but avoiding an increase in the cost of the camera. I can't.

  On the other hand, a technique has been proposed in which dust is not physically removed but an image of the dust portion is corrected by image processing from an image captured with dust attached.

  For example, Japanese Patent Application Laid-Open No. 2002-209147 discloses a digital camera including an imaging lens and an imaging unit, and the position of dust on the image based on image data output from the imaging unit in the dust detection mode. Dust position detection means for detecting the dust position, storage means for storing the dust position detected by the dust position detection means, and output from the imaging means in the imaging mode based on the dust position stored in the storage means. A digital camera having correction means for correcting image data is described. Specifically, the correction means described in the publication performs color correction so that the image at the position determined as dust is the same as the surrounding color. Furthermore, this publication describes that the position of dust is detected from reference subject image data obtained by imaging a reference subject of a single white color.

  On the other hand, although not for correcting the influence of dust, as a similar technique, a technique for correcting a defective pixel in which a dot-like (or line-like) data missing portion is generated on an image has been proposed. .

For example, Japanese Patent Laid-Open No. 2002-165092 discloses a defective pixel detection unit that detects a defective pixel that needs to be corrected by scanning pixels that form a digital image, and a correction that is necessary from around the detected defective pixel. Normal pixel detecting means for detecting non-normal pixels, luminance level difference detecting means for detecting a luminance level difference between the detected normal pixels and other normal pixels in a direction symmetrical to the defective pixel, and the detected An image processing apparatus including correction means for correcting the defective pixel by internal interpolation of two normal pixels having the smallest luminance level difference is described. Furthermore, this publication describes that when a defective pixel exists in an edge portion, a normal pixel in a direction substantially parallel to the edge direction is used as a normal pixel for interpolation.
JP 2002-209147 A JP 2002-165092 A

  However, since the technique described in Japanese Patent Application Laid-Open No. 2002-209147 is configured to detect the position of the dust by setting the digital camera to the dust detection mode when there is a possibility that dust is attached, An image shot in the shooting mode without detecting the dust position in the dust detection mode cannot be corrected. Therefore, after the image is taken, for example, when the image is observed on a personal computer or the like, even if black spots due to dust are observed, the data detected in the dust detection mode is If there is no reference subject image data), correction by this technique cannot be performed. Since the dust position image is also simply interpolated using surrounding pixel data, the image cannot be corrected satisfactorily when dust is present at the edge portion, for example.

  In addition, what is described in the above-mentioned Japanese Patent Application Laid-Open No. 2002-165092 performs interpolation using two normal pixels located between the defective pixels in order to correct the defective pixels. Defect pixels often occur in units of one pixel or in units of several pixels along the line direction, and in order to correct such a defective pixel, data of two normal pixels sandwiching the missing pixel is used. This is considered relatively effective. However, the loss of image data due to dust may result in, for example, a planar area extending over a plurality of pixels. In such a case, interpolation using only two normal pixels does not necessarily result in an optimal interpolation result. Not necessarily. Furthermore, this publication describes a technique for detecting flaws and dust in a negative film after being photographed and developed by irradiating infrared light. However, in this technique, dust is the cause. Black spots and the like formed on the film cannot be detected, and therefore the black spots and the like cannot be removed from the image.

  The present invention has been made in view of the above circumstances, and provides an image processing program, an image processing method, and an image processing apparatus capable of identifying and satisfactorily restoring dust image portions from an image taken by an imaging apparatus. It is intended to provide.

  In order to achieve the above object, an image processing program according to a first invention is an image for causing a computer to remove an image portion of dust that can be included in the image from an image captured by an imaging device. A processing program that selects one or more images as reference images in order of shooting date and time from the one or more images captured by the same imaging device as the imaging device that captured the processing target image. An image selecting step; a dust specifying step for specifying an image portion of dust existing at substantially the same position in both the processing target image and the reference image; and a dust image in the processing target image specified by the dust specifying step Causing the computer to execute a repairing step of repairing the portion using image information in the vicinity of the dust image portion in the processing target image. It is because of the program.

  The “date and time” includes a wide range of time concepts such as year, month, day, hour, minute, and second.

  “Dust” includes a wide range of dust, dust, dust, and the like, and is simply described as “dust” as a representative.

  An image processing program according to a second invention is the image processing program according to the first invention, wherein the imaging device includes an image sensor, and the dust is captured by the image sensor. It is characterized by being attached to the surface.

  Furthermore, an image processing program according to a third invention is the image processing program according to the first or second invention, wherein the imaging device includes an image sensor, and the substantially same position is The image sensor is characterized by being at substantially the same position on the imaging surface.

  An image processing program according to a fourth invention is the image processing program according to the first invention, wherein the image selection step selects the reference image from images having the same pixel configuration as the processing target image. It is characterized by being.

  An image processing program according to a fifth invention is the image processing program according to the first invention, wherein the dust specifying step calculates a lightness for each pixel of the processing target image and a lightness for each pixel of the reference image. Adding by the brightness addition step, the brightness addition step for adding the brightness for each pixel of the processing target image calculated by the brightness calculation step and the brightness for each pixel of the reference image for each pixel position, and the brightness addition step A pixel extraction step for extracting pixels whose brightness addition value is smaller than a predetermined value, and grouping the pixels in the processing target image extracted by the pixel extraction step so that adjacent pixels are included in the same group A group of pixels grouped by the grouping step in a computer. It characterized thereby identified as dust pixel groups constituting the image portion of the.

  The image processing program according to a sixth aspect is the image processing program according to the fifth aspect, wherein the dust specifying step averages the lightness of each pixel calculated in the lightness calculation step for all the pixels included in the image. The average brightness calculation step for performing the process of calculating the average image brightness for all the reference images selected in the image selection step, and a reference image in which the average image brightness calculated in the average brightness calculation step is a predetermined value or less. An image excluding step of excluding from the addition target in the brightness adding step.

  An image processing program according to a seventh invention is the image processing program according to the first invention, wherein the dust specifying step calculates a lightness for each pixel of the processing target image and a lightness for each pixel of the reference image. A calculation step, a pixel extraction step for extracting pixels whose brightness calculated by the brightness calculation step is smaller than a predetermined value, and pixels adjacent to the pixels extracted by the pixel extraction step are included in the same group. A grouping step for grouping, and a pixel group in the processing target image and a pixel group in the reference image among the pixel groups grouped by the grouping step are the processing target image and all the reference images A combination generation step for generating a combination of overlapping pixel groups, and the combination generation step A brightness adjustment step for adjusting the brightness for each image so that the number of pixels included in the pixel group is the same for all of the pixel groups belonging to the combination for each of the combinations of pixel groups generated by the step, and the brightness adjustment step The ratio calculation step for calculating the ratio of the number of pixels in the part where all of the pixel groups belonging to the combination overlap for the pixel group having the same number of pixels to the number of pixels in one pixel group, and the ratio calculation step. A determination step of determining that a pixel group belonging to the combination is a dust pixel group constituting a dust image portion when the ratio is equal to or greater than a predetermined value.

  An image processing program according to an eighth aspect of the present invention is the image processing program according to the seventh aspect, wherein the dust specifying step is performed for the pixel group belonging to the combination of the barycentric positions of the pixel groups having the same number of pixels by the brightness adjustment step. A centroid position calculating step for calculating all of the centroid positions is further included. When the difference between the centroid positions is equal to or less than a predetermined value, the pixel group belonging to the combination is a step of determining that the pixel group constituting the dust image portion is a dust pixel group.

  An image processing program according to a ninth invention further includes a processing information storage step for storing processing information of at least one of the dust specifying step and the repairing step in the image processing program according to the first invention, wherein the dust specifying When there is processing information stored by the processing information storage step with respect to the reference image selected by the image selection step, a processing information reading step for reading the processing information, and for each pixel of the processing target image The brightness calculation step for calculating the brightness of the pixel, the pixel extraction step for extracting a pixel whose brightness calculated by the brightness calculation step is smaller than a predetermined value, and the pixels adjacent to the pixels extracted by the pixel extraction step are the same. Grouping to include in a group of pixels to form a dust candidate pixel group A combination generation step for generating a combination in which the dust pixel group and the dust candidate pixel group included in the processing information read by the processing information reading step overlap each other, and the combination generated by the combination generation step. A brightness adjustment step for adjusting the brightness of the processing target image so that the number of pixels included in the dust candidate pixel group is equal to the number of pixels included in the dust pixel group, and the number of pixels by the brightness adjustment step. A ratio calculation step of calculating a ratio of the number of pixels overlapping the dust pixel group to the number of pixels of the dust pixel group with respect to the dust candidate pixel group having equality, and the ratio calculated by the ratio calculation step is a predetermined value or more A determination step of determining that the dust candidate pixel group is a dust pixel group constituting a dust image portion at a certain time; And wherein the Mukoto.

  An image processing program according to a tenth aspect of the invention is the image processing program according to the ninth aspect, wherein the dust specifying step includes the positions of the gravity center of the dust pixel group and the dust candidate pixel group in which the number of pixels is equalized by the brightness adjustment step A center-of-gravity position calculating step for calculating each of the center-of-gravity positions; and The step of determining that the dust candidate pixel group is a dust pixel group constituting a dust image portion when the difference between the gravity center position and the gravity center position of the dust candidate pixel group is equal to or less than a predetermined value. .

  An image processing program according to an eleventh aspect of the invention is the image processing program according to any of the first to tenth aspects of the invention, wherein the repair step sets a plurality of regions so as to include the dust image portion specified by the dust specifying step. A region setting step to calculate, an average variation calculating step for calculating an average value and variation of a pixel value of a portion not including a dust image portion for each region set by the region setting step, and the average variation calculating step. A region selection step for selecting a region having the smallest variation, and a replacement step for replacing the pixel value of the dust image portion with the pixel value of the portion other than the dust image portion of the region selected by the region selection step. It is characterized by that.

  An image processing program according to a twelfth invention is characterized in that, in the image processing program according to the eleventh invention, the variation calculated by the average variation calculating step is a variation in brightness.

  An image processing program according to a thirteenth invention is the image processing program according to the eleventh or twelfth invention, wherein the variation is dispersion.

  An image processing program according to a fourteenth aspect of the invention is the image processing program according to any of the eleventh to thirteenth aspects of the invention, wherein the restoration step converts each area set by the area setting step into a partial image area of dust. The method further includes a partial region setting step of dividing into a plurality of partial regions so that the average variation calculation step includes dust in the partial regions divided by the partial region setting step for each region set by the region setting step. A pixel value of each partial region calculated by the average variation calculation step for each region set by the region setting step. The variation sum calculation step of calculating the variation sum by adding the variation of values And the region selecting step is a step of selecting a region having the smallest variation sum calculated by the variation sum calculating step, and a partial region not including the dust image portion of the region selected by the region selecting step. An estimated value calculating step of calculating an estimated value of the pixel value of the partial area including the dust image portion based on at least one of the average value and the variation of the pixel value of the pixel value, wherein the replacing step includes a pixel of the dust image portion The value is replaced with the pixel value of the partial region adjacent to the partial region including the dust image portion of the region selected by the region selection step, and is replaced with the pixel value of the dust image portion replaced by the replacement step. Add the difference between the average pixel value of the image part of the dust after and the average pixel value calculated in the estimated value calculation step. And further comprising a pixel value addition step that.

  An image processing program according to a fifteenth aspect of the present invention is the image processing program according to any of the first to fourteenth aspects of the present invention, which compares an image before the restoration by the restoration step with an image after the restoration by the restoration step. A confirmation image display step for displaying the image to be obtained, and a restoration cancellation step for returning the image to be processed to the image before the restoration when the restoration for the restoration by the restoration step is selected. .

  An image processing program according to a sixteenth aspect of the present invention is the image processing program according to any one of the first to fifteenth aspects, wherein the processing target image is changed in sequence among images captured by the same imaging device, so that a plurality of images are processed. It is characterized by sequentially performing processing on images.

An image processing method according to a seventeenth aspect of the present invention is an image processing method for removing, by image processing, an image portion of dust that can be included in an image taken by an imaging device, and taking an image of a processing target image An image selection step of selecting one or more images as reference images in order of photographing date and time from the one or more images photographed by the same imaging device as the device, the processing target image and the reference image And a dust specifying step for specifying an image portion of dust existing at substantially the same position, and a dust image portion in the processing target image specified by the dust specifying step as a part of the dust image portion in the processing target image. And a repairing step of repairing using image information in the vicinity.

  An image processing apparatus according to an eighteenth aspect of the present invention is an image processing apparatus for removing, from an image captured by an imaging apparatus, an image portion of dust that can be included in the image, by capturing an image to be processed. An image selection means for selecting one or more images as reference images in order of the shooting date and time from the one or more images captured by the same imaging device as the imaging device; the processing target image; and the reference A dust specifying means for specifying an image portion of dust existing at substantially the same position in both of the images, and an image portion of the dust in the processing target image specified by the dust specifying means as the dust image portion in the processing target image And repairing means for repairing using image information in the vicinity of.

  According to the image processing program, the image processing method, and the image processing apparatus of the present invention, it is possible to identify and properly repair the dust image portion from the image captured by the imaging apparatus.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIGS. 1 to 7 show Embodiment 1 of the present invention, and FIG. 1 is a block diagram showing a configuration of a main part of the image processing apparatus.

  This image processing apparatus is configured in substantially the same manner as a general information processing apparatus such as a personal computer, for example, and functions as an image processing apparatus by executing an image processing program in the information processing apparatus. It can be said that the processing method is implemented.

  That is, as shown in FIG. 1, the image processing apparatus includes a processing unit 1, a program memory 2, an image memory 3, an operation unit 4, a display unit 5, and a bus 7. ing.

  The operation unit 4 is an operation means for performing an operation input related to the image processing apparatus. For example, an operation button or the like displayed on the display unit 5 is operated via an input device such as a mouse or a keyboard. Therefore, it is configured to function as an operation unit. In accordance with the operation from the operation unit 4, it is possible to correct a missing part of the image due to dust or to display a corrected image.

  The image memory 3 is an image storage means for storing image data transmitted through a communication line or a memory card after being photographed by an imaging device (not shown). Further, the image memory 3 can also store image data after the dust portion is removed as described later.

  The display unit 5 is a display means for displaying an image stored in the image memory 3 or displaying an operation button or the like for processing a dust portion in the image. In addition, various information related to the image processing apparatus can be displayed.

  The program memory 2 is a program storage unit that stores an image processing program and can also store various data when the image processing program is executed.

  The processing unit 1 executes the image processing program stored in the program memory 2 to detect dust from the image data stored in the image memory 3 in response to an operation input from the operation unit 4. This is a processing means for detecting and correcting a part, and displaying the images before and after the processing on the display unit 5, and also serves as a control means for controlling each circuit in the image processing apparatus. . The processing unit 1 further serves as an image selection unit, a dust identification unit, and a restoration unit.

  The bus 7 connects the processing unit 1, the program memory 2, the image memory 3, the operation unit 4, and the display unit 5 so that they can communicate with each other bidirectionally.

  Next, FIG. 2 is a flowchart showing an outline of dust removal processing.

  When this process is started, first, the camera that has captured the processing target image and the shooting mode and the shooting date and time when the processing target image was captured are specified (step S1). Here, the shooting mode refers to, for example, camera shooting that may cause the shape of dust on the captured image to change, such as the pixel configuration of the image (that is, the number of pixels in the vertical direction and the number of pixels in the horizontal direction). Refers to each parameter setting. The “shooting date / time” widely includes time concepts such as year, month, day, hour, minute, and second when the image is taken.

  Next, a plurality of images close in shooting date and time are selected as reference images from images shot in the same shooting mode by the same camera that shot the processing target image (step S2). Specifically, a process of selecting a plurality of predetermined images is performed in order from the image having the closest shooting date / time to the shooting date / time of the processing target image.

  Then, a process for identifying dust is performed (step S3), an automatic dust removal process is performed (step S4), a processing result is confirmed (step S5), and an image of the processing result is stored (step S6). This process ends.

  Next, FIG. 3 is a flowchart showing an example of the dust specifying process in step S3 of FIG.

When this process is started, for a plurality of selected images (that is, the processing target image and the reference image), the brightness Y indicating the brightness of each pixel of each image is changed to the RGB chromaticity value that is the pixel value of each pixel. Based on this, for example, the calculation is performed as shown in the following formula 1 (step S11). ●
[Equation 1]
Y = R + G + B
Thereby, the brightness image data to which the brightness Y for each pixel is given is calculated by the number of selected images.

  Subsequently, brightness image data calculated corresponding to each of the selected plurality of images is added for each pixel at the same position (step S12). Thereby, one piece of added brightness image data in which the brightness Y is added for each pixel is calculated.

  Then, for each pixel, a pixel having an added value of brightness Y that is equal to or smaller than a predetermined value is a pixel to which dust is attached (step S13). Since dust can be estimated to exist at the same position in images of the same resolution, here, the point that remains black even after adding pixel data of a plurality of images obtained by shooting different scenes is a point due to dust. I try to judge.

  Thereafter, the dust pixels are separated into dust pixel groups (groups) by a labeling process (step S14). Since it is conceivable that a missing area of image data over a plurality of pixels is generated by one dust, it is classified here which missing area is caused by one dust. For the labeling process, a generally known method can be used, and among the pixels estimated to have dust attached in step S13, the pixels adjacent to each other are labeled with the same label, The dust pixel group is separated by performing processing such as waving a label with the next number on another pixel that is in an isolated position from the pixel labeled with, and that is estimated to have dust attached thereto. be able to.

  Then, the position and size of the pixel group corresponding to each dust is specified (step S15), and this dust specifying process is terminated.

  Next, FIG. 4 is a flowchart showing an example of automatic dust removal processing in step S4 of FIG. 2, and FIG. 5 is a diagram showing small areas of a plurality of shapes set so as to include dust pixel groups.

  When the process shown in FIG. 4 is started, first, the first dust in the dust of the labeled pixel group is set as the noticeable dust (step S21).

  Then, for the target dust, a plurality of small regions including dust pixel groups are set as shown in FIG. 5 (step S22). In the example shown in FIG. 5, a first region R1 that forms a vertical rectangle, a second region R2 that forms a horizontal rectangle, and a third rectangle that forms a right diagonal rectangle so as to include the dust pixel group TG. A region R3 and a fourth region R4 forming a rectangular shape in the diagonally left direction are set.

  Subsequently, the first region R1 is set as a region of interest (step S23), and an average value and a variance value of RGB values of pixels other than dust in the region of interest are calculated (step S24).

  Then, it is determined whether or not the average value and the variance value of the RGB values have been calculated for all the regions (step S25). If not, the next region is set as the attention region (step S26), and the above steps are performed. The process proceeds to S24 and the above-described process is performed for the next area.

  If it is determined in step S25 that the calculation has been completed for all the regions, the region having the smallest variance value is selected from the regions R1 to R4 including the target dust (step S27). ).

  Thereafter, the RGB value of the dust pixel group is replaced with the average value of the RGB values of the pixels other than the dust pixel group in the selected region (step S28).

  Then, it is determined whether or not the processing has been completed for all dust (step S29). If not, the next dust is set as the noticeable dust (step S30), and the process proceeds to step S22 as described above. Repeat the process.

  Thus, if it is determined in step S29 that all dust has been processed, this processing ends.

  Next, FIG. 6 is a flowchart showing an example of processing result confirmation processing in step S5 of FIG. 2, and FIG. 7 is a diagram showing a confirmation screen displayed on the display unit by the image processing program.

  Before describing the flow of processing, first, the configuration of the confirmation screen 11 in the display unit 5 will be described with reference to FIG.

  The confirmation screen 11 shows a screen displayed as one window on a multi-window type operating system having a graphic user interface, for example.

  The confirmation screen 11 displays, for example, an enlarged image 12 of the dust portion of the original image to be processed on the left side, and an enlarged image 13 after dust removal processing is performed on the right side. Further, for example, in the lower left of the confirmation screen 11, there are a total number of pixel groups regarded as dust regions in the processing target image, and how many dust regions in the group have been processed so far. A confirmed dust area number display 14 is displayed. In the confirmation screen 1 on the right side of the confirmed dust area number display 14, a cancel button 15 for canceling the result of the dust removal process displayed as the enlarged image 13 and the next dust area are processed. Next button 16 for performing and end button 17 for ending the confirmation of the dust removal processing are displayed to be operable.

  Next, the process shown in FIG. 6 will be described. When this process is started, first, a confirmation screen 11 as shown in FIG. 7 is displayed (step S31).

  Then, the enlarged image 12 of the first dust portion of the original image and the enlarged image 13 after processing are generated and displayed (step S32).

  Thereafter, it waits for an operation input to be made via the operation unit 4. If it is detected that the next button 16 has been pressed (step S33), it is determined whether or not all dust parts have been confirmed (step S34), and all dust parts have been confirmed. If not, the enlarged image 12 of the next dust portion of the original image and the enlarged image 13 after processing are generated and displayed (step S35), and the operation input is waited for. Return to the above state.

  If it is detected that the cancel button 15 has been pressed (step S36), the target dust portion is replaced with the corresponding original image portion, that is, the state before processing is restored (step S37). After that, the process goes to step S34 to perform the process as described above.

  Then, when it is detected that the end button 17 has been pressed (step S38), or when it is determined that all dust portions have been confirmed in step S34, this processing is ended.

  In the above description, the RGB value is taken as an example of the pixel value of the pixel, but it is not necessarily limited to this. Furthermore, the variance value is used as an index indicating the variation when detecting the region having the smallest variation in pixel values from a plurality of regions, but this is not a limitation, and other statistics are used. Is also possible. However, at this time, it is preferable to use a statistic that can express the variation satisfactorily and that can perform arithmetic processing at high speed.

  According to the first embodiment, it is possible to detect a missing part of an image due to dust and correct it on the basis of an image after photographing.

  At this time, a plurality of images close to the shooting date and time are used to remove dust portions that are considered to be common to these images from images shot with the same camera in the same shooting mode. Therefore, it is not necessary to detect dust by photographing a white subject or the like with a camera. This simplifies the operation at the time of shooting, eliminates the need for shooting a reference image for dust detection, and is an image forgetting to perform the dust detection operation in an imaging device having a dust detection function. Even if it exists, it becomes possible to remove a dust part later.

  Since dust is detected by adding pixel values for each pixel at the same position in a plurality of images, it is possible to detect dust with higher accuracy.

  In addition, since a plurality of different areas are set so as to include a dust portion in the image, an area with the smallest variation is found from these areas, and correction is performed using the data of the area. Data considered most likely to replace the dust portion data can be used, and the correction accuracy and reliability can be improved. At this time, by taking a plurality of different areas in different directions including the dust portion, it is possible to select a direction that is least susceptible to the influence even if there is an edge or the like.

  Furthermore, since the image of the automatic dust removal processing result is displayed side by side with the pre-processing image, it can be easily compared and confirmed, and the part that has been mistakenly processed as dust can be returned to the state before the processing. .

  8 to 12 show Embodiment 2 of the present invention, and FIG. 8 is a flowchart showing an outline of dust removal processing.

  In the second embodiment, the same parts as those of the first embodiment are denoted by the same reference numerals and the description thereof is omitted, and only different points will be mainly described.

  The dust removal process shown in FIG. 2 uses another image to process one processing target image. However, the dust removal process shown in FIG. A plurality of images that may be present are automatically processed together.

  When this process is started, first, a plurality of images close to the shooting date and time are selected as images to be processed from images shot by the same camera in the same shooting mode (step S41).

  Then, a process for identifying dust is performed for each selected image (step S42).

  Thereafter, the first image is set as a processing target image (step S43), automatic dust removal processing is performed (step S44), the processing result is confirmed (step S45), and the processing result image is stored (step S44). S46).

  Subsequently, it is determined whether or not all the images selected in step S41 have been processed (step S47). If there is an unprocessed image, the next image is set as a processing target image (step S48). The process returns to step S44 and the above-described processing is repeated.

  On the other hand, if it is determined in step S47 that all the images have been processed, this processing ends.

  FIG. 9 is a flowchart showing an example of dust identification processing in step S42 of FIG.

  When this processing is started, for a plurality of selected images, brightness Y indicating the brightness of each pixel of each image is calculated based on the RGB chromaticity value of each pixel, for example, as shown in Equation 1 above ( Step S51).

  Next, the average value of the brightness Y for each image (image average brightness) is calculated using the result (step S52). As a result, the average brightness Y is calculated for each image.

  Then, an image in which the calculated average value of brightness Y is equal to or less than a predetermined value is excluded (step S53).

  For a plurality of images remaining after the processing of step S53, the brightness Y is added for each pixel at the same position (step S54). Thereby, one piece of added brightness image data in which the brightness Y is added for each pixel is calculated.

  After that, for each pixel of the added brightness image data, a pixel in which the added value of brightness Y is equal to or less than a predetermined value is a pixel to which dust has adhered (step S55).

  Further, the dust pixels are separated into dust pixel groups (groups) by a labeling process (step S56), the position and size of the pixel group corresponding to each dust are specified (step S57), and the dust specifying process is terminated. To do.

  FIG. 10 is a flowchart showing another example of the dust specifying process in step S42 of FIG.

  When this process is started, first, the brightness Y from the RGB value of each pixel is shown in Equation 1 above for two images taken successively from the same camera in the same shooting mode. (Step S61). Here, the two images photographed in succession do not mean a continuously photographed image, but a time between the time when the first image was photographed and the time when the second image was photographed. It means that there is no photographed image.

  Next, for each image, pixels whose pixel brightness Y is equal to or less than a predetermined threshold are extracted (step S62), and for each pixel, the extracted pixels are grouped as a dust candidate pixel group by a labeling process (step S63). .

  Then, the gravity center position of each dust candidate pixel group is calculated for each image (step S64).

  Next, among the images selected in step S41 in FIG. 8, the image captured at the oldest time is set as the first image, and the image captured next to the first image is defined as the second image. (Step S65).

  Thereafter, a combination in which the dust candidate pixel group of the first image overlaps with the dust candidate pixel group of the second image is extracted (step S66). That is, the first image and the second image in which the dust candidates are at substantially the same position are extracted.

  Subsequently, for each combination extracted in step S66, the difference between the centroid position of the dust candidate pixel group of the first image and the centroid position of the dust candidate pixel group of the second image is equal to or less than a first predetermined value. Thus, the dust having a ratio of the overlapping area of the pixel groups with respect to the area of the dust candidate pixel group having a large number of pixels is specified as dust (step S67).

  Depending on the type of dust and the photographing conditions, the brightness of the peripheral portion of the dust may change even with the same dust. For this reason, when there is a large difference in the number of pixels in the dust candidate pixel group, the brightness of an image with a large number of pixels in the dust candidate pixel group is increased, or processing for decreasing the brightness of an image with a small number of pixels is added. After eliminating the difference in the number of pixels, step S67 may be executed.

  Next, it is determined whether or not the second image has been taken last in the image selected in step S41 in FIG. 8 (step S68). The second image is newly set as the first image, and the next image is taken as the second image (step S69), and the process as described above is performed by returning to step S66. Repeat.

  On the other hand, if it is determined in step S68 that the image has been taken last, it is determined that all the images selected in step S41 have been identified, and the process ends.

  Next, FIG. 11 is a flowchart showing an example of automatic dust removal processing in step S44 of FIG. 8, and FIG. 12 is a diagram showing a plurality of rectangular areas set to include dust pixel groups.

  When the process shown in FIG. 11 is started, first, the first dust among the dusts identified in step S42 is set as attention dust (step S71).

  Then, a plurality of rectangular areas RA having different angles are set as shown in FIG. 12 with respect to this noticeable dust (step S72). The rectangular area RA is divided into a plurality of partial areas PA, and is set so as to include the image portion of the noted dust in one of the divided partial areas PA. In the example shown in FIG. 12, the rectangular area RA is configured by arranging five partial areas PA having a substantially square shape on a straight line. Then, three rectangular areas facing in the three directions of the downward direction, the diagonally downward right direction, and the right direction are set so as to include the dust pixel group TG in the partial area PA on the one end side.

  Subsequently, the first rectangular area RA among the plurality of rectangular areas RA is set as the attention area (step S73), and the average value and variance regarding the RGB values are set for the plurality of partial areas not including dust in the attention area. A value is calculated for each of R, G, and B (step S74).

  Then, it is determined whether or not the calculation in step S74 has been performed for all the rectangular areas RA (step S75). If not, the next rectangular area RA is set as the attention area (step S76). The process proceeds to S74 and the above-described processing is performed for the next rectangular area RA.

  If it is determined in step S75 that the calculation for all the rectangular areas RA has been completed, the variance values of R, G, and B in the partial area PA are added to each rectangular area RA, and Further, the sum of the variance values is taken for the partial area not including the target dust. Thereby, the sum of the variance values is calculated corresponding to each rectangular area RA. Then, the rectangular area RA that minimizes the sum of the calculated variance values is selected (step S77).

  Next, with respect to the selected rectangular area RA, the least square method is used based on the average value and the variance value for each partial area PA and for each RGB value. An estimated value of the average value is calculated for each of R, G, and B (step S78).

  Subsequently, the image of the partial area PA including dust is replaced with the image of the partial area PA not including dust (for example, the partial area PA adjacent to the partial area PA including dust) (step S79).

  Then, in step S78, the average value of the R, G, B values of the partial area PA including the dust replaced in step S79 is equal to the estimated value of the average value of R, G, B values calculated in step S78. The difference between the estimated value of the average value of each of the R, G, and B values calculated in step S and the average value of each of the R, G, and B values of the partial area PA replaced in step S79 is calculated for each pixel of the replaced partial area. Are added to the R, G, and B values, respectively (step S80).

  Thereafter, it is determined whether or not the processing has been completed for all dust (step S81). If not, the next dust is set as the noticeable dust (step S82), and the process proceeds to step S72 as described above. Repeat the process.

  Thus, if it is determined in step S82 that all dust has been processed, this process is terminated.

  In the above description, the dust candidate pixel group is related to dust based on both the barycentric position of each pixel group and the ratio of the areas of the overlapping portions in the pixel group with the same number of pixels. Although it is determined whether or not it is a pixel group, it is also possible to determine using only one of them. At this time, the processing can be simplified and speeded up.

  According to the second embodiment, the same effects as the first embodiment described above can be obtained, and a plurality of images shot in the same shooting mode by the same camera can be collectively removed. Become.

  And when detecting dust, since the image whose average brightness value is below a predetermined value is excluded from a plurality of images, the detection accuracy can be improved.

  Further, when detecting dust, the brightness is adjusted so that the number of pixels of the dust candidate pixel group in a plurality of images becomes equal, and then whether or not the dust is based on the deviation of the center of gravity position and the overlapping ratio. Can be further improved.

  Furthermore, since a plurality of rectangular areas having different angles are set when removing dust, even if there is an edge or the like, it is possible to select a direction that is least affected by the influence.

  In addition, the rectangular area is divided into a plurality of partial areas, and the RGB value of the partial area including dust is calculated by the least square method based on the average value and the variance value of the RGB values of the partial area not including dust. Therefore, calculation with high accuracy can be performed. At this time, since the difference from the average value of the adjacent partial areas is added to the calculated estimated value, the accuracy can be further improved.

  FIGS. 13 to 16 show a third embodiment of the present invention, and FIG. 13 is a block diagram showing a configuration of a main part of the image processing apparatus.

  In the third embodiment, the same parts as those in the first and second embodiments are denoted by the same reference numerals, description thereof is omitted, and only different points will be mainly described.

  This image processing apparatus is obtained by adding a processing history memory 6 to the configuration shown in FIG.

  The processing history memory 6 is processing history storage means for recording a history of dust removal processing relating to images processed in the past, and is connected to the bus 7 and includes the processing unit 1, program memory 2, and image memory. 3, the operation unit 4 and the display unit 5 are bidirectionally connected.

  Since there is a high possibility that the same dust is attached to images that are close to the shooting time, dust information (position and pixel information) processed in the past is stored in the processing history memory 6 as a dust removal processing history. In addition, when a new dust removal process is performed, dust can be efficiently detected by referring to the dust information stored in the processing history memory 6.

  Next, FIG. 14 is a flowchart showing an outline of dust removal processing.

  When this process is started, first, the camera that captured the processing target image, the shooting mode, and the shooting date and time are specified (step S91).

  Next, from the processing history memory 6, the dust removal processing information relating to the images whose shooting dates and times are close is read out from the images shot in the same shooting mode by the same camera that shot the processing target image ( Step S92).

  Then, processing for identifying dust is performed (step S93), automatic dust removal processing is performed (step S94), processing results are confirmed (step S95), and an image of the processing results is stored (step S96). This process ends.

  Next, FIG. 15 is a flowchart showing an example of the dust specifying process in step S93 of FIG.

  When this process is started, the brightness Y indicating the brightness of each pixel is calculated for the processing target image based on the RGB chromaticity value of each pixel as shown in Equation 1 (step S101).

  Subsequently, for each pixel of the processing target image, a pixel in which the calculated value of brightness Y is a predetermined value or less is extracted (step S102).

  Then, the extracted pixels are subjected to a labeling process to be grouped as a dust candidate pixel group (step S103).

  Thereafter, the dust candidate pixel group is compared with the dust pixel group at the same position in the image processed in the past read from the processing history memory 6 to determine whether or not there is an overlapping portion. Then, the dust candidate pixel group determined to have an overlapping portion is extracted together with the corresponding dust pixel group (the dust pixel group read from the processing history memory 6) (step S104).

  Next, for each combination, the brightness of the dust candidate pixel group is adjusted so that the number of pixels whose brightness Y is equal to or less than a predetermined threshold is equal to the number of pixels of the dust pixel group (step S105).

  When the number of pixels of the dust candidate pixel group and the number of pixels of the dust pixel group become equal in this way, the gravity center position of the dust candidate pixel group and the gravity center position of the dust pixel group are calculated for each combination (step S106).

  When the difference between these barycentric positions is equal to or smaller than a predetermined value and the ratio of the area of the overlapping portion to the area of the pixel group is equal to or larger than the predetermined value, the dust candidate pixel group in the processing target image is (Step S107), and thereafter, this process is terminated.

  The automatic dust removal process in step S94 to be executed next after returning to the process in FIG. 14 is the same process as in step S4 or step S44.

  Note that dust having no registered process information cannot be identified only by the process of step S93. Therefore, the process of step S3 or step S42 is executed after the process of step S93, and only the process of step S93 is performed. Then, dust that could not be identified is identified.

Next, FIG. 16 is a flowchart showing an example of processing result confirmation processing in step S95 of FIG.

  When this process is started, first, the confirmation screen 11 as shown in FIG. 7 is displayed (step S111).

  And the enlarged image 12 of the 1st dust part and the enlarged image 13 after a process are produced | generated and displayed (step S112).

  Thereafter, it waits for an operation input to be made via the operation unit 4. If it is detected that the next button 16 has been pressed (step S113), the position and pixel information of the displayed dust pixel group are registered in the processing history memory 6 in association with the image (step S113). S114).

  Next, it is determined whether or not all dust portions have been confirmed (step S115). If confirmation has not been performed for all dust portions, an enlarged image of the next dust portion and a post-processing image are processed. After the enlarged image is generated and displayed (step S116), the process returns to the above-described state where the operation input is awaited.

  If it is detected that the cancel button 15 has been pressed (step S117), the target dust portion is replaced with the corresponding original image portion, that is, the original state is restored (step S118). Thereafter, the process proceeds to step S115, and the above-described processing is performed.

  If it is detected that the end button 17 has been pressed (step S119), it is determined whether or not all dust portions have been confirmed (step S120), and all dust portions have been confirmed. If not, the position and pixel information of the unidentified dust pixel group are registered in the processing history memory 6 in association with the image (step S121). On the other hand, in this step S120 or step S115, all dust is collected. If it is determined that the part has been confirmed, this process is terminated.

  According to the third embodiment, the effects similar to those of the first and second embodiments described above are obtained, and the history of dust removal processing is stored. It becomes possible to carry out efficiently.

  Also, when confirming the processing result, if the confirmation is completed without performing all the dust portion confirmation, the pixel information of the unconfirmed dust pixel group is associated with the image and registered in the processing history memory. Therefore, it becomes possible to easily restart the process later from the point where it was stopped. The pixel information of the dust pixel group that has not been confirmed can be used for processing other images. Furthermore, it is also possible to return the dust that has been removed by the next button 16 to the original image as necessary.

  In each of the above-described embodiments, an example in which dust is detected and removed based on an image photographed by an imaging device such as a digital camera or a mobile phone with a camera has been described. The above technique is also applied to an image obtained by capturing a film photographed by a silver salt camera with a film scanner or the like, an image read by a scanner part in a copying machine, or an image read by a scanner part in a facsimile apparatus. It is possible to apply.

  In the above description, dust detection and the like are performed using images captured in the same imaging mode in the same imaging device, but the present invention is not limited to this. For example, even with images of different pixel configurations shot in different shooting modes, if there is information such as which part of the image sensor output was cut out, the corresponding pixel position can be specified. For this reason, it is possible to detect dust and the like for common portions.

  In the above description, an image whose shooting date and time are close to each other is used as a reference image. However, a reference image group may be selected in consideration of an interval between shooting dates and times. In other words, when a certain image group is photographed intensively within a short time interval, and the next image group is photographed several days later, for example, a group of image groups photographed intensively is collected. The reference image group can be considered.

  In addition, if the imaging device is a single-lens reflex digital camera, etc., it is highly likely that dust will adhere to the imaging surface of the imaging device when the lens is replaced. A group of images recorded while the same lens continues to be worn may be used as a group of reference images.

  In addition, this invention is not limited to the Example mentioned above, Of course, a various deformation | transformation and application are possible within the range which does not deviate from the main point of invention.

  The present invention can be suitably used for a method, a program, an apparatus, and the like for removing an image portion of dust included in an image.

1 is a block diagram illustrating a configuration of a main part of an image processing apparatus according to Embodiment 1 of the present invention. The flowchart which shows the outline | summary of the dust removal process in the said Example 1. FIG. The flowchart which shows the example of the dust specific process in step S3 of the said FIG. The flowchart which shows the example of the automatic dust removal process in step S4 of the said FIG. In the said Example 1, the figure which shows the small area | region of multiple shapes set so that a dust pixel group may be included. The flowchart which shows the example of the confirmation process of the process result in step S5 of the said FIG. The figure which shows the mode of the confirmation screen displayed on the display part by the image processing program in the said Example 1. FIG. The flowchart which shows the outline | summary of the dust removal process in Example 2 of this invention. The flowchart which shows an example of the dust specific process in step S42 of the said FIG. The flowchart which shows the other example of a dust specific process in step S42 of the said FIG. The flowchart which shows the example of the automatic dust removal process in step S44 of the said FIG. In the said Example 2, the figure which shows the several rectangular area | region set so that a dust pixel group might be included. FIG. 9 is a block diagram illustrating a configuration of a main part of an image processing apparatus according to Embodiment 3 of the present invention. 10 is a flowchart showing an outline of dust removal processing in the third embodiment. The flowchart which shows the example of the dust specific process in step S93 of the said FIG. The flowchart which shows the example of the confirmation process of the process result in step S95 of the said FIG.

Explanation of symbols

1. Processing unit (image selection means, dust identification means, restoration means)
2 ... Program memory 3 ... Image memory 4 ... Operation unit 5 ... Display unit 6 ... Processing history memory 7 ... Bus 11 ... Confirmation screen 12 ... Enlarged image of the dust part to be processed 13 ... Dust removal process Enlarged image 14 ... Confirmed number of dust areas 15 ... Cancel button 16 ... Next button 17 ... End button TG ... Dust pixel group R1, R2, R3, R4 ... Area RA ... Rectangular area PA ... Partial area
Agent Patent Attorney Susumu Ito

Claims (18)

An image processing program for removing, by image processing, an image portion of dust that can be included in an image from an image captured by an imaging device by a computer,
An image selection step of selecting one or more images as reference images in order of shooting date and time closest to the processing target image from one or more images captured by the same imaging device as the imaging device that captured the processing target image; ,
A dust specifying step for specifying an image portion of dust existing at substantially the same position in both the processing target image and the reference image;
A repairing step of repairing the dust image portion in the processing target image specified by the dust specifying step using image information in the vicinity of the dust image portion in the processing target image;
An image processing program for causing a computer to execute. The imaging device is configured to include an image sensor,
The image processing program according to claim 1, wherein the dust is attached to an imaging surface of the image sensor. The imaging device is configured to include an image sensor,
The image processing program according to claim 1, wherein the substantially same position is substantially the same position on the imaging surface of the image sensor.   The image processing program according to claim 1, wherein the image selection step is a step of selecting the reference image from images having the same pixel configuration as the processing target image. The dust identification step
A brightness calculation step for calculating brightness for each pixel of the processing target image and brightness for each pixel of the reference image;
A brightness addition step of adding the brightness for each pixel of the processing target image calculated by the brightness calculation step and the brightness for each pixel of the reference image for each same pixel position;
A pixel extraction step for extracting pixels in which the lightness addition value added in the lightness addition step is smaller than a predetermined value;
A grouping step for grouping pixels in the processing target image extracted by the pixel extraction step so that adjacent pixels are included in the same group;
Including
The image processing program according to claim 1, wherein the computer identifies the pixel group grouped by the grouping step as a dust pixel group constituting a dust image portion. The dust identification step
Average brightness for calculating the image average brightness by averaging the brightness for each pixel calculated in the brightness calculation step for all the pixels included in the image for all reference images selected in the image selection step A calculation step;
An image excluding step of excluding a reference image in which the image average brightness calculated by the average brightness calculating step is a predetermined value or less from an addition target by the brightness adding step;
The image processing program according to claim 5, further comprising: The dust identification step
A brightness calculation step for calculating brightness for each pixel of the processing target image and brightness for each pixel of the reference image;
A pixel extraction step for extracting pixels whose lightness calculated by the lightness calculation step is smaller than a predetermined value;
A grouping step for grouping the pixels extracted in the pixel extraction step so that adjacent pixels are included in the same group;
Among the pixel groups grouped by the grouping step, pixels in which the pixel group in the processing target image and the pixel group in the reference image overlap each other for the processing target image and all the reference images A combination generation step for generating a group combination;
For each combination of pixel groups generated by the combination generation step, a brightness adjustment step for adjusting the brightness for each image so that the number of pixels included in the pixel group is the same for all of the pixel groups belonging to the combination;
A ratio calculation step of calculating a ratio of the number of pixels of a portion where all of the pixel groups belonging to the combination overlap with respect to the number of pixels of one pixel group, with respect to the pixel group having the same number of pixels by the brightness adjustment step;
A determination step of determining that the pixel group belonging to the combination is a dust pixel group constituting a dust image portion when the ratio calculated in the ratio calculation step is equal to or greater than a predetermined value;
The image processing program according to claim 1, further comprising: The dust identification step
A centroid position calculating step of calculating the centroid position of the pixel group having the same number of pixels by the brightness adjustment step for all of the pixel groups belonging to the combination;
In the determination step, the ratio calculated in the ratio calculation step is equal to or greater than a predetermined value, and the difference between the most distant centroid positions among the centroid positions calculated in the centroid position calculation step is a predetermined value. 8. The image processing program according to claim 7, wherein the image processing program is a step of determining that a pixel group belonging to the combination is a dust pixel group constituting a dust image portion when: A processing information storage step for storing processing information of at least one of the dust identification step and the repair step;
In the dust specifying step, when the processing information stored in the processing information storing step is present with respect to the reference image selected in the image selecting step,
A processing information reading step for reading the processing information;
A brightness calculation step of calculating the brightness of each pixel of the processing target image;
A pixel extraction step for extracting pixels whose lightness calculated by the lightness calculation step is smaller than a predetermined value;
Grouping the pixels extracted by the pixel extraction step into a dust candidate pixel group by grouping adjacent pixels into the same group; and
A combination generation step for generating a combination in which the dust pixel group included in the processing information read out by the processing information reading step and the dust candidate pixel group overlap;
For each of the combinations generated by the combination generation step, the brightness of the processing target image is set so that the number of pixels included in the dust candidate pixel group is equal to the number of pixels included in the dust pixel group. A brightness adjustment step to adjust;
A ratio calculating step for calculating a ratio of the number of pixels overlapping the dust pixel group to the number of pixels of the dust pixel group for the dust candidate pixel group having the same number of pixels by the brightness adjustment step;
A determination step of determining that the dust candidate pixel group is a dust pixel group constituting a dust image portion when the ratio calculated in the ratio calculation step is equal to or greater than a predetermined value;
The image processing program according to claim 1, further comprising: The dust identification step
A centroid position calculating step for calculating the centroid position of the dust pixel group and the centroid position of the dust candidate pixel group in which the number of pixels is equalized by the brightness adjustment step;
In the determination step, the ratio calculated in the ratio calculation step is a predetermined value or more, and the difference between the gravity center position of the dust pixel group and the gravity center position of the dust candidate pixel group calculated in the gravity center position calculation step is determined. 10. The image processing program according to claim 9, wherein when the value is equal to or smaller than a predetermined value, the dust candidate pixel group is a step of determining that the dust pixel group is a dust pixel group constituting a dust image portion. The repair step is
A region setting step for setting a plurality of regions so as to include the dust image portion specified by the dust specifying step;
For each region set by the region setting step, an average variation calculating step for calculating an average value and a variation of a pixel value of a portion not including the dust image portion,
A region selection step for selecting a region having the smallest variation calculated by the average variation calculation step;
A replacement step of replacing the pixel value of the dust image portion with a pixel value of a portion other than the dust image portion of the region selected by the region selection step;
The image processing program according to claim 1, further comprising:   12. The image processing program according to claim 11, wherein the variation calculated in the average variation calculating step is a variation in brightness.   13. The image processing program according to claim 11, wherein the variation is a variance. The repair step is
A partial region setting step of dividing each region set by the region setting step into a plurality of partial regions so that the dust image portion is included in one partial region;
The average variation calculating step includes, for each region set by the region setting step, an average pixel value of each partial region that does not include the dust image portion of the partial region divided by the partial region setting step, and A step of calculating variation,
For each region set by the region setting step, the method further includes a variation sum calculation step of calculating a variation sum by adding the variation of the pixel value of each partial region calculated by the average variation calculation step,
The region selection step is a step of selecting a region having the smallest variation sum calculated by the variation sum calculation step,
Based on at least one of the average value and the variation of the pixel values of the partial regions not including the dust image portion in the region selected by the region selection step, the estimated value of the pixel value of the partial region including the dust image portion is calculated. An estimated value calculating step for calculating,
The replacement step is a step of replacing the pixel value of the dust image portion with the pixel value of the partial region adjacent to the partial region including the dust image portion of the region selected by the region selection step;
The difference between the average value of the pixel values of the dust image portion after the replacement and the average value of the pixel values calculated by the estimated value calculation step is calculated as the pixel value of the dust image portion replaced by the replacement step. The image processing program according to claim 11, further comprising a pixel value adding step of adding. A confirmation image display step for displaying the image before the repair by the repair step and the image after the repair by the repair step so as to be compared;
A restoration cancellation step for returning the processing target image to the image before the restoration when the selection to cancel the restoration by the restoration step is performed;
The image processing program according to claim 1, further comprising:   The process for a plurality of images is sequentially performed by sequentially changing the processing target image among images captured by the same imaging device. The image processing program described. An image processing method for removing, by image processing, an image portion of dust that can be included in an image taken by an imaging device,
An image selection step of selecting one or more images as reference images in order of shooting date and time closest to the processing target image from one or more images captured by the same imaging device as the imaging device that captured the processing target image; ,
A dust specifying step for specifying an image portion of dust existing at substantially the same position in both the processing target image and the reference image;
A repairing step of repairing the dust image portion in the processing target image specified by the dust specifying step using image information in the vicinity of the dust image portion in the processing target image;
An image processing method comprising: An image processing device for removing, from an image captured by an imaging device, an image portion of dust that can be included in the image by image processing,
Image selecting means for selecting one or more images as reference images in order of shooting date and time closest to the processing target image from one or more images captured by the same imaging device as the imaging device that captured the processing target image; ,
Dust specifying means for specifying an image portion of dust existing at substantially the same position in both the processing target image and the reference image;
Repairing means for repairing the dust image portion in the processing target image specified by the dust specifying means using image information in the vicinity of the dust image portion in the processing target image;
An image processing apparatus comprising:

Images