WO2012133228A1

WO2012133228A1 - Image processing apparatus and image processing program

Info

Publication number: WO2012133228A1
Application number: PCT/JP2012/057604
Authority: WO
Inventors: 豪松本
Original assignee: 株式会社ニコン
Priority date: 2011-03-31
Filing date: 2012-03-23
Publication date: 2012-10-04

Abstract

Provided are an image processing apparatus that is capable of more easily detecting a foreign material in a photographed image, and an image processing program. An image processing apparatus (50) of the present invention is characterized in being provided with: a position selecting unit (34), which can select a specific position in a first photographed image (G1); a range determining unit (61), which determines a fixed range (E1) in the first photographed image (G1), said range including the position selected by the position selecting unit (34); an image selecting unit (62), which selects a second photographed image (G2) that is different from the first photographed image (G1); and a foreign material image information acquiring unit (63), which acquires information of a foreign material shadow photographed in the second photographed image (G2), from a corresponding range (E2) in the second photographed image (G2), said corresponding range corresponding to the fixed range (E1) in the first photographed image (G1).

Description

Image processing apparatus and image processing program

The present invention relates to an image processing apparatus and an image processing program.

When capturing an image with an electronic camera or the like, if a foreign object adheres to the optical filter or the like, a shadow of the foreign object may appear in the captured image. As an image processing apparatus for correcting an image defect site due to a foreign object shadow or the like reflected in the image in this way, information such as the coordinates of the foreign object position is obtained from the reference image in which the foreign object shadow is reflected, and normal imaging is performed based on the information. There is known an image processing apparatus that corrects an image defect caused by a foreign object shadow reflected in a captured image.
Further, a technique is known in which a foreign object is detected from a change in transmittance (change in luminance information in each pixel) due to a foreign object shadow in an image to be corrected, and correction for deleting the foreign object shadow is performed (see Patent Document 1).

JP 2004-220553 A

However, the above prior art needs to capture a reference image in which foreign matter is reflected. In addition, the coordinates of the foreign substance position may differ between the captured image and the reference image.

An object of the present invention is to provide an image processing apparatus and an image processing program that can more easily detect foreign matter in a captured image.

The present invention solves the above problems by the following means.
According to the first aspect of the present invention, the position selection unit capable of selecting a specific position in the first photographed image, and a constant including the position selected by the position selection unit in the first photographed image. A range determining unit for determining a range; an image selecting unit for selecting a second captured image different from the first captured image; and the predetermined range of the first captured image in the second captured image And a foreign object shadow information acquisition unit that acquires foreign object shadow information that has moved into the second captured image from a corresponding range corresponding to the image processing apparatus.

In the first aspect, the image selection unit may select an image shot under shooting conditions close to the shooting conditions of the first shot image as the second shot image.
In the first aspect, the imaging condition is that the camera body is the same as the camera body that captured the first captured image, and at least one of an aperture value, an imaging distance, a lens, a focal length, and an imaging time. May be the same.
In the first aspect, the foreign object shadow information acquisition unit may acquire the foreign object shadow information using luminance information.
In the first aspect, the image selection unit has the foreign object shadow information in the corresponding range when the second captured image has the same file type as the first captured image, and You may select the image whose brightness | luminance around a corresponding range is the same as the 1st picked-up image, or the nearest.
In the first aspect, the image selection unit has the foreign object shadow information in the corresponding range when the second captured image has the same file type as the first captured image, and An image whose hue around the range is the same as or closest to the first photographed elephant image may be selected.
In the first aspect, the image selection unit selects, as the second captured image, an image having the foreign object shadow information in the corresponding range and having the smallest standard deviation of luminance around the corresponding range. May be.
In the first aspect, the image selection unit may create data for correcting the first photographed image from a brightness profile averaged over the corresponding range in the second photographed image.
In the first aspect, the image selection unit may select a photographed image other than the first photographed image with a different shutter speed as the second photographed image.
In the first aspect, the image processing apparatus may include an image correction unit.

According to the second aspect of the present invention, the second photographed image for selecting the second photographed image for acquiring the foreign object shadow information from the stored photographed image based on the photographing condition of the first photographed image to be corrected. The selection unit extracts a foreign object shadow candidate that may be a foreign object shadow from among the shadows reflected in the captured image, and for each captured image, coordinates of the position where the foreign object shadow candidate exists and the coordinates The foreign object shadow information acquisition unit that acquires the luminance reduction amount of the foreign object shadow candidate part, the coordinates of the position where the foreign object shadow candidate exists in the second captured image, and the foreign object shadow acquired by the foreign object shadow information acquisition unit There is provided an image processing apparatus comprising: a correction unit that corrects the first captured image based on a luminance decrease amount of a candidate portion.

According to the third aspect of the present invention, the second photographed image for selecting the second photographed image for acquiring the foreign object shadow information from the stored photographed image based on the photographing condition of the first photographed image to be corrected. The selection unit extracts a foreign object shadow candidate that may be a foreign object shadow from among the shadows reflected in the captured image, and for each captured image, coordinates of the position where the foreign object shadow candidate exists and the coordinates A foreign object shadow information acquisition unit that acquires a luminance decrease amount of a part of the foreign object shadow candidate, and adopts the foreign object shadow candidate as a foreign object shadow, and the coordinates of the foreign object shadow candidate acquired by the foreign object shadow information acquisition unit are An image processing apparatus comprising: a display unit configured to display a first photographed image.

In the second aspect or the third aspect, the second photographed image selection unit may select an image photographed under photographing conditions close to the photographing condition of the first photographed image as the second photographed image.
2nd aspect or 3rd aspect WHEREIN: The said imaging conditions are the camera body same as the camera body which image | photographed said 1st picked-up image, an aperture value, a shooting distance, a lens, a focal distance, and shooting time And at least one of them may be the same.
In the second aspect or the third aspect, when the standard deviation of the peripheral brightness of the foreign object shadow candidate in the second photographed image is a threshold value or more, the foreign object shadow candidate is not adopted as the foreign object shadow. An image processing apparatus.
In the second aspect or the third aspect, when the standard deviation of a certain range including the position of coordinates corresponding to the foreign object shadow candidate in the first photographed image is equal to or smaller than a threshold value, the foreign object shadow candidate is determined as the foreign object shadow. The image processing apparatus is characterized in that it is not employed.
In the second aspect or the third aspect, there are a plurality of second photographed images selected by the second photographed image selection unit, and foreign object shadow candidates existing at the same coordinate positions are derived from the plurality of second photographed images. If selected, a certain range of brightness including the foreign object shadow candidate and a certain range of brightness including the portion corresponding to the position of the coordinate in the first captured image are the closest foreign object shadow candidates of the second captured image. Is used as a foreign object shadow.
In the second aspect or the third aspect, there are a plurality of second photographed images selected by the second photographed image selection unit, and foreign object shadow candidates existing at the same coordinate positions are derived from the plurality of second photographed images. If selected, a certain range of hues including the foreign object shadow candidate and a certain range of hues including the portion corresponding to the position of the coordinate in the first captured image are the closest foreign object shadow candidates of the second captured image. Is used as a foreign object shadow.

According to a fourth aspect of the present invention, using the shooting information of the first captured image to be corrected, the selection unit that selects the second captured image, and the foreign object information that is information regarding the foreign object shadow from the second captured image are acquired. An image processing apparatus.

According to a fifth aspect of the present invention, there is provided a position selection unit capable of selecting a specific position in the first photographed image, a range determination unit for determining a certain range including the position selected by the position selection unit, An image selection unit that selects a second captured image different from the one captured image, and a foreign object shadow information acquisition unit that acquires foreign object shadow information from a corresponding range corresponding to the certain range in the second captured image. An image processing apparatus characterized by the above.

The sixth aspect of the present invention provides an image processing program for causing a computer to execute the function of the image processing apparatus of the above aspect.

According to the present invention, it is possible to provide an image processing apparatus and an image processing program that can more easily detect a foreign object in a captured image.

It is a figure which shows the structure of the camera of an interchangeable lens type in embodiment of this invention. 1 is a block diagram illustrating a personal computer and peripheral devices that constitute a camera and an image processing apparatus. 1 is a functional block diagram of an image processing apparatus that performs an image defect correction process according to a first embodiment. FIG. 3 is a functional block diagram of a defect determination unit in the image processing apparatus according to the first embodiment. It is a flowchart explaining the flow of the foreign matter shadow information detection by the image processing apparatus of 1st Embodiment, and the correction process of an image. It is an example of the 1st picked-up image displayed on the monitor in 1st Embodiment and 2nd Embodiment. It is an example of the 2nd picked-up image in 1st Embodiment and 2nd Embodiment. It is an example of the 2nd picked-up image in 1st Embodiment and 2nd Embodiment. It is a functional block diagram of an image processing apparatus that performs foreign object shadow information acquisition and image correction processing according to the second embodiment. It is a functional block diagram of the foreign substance shadow information acquisition part in the image processing apparatus of 2nd Embodiment. It is a flowchart of foreign object shadow information acquisition of a 2nd embodiment. It is a flowchart of the correction process of the image of 2nd Embodiment.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a diagram illustrating a configuration of a lens-interchangeable camera 1. FIG. 2 is a block diagram showing a personal computer (PC) 30 and peripheral devices constituting the camera 1 and the image processing apparatus.
The camera 1 is an electronic still camera that outputs image data obtained by converting a subject image into an electrical signal. In the present embodiment, image data captured by the camera 1 is acquired by a PC 30 functioning as an image processing apparatus, and foreign object shadow detection and correction processing described later is performed.

The camera 1 includes a camera body 2 and an imaging lens 3 that can be attached to and detached from the camera body 2. The imaging lens 3 includes an imaging optical system 3L having a plurality of optical lens groups (shown as one lens in FIG. 1).

The camera body 2 includes a shutter 11, an optical filter 12, an image sensor 13, an image processing unit 14, an operation unit 15, a display unit 16, a memory card interface (hereinafter referred to as “memory card IF”) 17, an external interface (hereinafter referred to as “memory card IF”). , “External IF”) 18, a power source (not shown), a control unit 20 and the like.

The shutter 11 adjusts the exposure time by shielding and passing photographing light from the imaging optical system 3L toward the image sensor 13.
The optical filter 12 is configured by an optical low-pass filter or the like that prevents generation of false colors (color moire) or the like during imaging. Note that reference numeral 4 in FIG. 1 indicates foreign matters such as dust and dirt attached to the surface of the optical filter 12. If the foreign matter 4 is attached to the optical filter 12 that is on the optical path of the photographing light and transmits the photographing light, pixels that are affected by the foreign matter shadow in the image data acquired by the image sensor 13. May be included.

The imaging element 13 is a photoelectric conversion element such as a CCD or CMOS, and has a rectangular imaging area composed of a plurality of pixels. The image sensor 13 converts the image formed on the image plane into an electrical signal and outputs an image signal (imaging signal) corresponding to the subject image. The image signal output from the image sensor 13 is A / D converted and input to the image processing unit 14 after predetermined analog signal processing.

The image processing unit 14 performs image processing such as interpolation, gradation conversion, and edge enhancement on the image data input from the image sensor 13.
The operation unit 15 includes a release button for inputting a signal for determining a shooting timing by the photographer, a selection button for mode switching, and the like. And the operation part 15 inputs those operation information into a control part.

The display unit 16 is configured by an LCD or the like, and displays various setting menus in the camera 1, a reproduced image based on an image captured by the image sensor 13 and image data stored in a memory card.

The memory card IF 17 serves as an interface with a memory card (card-like removable memory) 40.
The external IF 18 serves as an interface with an external device such as the PC 30 via a predetermined cable or a wireless transmission path.
The power supply supplies power to each functional unit of the camera 1.

The control unit 20 controls each functional unit in the camera 1 in an integrated manner. That is, when a shooting instruction signal is input from the release switch in the operation unit 15, the control unit 20 controls the driving of the shutter 11 to control a shooting operation such as an exposure operation. Further, the control unit 20 stores the image data acquired by the image sensor 13 and subjected to image processing by the image processing unit 14 in a memory card 40 or the like that is detachably attached to the memory card IF 17. ,save. Further, the control unit 20 calls the image data stored in the memory card 40 or the like and causes the display unit 16 to display the image data.

The camera 1 configured as described above is controlled by the control unit 20 at the time of shooting and operates as follows.
When the release button of the operation unit 15 is pressed by the operator, the shutter 11 is opened for a predetermined time. The imaging device 13 generates an image signal corresponding to the optical image formed in the imaging region by the imaging optical system 3L. The image signal generated by the image sensor 13 is input as image data to the image processing unit 14 and is subjected to predetermined image processing by the image processing unit 14. Then, the captured image is displayed on the display unit 16 based on the image data processed by the image processing unit 14, and the image data is subjected to compression processing by JPEG, RAW, or the like as necessary, and is stored in the memory card IF 17. Store / store in the installed memory card 40. In addition, image data is sent to a PC 30 (described later) connected via the external IF 18.

Here, the image data stored / saved in the memory card 40 or transmitted to the PC 30 via the external IF 18 is recorded in, for example, an Exif (EXchange Image File format) file. In the Exif file, additional information such as the camera body 2 type, camera manufacturer name, imaging pixel information, aperture value (F value) and exposure conditions such as focus, shutter speed and exposure correction, shooting time, and shooting distance are recorded together. Is done.

Although not shown, the personal computer (PC) 30 serves as an interface with an external device such as the camera 1 via a memory card IF, a cable, or a wireless transmission path that serves as an interface with the CPU, memory, hard disk, and memory card 40. External IF etc. are provided.
The PC 30 is connected to a monitor 31, a printer 32, and the like as an image display unit, and the user arbitrarily selects a specific position where a foreign object such as a foreign object shadow appears in a captured image displayed on the monitor 31. A mouse 34 is connected as a selection unit to be indicated.

Further, the CD-ROM 33 connected to the PC 30 is preinstalled with an application program for detecting a foreign object such as a foreign object shadow and correcting the image based on information of the detected foreign object.

(First embodiment)
Next, a first embodiment for correcting foreign matter in a first photographed image that has been photographed will be described.
In the first embodiment, foreign object shadow information such as foreign object shadow in a first photographed image that has been photographed is detected, and the first captured image is corrected based on the detected foreign object shadow information (foreign object shadow information detection). • Perform correction processing.

The foreign object shadow information detection / correction processing is performed by the PC 30 on the image data after completion of image processing provided via the memory card 30 or transmitted from the external interface 18 of the camera 1.
As described above, the foreign object shadow information detection and image correction processing application program is installed in the PC 30, and the PC 30 executes the program to perform foreign object shadow information detection and image correction processing. It functions as the device 50.

FIG. 3 is a functional block diagram of the image processing apparatus 50 that performs foreign object shadow information detection / correction processing. FIG. 4 is a functional block diagram of the foreign object shadow information detection unit 60 in the image processing apparatus 50.
As shown in FIG. 3, the image processing apparatus 50 functioning by the PC 30 and the program is based on the foreign object shadow information detection unit 60 that detects foreign object shadow information and the foreign object shadow information detected by the foreign object shadow information detection unit 60. And a correction unit 56 that corrects the foreign object shadow caused by the foreign object.
As shown in FIG. 4, the foreign object shadow information detection unit 60 includes a range determination unit 61, an image selection unit 62, and a foreign object shadow information acquisition unit 63. The foreign object shadow information acquisition unit 63 further includes a connected region determination unit 64, a target region extraction unit 65, and a luminance determination unit 66.

FIG. 5 is a flowchart for explaining the flow of foreign object shadow information detection and image correction processing in the image processing apparatus 50. The flow of foreign object shadow information detection and image correction processing in the image processing apparatus 50 will be described with reference to the flowchart of FIG. In FIG. 5 and the following description, the step is also abbreviated as “S”.

First, in step S01, the PC 30 including the image processing device 50 reads data of the first captured image G1 (correction target) input via the memory card 40 or the external interface 18 of the camera 1. .

In step S02, the read first photographed image G1 is displayed on the monitor 31. FIG. 6 shows an example of the first captured image G1 displayed on the monitor 31. In the first photographed image G1 of FIG. 6, there is a foreign object shadow 4A in a portion surrounded by trees. Such a foreign object shadow is caused by, for example, a foreign substance (foreign substance 4) (see FIG. 1) attached to the front surface of the optical filter 12.

Here, the user can recognize the foreign object shadow part, but the luminance gradient of the foreign object shadow 4A and the luminance gradient of the trees overlap, and it may be difficult to detect the foreign object shadow 4A information from the luminance gradient. is there.

Therefore, first, in step S03, when the user clicks the position of the foreign object shadow 4A of the first captured image G1 displayed on the monitor 31 with the mouse 34, the range determination unit 61 centers the click position with the mouse 34. Thus, a detection range E1 having a predetermined size including the foreign object shadow and its periphery is determined.
Here, the detection range E1 may be defined in advance, or the user may be able to set an arbitrary size by moving the mouse 34.

In step S 04, the image selection unit 62 selects a second image that has the same or substantially the same image capturing condition as the first captured image G 1 among the plurality of second captured images stored in the database 36 of the storage unit 35. Search and select image G2.

In this case, the determination of the identity of the shooting conditions is performed based on shooting information represented by the Exif information of the first shot image G1 and the second shot image G2.
Specifically, the criterion is that the camera body (camera type) 2, aperture value (F value), focal length, shooting time, lens, shooting distance, etc. in the shooting information such as Exif information are the same.
Among these photographing conditions, it is essential in the present embodiment that the camera body 2 is the same. In the present embodiment, when the same camera body 2 is used for shooting, even if the captured images are different, the foreign matter attached to the optical filter 12 disposed in the camera body 2 moves to the same position as a shadow. This is because the foreign object shadow is corrected by using the.

At least one of the other shooting conditions, in particular, the aperture values preferably match, and the shooting distance, the focal length, the shooting time, and the like are more preferably the same. More preferably, all conditions are met.
One or more or a plurality of second captured images G2 selected by the image selection unit 62 are selected according to the selection condition. If no image is selected, a warning message such as “zero selected number” may be displayed on the monitor 31.
7A and 7B show a case where a plurality of second photographed images G2 are selected by the image selection unit 62, and are diagrams showing the plurality of images G2 (1) and G2 (2), respectively.

Next, in step S05, the foreign object shadow information acquisition unit 63 specifies a corresponding range E2 corresponding to a predetermined detection range E1 in the first captured image G1 among the plurality of selected second captured images G2. .

In step S06, the connected region determination unit 64 obtains a luminance plane from the image data of each RGB pixel that forms an image in the corresponding range E2, applies inverse gamma correction to the luminance plane, and linearizes the input / output relationship. return. Then, a luminance gradient plane is generated from the luminance plane after inverse gamma correction.
In the luminance gradient plane, a high pixel value is shown at the peripheral part of the foreign object shadow, and a low pixel value is shown in a uniform plane such as the sky. That is, a high pixel value is shown in the edge portion, and thereby the edge portion of the foreign object shadow can be detected.

Next, the connected area determination unit 64 sets a predetermined threshold value and converts the luminance gradient plane into a binary image. That is, a pixel having a luminance gradient value higher than the threshold is replaced with “1”, and a pixel having a brightness gradient value lower than the threshold is replaced with “0”.
In step S07, the connected area determination unit 64 extracts an area (connected pixel area) in which “1” pixels are connected from the binary image thus obtained. The connected pixel region is, for example, a pixel region that is continuous (near 4) on at least one of four sides of a rectangular pixel. This connected pixel region is a region that may have a foreign object shadow.

In step S08, the target region extraction unit 65 extracts a correction candidate region from the connected pixel regions extracted by the connected region determination unit 64 based on the size (large or small) of the foreign object shadow. The determination of the size of the foreign object shadow in the target area extraction unit 65 is defined by the number of pixels.
Here, the target area extraction unit 65 uses the absolute size of the foreign object shadow as a reference when extracting the correction candidate area from the connected pixel area. Here, assuming that the size of a general foreign object and the size of a foreign object shadow are substantially equal, the detection target is a size of a minimum of 100 μm × 100 μm to a maximum of 400 μm × 400 μm.

The target area extraction unit 65 uses the area represented by the number of pixels of the foreign object shadow as a threshold, and the size (number of pixels) of the connected pixel area extracted from the binary image in the connected area determination unit 64 is the threshold. Those included are determined as foreign object shadow candidates that can be corrected, and are selected.

Further, in step S9, the luminance determining unit 66 uses the luminance value described in the previous connected region determining unit 64 and the luminance plane corresponding to the image plane to compare the internal and external luminances in the connected pixel region.
Here, the dust shadow has a lower internal luminance than the peripheral luminance, and the luminance ratio (internal luminance / peripheral luminance) is much smaller than one. Therefore, a predetermined threshold value with a small luminance ratio is set, and it is determined and selected as a foreign object shadow only when the luminance ratio of the connected image region is smaller than this threshold value.

In step S11, the foreign object shadow information acquisition unit 63 acquires a gain map and foreign object position information (coordinates) in the range E2 of the second captured image G2 having a shadow determined to be a foreign object shadow.

Here, when there are a plurality of images detected that there are foreign object shadows, the standard deviation of the luminance in the range E2 in each of the plurality of second captured images G2 is calculated. Then, a photographed image in which the variance of the standard deviation of the range E2 is equal to or less than a predetermined threshold is selected from the second photographed image G2.

The variance is equal to or less than the predetermined threshold value because the range E2 is a flat image, that is, an image with little variation in luminance, and foreign object shadows can be easily determined.
In this embodiment, it is used that the variance is equal to or less than a predetermined value, but it may be used that the image has few edges and the histogram is uniform.

In this embodiment, the fact that the luminance dispersion is small is used for narrowing down the number of images, but the present invention is not limited to this. For example, the gain map and foreign object position information (coordinates) may be obtained by averaging the gain maps of a plurality of images without narrowing down the number of images.
Moreover, you may select the image with the nearest brightness | luminance with the detection range E of the 1st picked-up image G1.
Furthermore, you may select the image with the closest hue with the detection range E of the 1st picked-up image G1.
Alternatively, an image with the closest shooting condition may be selected.
Further, a shutter with a different shutter speed may be selected. Different shutter speeds are more likely to be in different scenes and more likely to be shooting different subjects, so foreign objects attached to both are more likely to be shadows of foreign objects. It is.
Further, the second captured image G2 may be selected by weighting the above-described conditions (luminance, hue, standard deviation) and making a comprehensive determination.

Next, in step S11, the correction unit 56 uses the gain map and foreign object position information (coordinates) as foreign object shadow information reflected in the second captured image G2 selected as described above. Correction is performed to correct or reduce the luminance of the foreign object shadow by correcting the captured image.

In other words, the correction unit 56 uses the gain map acquired from the predetermined detection range E in the second captured image G2 for the predetermined detection range E in the first captured image G1, and uses the gain map acquired from the first captured image G1. Gain correction is performed by multiplying the luminance of the predetermined detection range E by the reciprocal of the luminance signal. Thus, image correction is performed to reduce or eliminate a change in luminance due to a foreign object such as a foreign object shadow appearing in a predetermined detection range E in the first captured image G1.

Here, it is also possible to display the first photographed image G1 subjected to the image correction again on the monitor 31, and allow the user to select whether or not to perform foreign object shadow information detection and image correction processing again. is there.

As described above, the image processing apparatus 50 according to the present embodiment includes a selection unit (mouse) 34 that can arbitrarily select a specific position where a foreign object such as a foreign object is reflected in a captured image, and a fixed value including the selected position. A range determination unit 61 that determines a range and an image selection unit 62 that selects a second captured image different from the captured image are provided.
This eliminates the troublesome trouble of shooting a reference image in which foreign objects such as a foreign object are reflected before normal shooting. For example, the first captured image to be detected displayed on the monitor 31 is displayed. The foreign object detection function can be executed by simply performing a simple operation of clicking (selecting) with a mouse 34 a specific position where a foreign object shadow is reflected.

The foreign object shadow information acquisition unit 63 in the image processing device 50 includes the second captured image from the range E determined by the range determination unit 61 in the second captured image close to the imaging condition selected by the image selection unit 62. Acquire foreign object shadow information reflected inside.
Thereby, the foreign object shadow information which becomes a defect in the first captured image can be detected with high accuracy. Further, it is possible to appropriately correct a defect in the captured image based on the detected foreign object shadow information.

(Second Embodiment)
Next, a description will be given of a second embodiment for correcting foreign matter in the first photographed image.
In the second embodiment, the foreign object shadow information in the second image selected based on the selected first captured image is acquired, and the first captured image is corrected based on the detected foreign object shadow information ( Foreign object shadow information acquisition / correction processing) is performed.

As in the first embodiment, an application program for foreign object shadow information acquisition and image correction processing is installed in the PC 30. By executing this program, the PC 30 executes foreign object shadow information acquisition and image correction processing. It functions as the image processing apparatus 150 to perform.

FIG. 8 is a functional block diagram of the image processing apparatus 150 that performs foreign object shadow information acquisition / correction processing. FIG. 9 is a functional block diagram of the foreign object shadow information acquisition unit 163 in the image processing apparatus 150.

The image processing apparatus 150 displays a storage area 151 for storing image data provided via the memory card 30 or transmitted from the external interface 18 of the camera 1, and a captured image stored in the storage area 151. A monitor 152 and a display control unit 153 for controlling the display are provided.

Further, the image processing apparatus 150 selects a foreign object shadow candidate from the captured image data stored in the storage area 151, and acquires a foreign object shadow candidate coordinate and a luminance decrease amount of the foreign object shadow candidate part. 163, a first captured image selection unit 154 that can select an image to be corrected via the mouse 34 and the like, and a second captured image is selected from the storage area 151 based on the selected first captured image. A second captured image selection unit 155, and a correction unit 156 that corrects a corresponding portion in the first captured image based on the foreign object shadow information detected by the foreign object shadow information acquisition unit 163 in the second captured image. ing.

As shown in FIG. 9, the foreign object shadow information acquisition unit 163 includes a connected region determination unit 164, a target region extraction unit 165, and a luminance determination unit 166. The foreign object shadow information acquisition unit 163 is the same as the foreign object shadow information acquisition unit 63 of the first embodiment.

Next, foreign object shadow information acquisition processing and correction processing will be described.
(Acquire foreign object shadow information)
A captured image is input to the storage area 151 of the image processing apparatus 150 via the memory card 40 or the external interface 18 of the camera 1. Then, the foreign object shadow information acquisition unit 163 of the image processing apparatus 150 acquires foreign object shadow information from the captured image input to the storage area 151.
Here, the foreign object shadow information includes the coordinates of the position where the foreign object shadow exists, the brightness reduction amount (gain map) of the part of the foreign object shadow, and the like.
This foreign object shadow information acquisition may be automatically performed when the image is transferred to the storage area, or may be started by the operation of the operator.

FIG. 10 is a flowchart for explaining the flow of foreign object shadow information acquisition in the foreign object shadow information acquisition unit 163. In FIG. 10 and the following description, the step is also abbreviated as “S”.
The acquisition of foreign object shadow information in the following flowchart is the same as steps S06 to S10 in the first embodiment.
First, in step S51, the connected region determination unit 164 obtains a luminance plane of image data of each pixel by RGB forming an image, performs inverse gamma correction on the brightness plane, and returns the input / output relationship to linear. Then, a luminance gradient plane is generated from the luminance plane after inverse gamma correction.
In the luminance gradient plane, a high pixel value is shown at the peripheral part of the foreign object shadow, and a low pixel value is shown in a uniform plane such as the sky. That is, a high pixel value is shown in the edge portion, and thereby the edge portion of the foreign object shadow can be detected.

Next, the connected area determination unit 164 sets a predetermined threshold value and converts the luminance gradient plane into a binary image. That is, a pixel having a luminance gradient value higher than the threshold is replaced with “1”, and a pixel having a luminance gradient value lower than the threshold is replaced with “0”.
In step S52, the connected region determination unit 164 extracts a region (connected pixel region) in which the pixels “1” are connected from the binary image thus obtained. The connected pixel region is, for example, a pixel region that is continuous (near 4) on at least one of four sides of a rectangular pixel. This connected pixel region is a region that may have a foreign object shadow.

In step S 53, the target area extraction unit 165 extracts a correction candidate area from the connected pixel areas extracted by the connected area determination unit 164 based on the size (large or small) of the foreign object shadow. The determination of the size of the foreign object shadow in the target area extraction unit 165 is defined by the number of pixels.
Here, the target area extraction unit 165 uses the absolute size of the foreign object shadow as a reference when extracting the correction candidate area from the connected pixel area. Here, assuming that the size of a general foreign object and the size of a foreign object shadow are substantially equal, the detection target is a size of a minimum of 100 μm × 100 μm to a maximum of 400 μm × 400 μm.

The target area extraction unit 165 uses the area represented by the number of pixels of the foreign object shadow as a threshold, and the size (number of pixels) of the connected pixel area extracted from the binary image by the connected area determination unit 164 is the threshold. Select what is contained within.

In step S54, the luminance determination unit 166 uses the luminance value V (Value) obtained by the HSV conversion described in the previous connected region determination unit 164 and the luminance plane corresponding to the image plane, and uses the luminance value V (Value) in the connected pixel region. Compare brightness.
Here, the foreign object shadow has a lower internal luminance than the peripheral luminance, and the luminance ratio (internal luminance / peripheral luminance) is much smaller than one. For this reason, a predetermined threshold value with a small luminance ratio is set, and it is determined as a foreign object shadow only when the luminance ratio of the connected image region is smaller than this threshold value.

Here, when the standard deviation of the brightness in a certain range around the foreign object shadow is equal to or greater than the threshold, it is not adopted as the foreign object shadow. This is because when the standard deviation of the brightness around the detected foreign object shadow is large, the reliability of the actual foreign object shadow is low due to the shadow on the texture.

In step S55, the foreign object shadow information acquisition unit 163 acquires the brightness decrease amount of the foreign object shadow and the foreign object position information (coordinates), and stores them in the storage area 151 in association with each captured image.

(Select correction target image)
FIG. 11 is a flowchart showing the foreign object shadow removal operation.
In step S61, the display control unit 153 of the PC causes the monitor 152 to display the captured image stored in the storage area 151 based on the operation of the PC operation unit 34 by the operator.
At this time, the PC control unit may display the captured image in the form of a thumbnail image reduced to display a list of a large number of images, or may sequentially select one image.

In step S 62, the first image selection unit 154 selects the first captured image G 1 for which the removal of the foreign object shadow is desired from the captured images displayed on the monitor 152 through the operation of the PC operation unit by the operator. select.
FIG. 6 shows a first captured image G1 as an example of the selected image. In the first photographed image G1 in FIG. 6, a foreign object shadow 4A is present in a portion of trees. Such a foreign object shadow is caused by, for example, a foreign substance (foreign substance 4) (see FIG. 1) attached to the front surface of the optical filter 12. However, in the first captured image G1 of the present embodiment, the brightness gradient of the foreign object shadow 4A and the brightness gradient of the trees overlap, and it is difficult to detect the foreign object shadow by the method of detecting the foreign object shadow by the brightness gradient. It has become.

(Second image selection)
In step S63, when the first photographed image G1 is selected, the second photographed image selection unit 155 has the same photographing condition as that of the first photographed image G1 among the plurality of photographed images stored in the storage area. A second photographed image G2 that is substantially the same is searched and selected.

In this case, the determination of the identity of the imaging conditions is performed based on the Exif information of the first captured image G1 and the Exif information of the second captured image G2.
Specifically, the criterion is that the camera body (camera type) 2, aperture value (F value), focal length, shooting time, lens, shooting distance, and the like in the Exif information are the same.
Among these photographing conditions, it is essential in the present embodiment that the camera body 2 is the same. In the present embodiment, when the same camera body 2 is used for shooting, even if the captured images are different, the foreign matter attached to the optical filter 12 disposed in the camera body 2 moves to the same position as a shadow. This is because the foreign object shadow is corrected by using the.

At least one or more other imaging conditions, in particular, it is preferable that the aperture values coincide, and it is more preferable that the focal length, the imaging time, and the like are the same. More preferably, all conditions are met.
One or more or a plurality of second photographed images G2 selected by the second photographed image selection unit 152 are selected according to the selection condition. If no sheet is selected, a warning display such as the number of selected sheets may be displayed on the monitor 152.
FIGS. 7A and 7B show two images G2 (1) and G2 (2), respectively, when a plurality of second photographed images G2 are selected by the second photographed image selection unit 152 in the second embodiment.
Here, regarding G2 (1), there is a foreign object shadow 4Aa in the empty part, and since it is far from the edge of the mountain, the detection of the foreign object shadow is easy. The image G2 (2) is also detectable because there is a foreign object shadow 4Ab in the sky (the color is dark but detectable).

(Acquisition of foreign object shadow information applied to the first captured image from the second image)
In step S64, when the second captured image G2 is selected, the image processing apparatus 150 reads out the foreign object shadow information (the coordinates of the position of the foreign object shadow and the brightness reduction amount (gain map)) of the second captured image.

(display)
In step S65, the position of the foreign object shadow detected in each of the second captured images G2 is displayed as a foreign object shadow position in the first captured image so as to overlap the monitor screen on which the first captured image is displayed.
That is, the foreign object shadow A1 is detected in the first second captured image, the A2 foreign object shadow is detected in the same position in the second second captured image, and the B foreign object shadow is detected in a different position. When a foreign object shadow C is detected at a position different from A3 and A1 and B in the same position as A1 in the image, the positions A, B, and C are displayed in the first image on the monitor screen.

(correction)
In step S66, the operator selects a portion desired to be corrected among the positions A, B, and C displayed on the monitor screen via the operation unit. When the processing start is selected, the selected portion is corrected.
Here, when the first captured image is selected without performing display or selection, correction may be automatically performed from the selected second image.

Here, when a foreign object shadow is detected at the same position of a plurality of second captured images, 1) the brightness of a certain range including the foreign object shadow in each of the plurality of second captured images is represented by the coordinates in the first captured image. Compare with a certain range of brightness including the part corresponding to the position. Then, the foreign object shadow of the second captured image having the brightness closest to the brightness around the corresponding portion in the first captured image is adopted as the foreign object shadow. This is because if the brightness around the foreign object shadow is close, the amount of gain reduction caused by the foreign object shadow is considered to be equal.

2) Further, instead of 1), a certain range of hues including the foreign object shadow in each of the plurality of second captured images is replaced with a certain range of hues including a portion corresponding to the position of the coordinates in the first captured image. Compare. Then, the foreign object shadow of the second captured image having the hue closest to the hue around the corresponding portion in the first captured image may be adopted as the foreign object shadow. This is also because if the hue around the foreign object shadow is close, the amount of gain decrease due to the foreign object shadow is considered to be equal.
3) Further, instead of 1) and 2), both of them may be referred to.
4) Furthermore, you may average the luminance fall amount of the same position in a some 2nd picked-up image.

The correction unit 156 corrects the first captured image by using the gain map and the foreign object position information (coordinates) as the foreign object shadow information reflected in the second captured image G2 selected as described above, thereby correcting the foreign object shadow. Perform correction to reduce or eliminate brightness.

That is, the correction unit 156 performs gain correction by multiplying the luminance of the first captured image G1 by the reciprocal of the luminance signal using the gain map acquired from the second captured image G2 in the first captured image G1. . Thus, image correction is performed to reduce or eliminate a change in luminance due to a foreign object such as a foreign object shadow appearing in the first captured image G1.

As described above, according to the second embodiment, when a foreign object shadow can be detected in the second captured image at a position where it is difficult to specify the foreign object shadow in the first captured image, the foreign object shadow is also positioned at the same position in the first captured image. And the first captured image can be corrected based on the foreign object shadow information of the second captured image. Thus, by using the detection results of the foreign object shadows of a plurality of captured images, it is possible to correct the foreign object shadows in an image in which the foreign object shadows are difficult to detect.

(Deformation)
The present invention is not limited to the above-described embodiment, and various modifications and changes as described below are possible, and these are also within the scope of the present invention.

(1) In the second embodiment, a description has been given of a mode in which foreign object shadow information acquisition is performed for all input captured images when a captured image is input to the storage area 51. However, the present invention is not limited to this. For example, after the first captured image is selected and the second captured image is selected based on the first captured image, foreign object shadow information may be acquired for the selected second captured image.
According to this modification, the foreign object shadow information is acquired only for the selected second photographed image, so the overall processing amount is reduced. On the other hand, in the case of the present embodiment, since the foreign object shadow information is acquired in advance for all the captured images, the processing time after the second captured image is selected is reduced.

(2) In the above embodiment, an example in which image data captured by the camera 1 which is an electronic still camera is processed has been described. However, the present invention is not necessarily limited to this content. The present invention can also be applied to image data taken by a video camera that handles moving images. It can also be applied to image data taken with a camera-equipped mobile phone. Furthermore, it can be applied to a copying machine, a scanner, and the like. That is, the present invention can be applied to any image data captured using an image sensor.

(3) In the above-described embodiment, the example in which the image data captured by the camera 1 which is an electronic still camera is processed by the PC (personal computer) 30 to remove the influence of the foreign matter has been described. However, the present invention is not limited to this content. Absent. The camera 1 may be provided with such a program. In that case, the defect position on the liquid crystal screen of the camera 1 may be specified by a touch panel method, a cross key provided on the back of the camera, or the like.
Further, such a program may be provided in a printer, a projection device, or the like. That is, the present invention can be applied to any apparatus that handles image data.

(4) The program executed by the PC 30 is not limited to a recording medium such as a CD-ROM, and can be provided through a data signal such as the Internet.
In addition, although embodiment and a deformation | transformation form can also be used in combination suitably, detailed description is abbreviate | omitted. Further, the present invention is not limited to the embodiment described above.

1: Camera, 2: Camera body, 12: Optical filter, 13: Image sensor, 17: Memory card IF, 18: External IF, 30: Personal computer (PC), 33: CD-ROM, 34: Mouse (selection) Part), 36: memory card, 50: image processing device, 56: correction part, 60: foreign object shadow information detection part, 61: range determination part, 62: image selection part, 63: foreign object shadow information acquisition part 151: storage area 152: monitor, 153: constant control unit, 154: first captured image selection unit, 155: second captured image selection unit, 156: correction unit, 163: foreign object shadow information acquisition unit, G1: first captured image, G2: second captured image, E: detection range

Claims

A position selection unit capable of selecting a specific position in the first captured image;
In the first captured image, a range determination unit that determines a certain range including the position selected by the position selection unit;
An image selection unit for selecting a second captured image different from the first captured image;
A foreign object shadow information acquisition unit that acquires foreign object shadow information that has moved into the second captured image from a corresponding range corresponding to the certain range of the first captured image in the second captured image; Providing,
An image processing apparatus.
The image processing apparatus according to claim 1,
The image selection unit selects, as the second photographed image, an image photographed under a photographing condition close to the photographing condition of the first photographed image;
An image processing apparatus.
The image processing apparatus according to claim 2,
The shooting condition is that the camera body is the same as the camera body that shot the first captured image, and at least one of an aperture value, a shooting distance, a lens, a focal length, and a shooting time is the same. Including,
An image processing apparatus.
The image processing apparatus according to any one of claims 1 to 3,
The foreign object shadow information acquisition unit acquires the foreign object shadow information using luminance information;
An image processing apparatus.
The image processing apparatus according to any one of claims 1 to 4,
The image selection unit has the foreign object shadow information in the corresponding range when the second captured image has the same file type as the first captured image, and the peripheral luminance of the corresponding range. Select an image that is the same as or closest to the first captured image,
An image processing apparatus.
The image processing apparatus according to any one of claims 1 to 4,
The image selection unit has the foreign object shadow information in the corresponding range when the second captured image has the same file type as the first captured image, and the hue around the range is Select the same or closest image as the first captured elephant image,
An image processing apparatus.
The image processing apparatus according to any one of claims 1 to 6,
The image selection unit selects, as the second captured image, an image having the foreign object shadow information in the corresponding range and having the smallest standard deviation of the peripheral luminance of the corresponding range;
An image processing apparatus.
The image processing apparatus according to any one of claims 1 to 7,
The image selection unit creates data for correcting the first captured image from the average brightness profile of the corresponding range in the second captured image;
An image processing apparatus.
The image processing apparatus according to any one of claims 1 to 8,
The image selection unit selects a photographed image other than the first photographed image with a different shutter speed as the second photographed image;
An image processing apparatus.
The image processing apparatus according to any one of claims 1 to 9,
Providing an image correction unit;
An image processing apparatus.
A second captured image selection unit that selects a second captured image for acquiring foreign object shadow information from a stored captured image based on the capturing condition of the first captured image to be corrected;
A foreign object shadow candidate that may be a foreign object shadow is extracted from the shadows reflected in the captured image, and the coordinates of the position where the foreign object shadow candidate exists and the foreign object shadow candidate are extracted for each captured image. A foreign object shadow information acquisition unit for acquiring the luminance decrease amount of the part;
The first captured image is corrected based on the coordinates of the position where the foreign object shadow candidate is present in the second captured image acquired by the foreign object shadow information acquisition unit and the luminance decrease amount of the portion of the foreign object shadow candidate. A correction unit that performs,
An image processing apparatus.
A second captured image selection unit that selects a second captured image for acquiring foreign object shadow information from a stored captured image based on the capturing condition of the first captured image to be corrected;
A foreign object shadow candidate that may be a foreign object shadow is extracted from the shadows reflected in the captured image, and the coordinates of the position where the foreign object shadow candidate exists and the foreign object shadow candidate are extracted for each captured image. A foreign object shadow information acquisition unit for acquiring the luminance decrease amount of the part;
A display unit that employs the foreign object shadow candidate as a foreign object shadow and displays the coordinates of the foreign object shadow candidate acquired by the foreign object shadow information acquisition unit on the first captured image;
An image processing apparatus.
The image processing device according to claim 11 or 12,
The second captured image selection unit selects, as the second captured image, an image captured under a capturing condition close to the capturing condition of the first captured image;
An image processing apparatus.
The image processing apparatus according to claim 13,
The imaging condition is that the camera body is the same as the camera body that captured the first captured image, and that at least one of an aperture value, an imaging distance, a lens, a focal length, and an imaging time is the same, Including,
An image processing apparatus.
The image processing apparatus according to any one of claims 11 to 14,
If the standard deviation of the peripheral brightness of the foreign object shadow candidate in the second photographed image is equal to or greater than a threshold value, do not adopt the foreign object shadow candidate as the foreign object shadow;
An image processing apparatus.
The image processing apparatus according to any one of claims 11 to 15,
When the standard deviation of a certain range including the position of the coordinates corresponding to the foreign object shadow candidate in the first photographed image is equal to or less than a threshold, do not adopt the foreign object shadow candidate as the foreign object shadow;
An image processing apparatus.
The image processing apparatus according to any one of claims 11 to 16,
When there are a plurality of second photographed images selected by the second photographed image selection unit and a foreign object shadow candidate present at the same coordinate position is selected from the plurality of second photographed images,
The foreign object shadow candidate of the second captured image that is closest to the predetermined range of brightness including the foreign object shadow candidate and the predetermined range of brightness including the portion corresponding to the position of the coordinate in the first captured image is defined as a foreign object shadow. Adopting,
An image processing apparatus.
The image processing apparatus according to any one of claims 11 to 16,
When there are a plurality of second photographed images selected by the second photographed image selection unit and a foreign object shadow candidate present at the same coordinate position is selected from the plurality of second photographed images,
The foreign object shadow candidate of the second captured image that is closest to the fixed range of hue including the foreign object shadow candidate and the fixed range of hue including the portion corresponding to the position of the coordinate in the first captured image is defined as a foreign object shadow. Adopting,
An image processing apparatus.
A selection unit that selects the second photographed image using the photographing information of the first photographed image to be corrected;
An acquisition unit that acquires foreign object information that is information related to a foreign object shadow from the second captured image;
An image processing apparatus comprising:
A position selection unit capable of selecting a specific position in the first captured image;
A range determination unit for determining a certain range including the position selected by the position selection unit;
An image selection unit for selecting a second captured image different from the first captured image;
An image processing apparatus comprising: a foreign object shadow information acquisition unit configured to acquire foreign object shadow information from a corresponding range corresponding to the certain range in the second captured image.
Causing a computer to execute the function of the image processing apparatus according to claim 1;
An image processing program characterized by the above.