JP2012222509A - Image processor and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor and image processing program capable of easily detecting a foreign matter in a captured image.SOLUTION: An image processor includes: a position selection part 34 for selecting a specification position in a first captured image; a range determination part 61 for determining a fixed range including the position selected by the position selection part 34 in the first captured image; an image selection part 62 for selecting a second captured image different from the first captured image; and a foreign matter shadow information acquisition part 63 for acquiring foreign matter shadow information appearing in the second captured image from a correspondence range corresponding to the fixed range of the first captured image in the second captured image.

Description

  The present invention relates to an image processing apparatus and an image processing program.

When capturing an image with an electronic camera or the like, if a foreign object adheres to the optical filter or the like, a shadow of the foreign object may appear in the captured image. As described above, information such as the coordinates of the position of the foreign object was obtained from the reference image in which the foreign object shadow was reflected, and the image was normally shot based on the information. There is known an image processing apparatus that corrects an image defect caused by a foreign object shadow reflected in a captured image.
Further, a technique is known in which a foreign object is detected from a change in transmittance (change in luminance information in each pixel) due to a foreign object shadow in an image to be corrected, and correction for deleting the foreign object shadow is performed (see Patent Document 1).

JP 2004-220553 A

  However, in the above prior art, it is necessary to take a reference image in which foreign matter is reflected. In addition, the coordinates of the foreign substance position may differ between the captured image and the reference image.

  The subject of this invention is providing the image processing apparatus and image processing program which can detect the foreign material in a picked-up image more easily.

  The present invention solves the above problems by the following means. In addition, in order to make an understanding easy, although the code | symbol corresponding to embodiment of this invention is attached | subjected and demonstrated, it is not limited to this.

According to the first aspect of the present invention, the position selection unit (34) capable of selecting a specific position in the first captured image (G1) and the position selection unit (in the first captured image (G1)) 34) A range determination unit (61) for determining a certain range (E1) including the position selected by the image selection, and an image selection for selecting a second captured image (G2) different from the first captured image (G1). And the second captured image from the corresponding range (E2) corresponding to the predetermined range (E1) of the first captured image (G1) in the second captured image (G2). An image processing apparatus (50) comprising: a foreign object shadow information acquisition unit (63) that acquires foreign object shadow information transferred into the image (G2).
A second aspect of the present invention is the image processing device (50) according to the first aspect, wherein the image selection unit (62) uses the first photographing as the second photographed image (G2). An image processing apparatus (50) characterized by selecting an image photographed under a photographing condition close to the photographing condition of the image (G1).
The invention according to claim 3 is the image processing apparatus (50) according to claim 2, wherein the photographing condition is the same camera body as the camera body that photographed the first photographed image (G1). And an image processing apparatus (50) characterized in that at least one of an aperture value, a shooting distance, a lens, a focal length, and a shooting time is the same.
The invention described in claim 4 is the image processing apparatus (50) according to any one of claims 1 to 3, wherein the foreign object shadow information acquisition unit (63) uses the luminance information to detect the foreign object shadow. An image processing apparatus (50) characterized by acquiring information.
Invention of Claim 5 is an image processing apparatus (50) of any one of Claim 1 to 4, Comprising: The said image selection part (62) is said 2nd picked-up image (G2). When the same file type as the first captured image (G1) is used, the foreign object shadow information is included in the corresponding range (E2), and the luminance around the corresponding range (E2) is the first. The image processing apparatus (50) is characterized by selecting an image that is the same as or closest to the captured image (G1).
A sixth aspect of the present invention is the image processing device (50) according to any one of the first to fourth aspects, wherein the image selection unit (62) includes the second captured image (G2). When the same file type as the first photographed image (G1) is used, the foreign object shadow information is included in the corresponding range (E2), and the hue around the range is the first photographed elephant image. The image processing apparatus (50) is characterized in that the same or closest image is selected.
A seventh aspect of the present invention is the image processing device (50) according to any one of the first to sixth aspects, wherein the image selection unit (62) includes the second captured image (G2). As an image processing apparatus (50), an image having the foreign object shadow information in the corresponding range (E2) and having the smallest standard deviation of luminance around the corresponding range (E2) is selected. ).
The invention according to claim 8 is the image processing device (50) according to any one of claims 1 to 7, wherein the image selection unit (62) is configured to provide the second captured image (G2). The image processing device (50) is characterized in that data for correcting the first photographed image (G1) is created from the averaged luminance profile of the corresponding range (E2).
The invention according to a ninth aspect is the image processing device (50) according to any one of the first to eighth aspects, wherein the image selection unit (62) includes the first captured image (G1). The image processing apparatus (50) is characterized in that one having a different shutter speed is selected as the second captured image (G2) from the other captured images.
A tenth aspect of the present invention is the image processing apparatus (50) according to any one of the first to ninth aspects, further comprising an image correcting unit (51). (50).
The invention described in claim 11 is an image processing program that causes a computer (30) to execute the function of the image processing apparatus (50) described in any one of claims 1 to 10.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus and image processing program which can detect the foreign material in a picked-up image more easily can be provided.

It is a figure which shows the structure of the camera of an interchangeable lens type in embodiment of this invention. 1 is a block diagram illustrating a personal computer and peripheral devices that constitute a camera and an image processing apparatus. It is a functional block diagram of the image processing apparatus which performs an image defect correction process. It is a functional block diagram of a defect determination unit in the image processing apparatus. 6 is a flowchart illustrating a flow of foreign object shadow information detection and image correction processing by the image processing apparatus. It is an example of the 1st picked-up image displayed on the monitor. Both (a) and (b) are examples of the second photographed image.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a diagram illustrating a configuration of a lens-interchangeable camera 1. FIG. 2 is a block diagram showing a personal computer (PC) 30 and peripheral devices constituting the camera 1 and the image processing apparatus.
The camera 1 is an electronic still camera that outputs image data obtained by converting a subject image into an electrical signal. In the present embodiment, image data captured by the camera 1 is acquired by a PC 30 functioning as an image processing apparatus, and foreign object shadow detection and correction processing described later is performed.

  The camera 1 includes a camera body 2 and an imaging lens 3 that can be attached to and detached from the camera body 2. The imaging lens 3 includes an imaging optical system 3L having a plurality of optical lens groups (shown as one lens in FIG. 1).

  The camera body 2 includes a shutter 11, an optical filter 12, an image sensor 13, an image processing unit 14, an operation unit 15, a display unit 16, a memory card interface (hereinafter referred to as “memory card IF”) 17, an external interface (hereinafter referred to as “memory card IF”). , “External IF”) 18, a power source 19, a control unit 20, and the like.

The shutter 11 adjusts the exposure time by shielding and passing photographing light from the imaging optical system 3L toward the image sensor 13.
The optical filter 12 is configured by an optical low-pass filter or the like that prevents generation of false colors (color moire) or the like during imaging. Note that reference numeral 4 in FIG. 1 indicates foreign matters such as dust and dirt attached to the surface of the optical filter 12. If the foreign matter 4 is attached to the optical filter 12 that is on the optical path of the photographing light and transmits the photographing light, pixels that are affected by the foreign matter shadow in the image data acquired by the image sensor 13. May be included.

  The imaging element 13 is a photoelectric conversion element such as a CCD or a CMOS, and has a rectangular imaging area composed of a plurality of pixels. The image sensor 13 converts the image formed on the image plane into an electrical signal and outputs an image signal (imaging signal) corresponding to the subject image. The image signal output from the image sensor 13 is A / D converted and input to the image processing unit 14 after predetermined analog signal processing.

The image processing unit 14 performs image processing such as interpolation, gradation conversion, and edge enhancement on the image data input from the image sensor 13.
The operation unit 15 includes a release button for inputting a signal for determining a shooting timing by the photographer, a selection button for mode switching, and the like. And the operation part 15 inputs those operation information into a control part.

  The display unit 16 is configured by an LCD or the like, and displays various setting menus in the camera 1, reproduced images based on images captured by the image sensor 13 and image data stored in a memory card.

The memory card IF 17 serves as an interface with a memory card (card-like removable memory) 40.
The external IF 18 serves as an interface with an external device such as the PC 30 via a predetermined cable or a wireless transmission path.
The power source 19 supplies power to each functional unit of the camera 1.

  The control unit 20 performs overall control of each functional unit in the camera 1. That is, when a shooting instruction signal is input from the release switch in the operation unit 15, the control unit 20 controls the driving of the shutter 11 to control a shooting operation such as an exposure operation. Further, the control unit 20 stores the image data acquired by the image sensor 13 and subjected to image processing by the image processing unit 14 in a memory card 40 or the like that is detachably attached to the memory card IF 17. ,save. Further, the control unit 20 calls the image data stored in the memory card 40 or the like and causes the display unit 16 to display the image data.

The camera 1 configured as described above is controlled by the control unit 20 at the time of shooting and operates as follows.
When the release button of the operation unit 15 is pressed by the operator, the shutter 11 is opened for a predetermined time. The imaging device 13 generates an image signal corresponding to the optical image formed in the imaging region by the imaging optical system 3L. The image signal generated by the image sensor 13 is input as image data to the image processing unit 14 and is subjected to predetermined image processing by the image processing unit 14. Then, the captured image is displayed on the display unit 16 based on the image data processed by the image processing unit 14, and the image data is subjected to compression processing by JPEG, RAW, or the like as necessary, and is stored in the memory card IF 17. Store / store in the installed memory card 40. In addition, image data is sent to a PC 30 (described later) connected via the external IF 18.

  Here, the image data stored / saved in the memory card 40 or transmitted to the PC 30 via the external IF 18 is recorded in, for example, an Exif (EXchange Image File format) file. In the Exif file, additional information such as the camera body 2 type, camera manufacturer name, imaging pixel information, aperture value (F value) and exposure conditions such as focus, shutter speed and exposure correction, shooting time, and shooting distance are recorded together. Is done.

  Although not shown, the personal computer (PC) 30 serves as an interface with an external device such as the camera 1 via a memory card IF, a cable, or a wireless transmission path that serves as an interface with the CPU, memory, hard disk, and memory card 40. External IF etc. are provided. The PC 30 is connected to a monitor 31, a printer 32, and the like as an image display unit, and the user arbitrarily selects a specific position where a foreign object such as a foreign object shadow appears in a captured image displayed on the monitor 31. A mouse 34 is connected as a selection unit to be indicated.

  Further, an application program for correcting an image based on information on the detection of a foreign object such as a foreign object shadow and the detected foreign object is installed in the CD-ROM 33 connected to the PC 30 in advance.

  Next, processing for detecting foreign object shadow information such as foreign object shadow in the photographed first captured image and correcting the first captured image based on the detected foreign object shadow information (foreign object shadow information detection / correction processing) ).

The foreign object shadow information detection / correction processing is performed by the PC 30 on the image data after completion of image processing provided via the memory card 30 or transmitted from the external interface 18 of the camera 1.
As described above, the foreign object shadow information detection and image correction processing application program is installed in the PC 30, and the PC 30 executes the program to perform foreign object shadow information detection and image correction processing. It functions as the device 50.

FIG. 3 is a functional block diagram of the image processing apparatus 50 that performs foreign object shadow information detection / correction processing. FIG. 4 is a functional block diagram of the foreign object shadow detection unit 60 in the image processing apparatus 50.
As shown in FIG. 3, the image processing apparatus 50 functioning by the PC 30 and the program is based on the foreign object shadow detection unit 60 that detects foreign object shadow information and the foreign object shadow information detected by the foreign object shadow detection unit 60. And a correction unit 51 that corrects a foreign object shadow caused by the foreign object.
As shown in FIG. 4, the foreign object shadow detection unit 60 includes a range determination unit 61, an image selection unit 62, and a foreign object shadow information acquisition unit 63. The foreign object shadow information acquisition unit 63 further includes a connected region determination unit 64, a target region extraction unit 65, and a luminance determination unit 66.

  FIG. 5 is a flowchart for explaining the flow of foreign object shadow information detection and image correction processing in the image processing apparatus 50. The flow of foreign object shadow information detection and image correction processing in the image processing apparatus 50 will be described with reference to the flowchart of FIG. In FIG. 5 and the following description, the step is also abbreviated as “S”.

  First, in step S01, the PC 30 including the image processing device 50 reads data of the first (target) captured image G1 input via the memory card 40 or the external interface 18 of the camera 1.

  In step S02, the read first photographed image G1 is displayed on the monitor 31. FIG. 6 shows an example of the first captured image G1 displayed on the monitor 31. In the first photographed image G1 of FIG. 6, there is a foreign object shadow 4A in a portion surrounded by trees. Such a foreign object shadow is caused by, for example, a foreign substance (foreign substance 4) (see FIG. 1) attached to the front surface of the optical filter 12.

  Here, the user can recognize the foreign object shadow part, but the luminance gradient of the foreign object shadow 4A and the luminance gradient of the trees overlap, and it may be difficult to detect the foreign object shadow 4A information from the luminance gradient. is there.

Therefore, first, in step S03, when the user clicks the position of the foreign object shadow 4A of the first captured image G1 displayed on the monitor 31 with the mouse 34, the range determination unit 61 centers the click position with the mouse 34. Thus, a detection range E having a predetermined size including the foreign object shadow and its periphery is determined.
Here, the detection range E may be defined in advance, or the user may be able to set an arbitrary size by moving the mouse 34.

  In step S <b> 04, the image selection unit 62 selects a second image that has the same or substantially the same image capturing condition as the first image G <b> 1 among the plurality of second image captured in the database 36 of the storage unit 35. Search and select image G2.

In this case, the determination of the identity of the shooting conditions is performed based on shooting information represented by the Exif information of the first shot image G1 and the second shot image G2.
Specifically, the criterion is that the camera body (camera type) 2, aperture value (F value), focal length, shooting time, lens, shooting distance, etc. in the shooting information such as Exif information are the same.
Among these photographing conditions, it is essential in the present embodiment that the camera body 2 is the same. In the present embodiment, when the same camera body 2 is used for shooting, even if the captured images are different, the foreign matter attached to the optical filter 12 disposed in the camera body 2 moves to the same position as a shadow. This is because the foreign object shadow is corrected by using the.

At least one of the other shooting conditions, in particular, the aperture values preferably match, and the shooting distance, the focal length, the shooting time, and the like are more preferably the same. More preferably, all conditions are met.
One or more or a plurality of second captured images G2 selected by the image selection unit 62 are selected according to the selection condition. If no image is selected, a warning message such as “zero selected number” may be displayed on the monitor 31.
FIG. 7 is a diagram illustrating two images G2 (1) and G2 (2) when a plurality of second photographed images G2 are selected by the image selection unit 62.

  Next, in step S05, the foreign object shadow information acquisition unit 63 specifies a corresponding range E2 corresponding to a predetermined detection range E in the first captured image G1 among the plurality of selected second captured images G2. .

In step S06, the connected region determination unit 64 obtains a luminance plane from the image data of each RGB pixel that forms an image in the corresponding range E2, applies inverse gamma correction to the luminance plane, and linearizes the input / output relationship. return. Then, a luminance gradient plane is generated from the luminance plane after inverse gamma correction.
In the luminance gradient plane, a high pixel value is shown at the peripheral part of the foreign object shadow, and a low pixel value is shown in a uniform plane such as the sky. That is, a high pixel value is shown in the edge portion, and thereby the edge portion of the foreign object shadow can be detected.

Next, the connected area determination unit 64 sets a predetermined threshold value and converts the luminance gradient plane into a binary image. That is, a pixel having a luminance gradient value higher than the threshold is replaced with “1”, and a pixel having a brightness gradient value lower than the threshold is replaced with “0”.
In step S07, the connected area determination unit 64 extracts an area (connected pixel area) in which “1” pixels are connected from the binary image thus obtained. The connected pixel region is, for example, a pixel region that is continuous (near 4) on at least one of four sides of a rectangular pixel. This connected pixel region is a region that may have a foreign object shadow.

In step S08, the target region extraction unit 65 extracts a correction candidate region from the connected pixel regions extracted by the connected region determination unit 64 based on the size (large or small) of the foreign object shadow. The determination of the size of the foreign object shadow in the target area extraction unit 65 is defined by the number of pixels.
Here, the target area extraction unit 65 uses the absolute size of the foreign object shadow as a reference when extracting the correction candidate area from the connected pixel area. Here, assuming that the size of a general foreign object and the size of a foreign object shadow are substantially equal, the detection target is a size of a minimum of 100 μm × 100 μm to a maximum of 400 μm × 400 μm.

  The target area extraction unit 65 uses the area represented by the number of pixels of the foreign object shadow as a threshold, and the size (number of pixels) of the connected pixel area extracted from the binary image in the connected area determination unit 64 is the threshold. Those included are determined as foreign object shadow candidates that can be corrected, and are selected.

  Further, in step S9, the luminance determining unit 66 uses the luminance value described in the previous connected region determining unit 64 and the luminance plane corresponding to the image plane to compare the internal and external luminances in the connected pixel region. Here, the dust shadow has a lower internal luminance than the peripheral luminance, and the luminance ratio (internal luminance / peripheral luminance) is much smaller than one. Therefore, a predetermined threshold value with a small luminance ratio is set, and it is determined and selected as a foreign object shadow only when the luminance ratio of the connected image region is smaller than this threshold value.

  In step S11, the foreign object shadow information acquisition unit 63 acquires a gain map and foreign object position information (coordinates) in the range E2 of the second captured image G2 having a shadow determined to be a foreign object shadow.

  Here, when there are a plurality of images detected as having a foreign object shadow, the standard deviation of the luminance in the range E2 in each of the plurality of second captured images G2 is calculated. Then, a photographed image in which the variance of the standard deviation of the range E2 is equal to or less than a predetermined threshold is selected from the second photographed image G2.

The variance is equal to or less than the predetermined threshold value because the range E2 is a flat image, that is, an image with little variation in luminance, and foreign object shadows can be easily determined.
In this embodiment, it is used that the variance is equal to or less than a predetermined value, but it may be used that the image has few edges and the histogram is uniform.

In this embodiment, the fact that the luminance dispersion is small is used for narrowing down the number of images, but the present invention is not limited to this. For example, the gain map and foreign object position information (coordinates) may be obtained by averaging the gain maps of a plurality of images without narrowing down the number of images.
Moreover, you may select the image with the nearest brightness | luminance with the detection range E of the 1st picked-up image G1.
Furthermore, you may select the image with the closest hue with the detection range E of the 1st picked-up image G1.
Alternatively, an image with the closest shooting condition may be selected.
Further, a shutter with a different shutter speed may be selected. Different shutter speeds are more likely to be in different scenes and more likely to be shooting different subjects, so foreign objects attached to both are more likely to be shadows of foreign objects. It is.
Further, the second captured image G2 may be selected by weighting the above-described conditions (luminance, hue, standard deviation) and making a comprehensive determination.

  Next, in step S11, the correction unit 51 uses the gain map and foreign object position information (coordinates) as foreign object shadow information reflected in the second captured image G2 selected as described above. Correction is performed to correct or reduce the luminance of the foreign object shadow by correcting the captured image.

  That is, the correction unit 51 uses the gain map acquired from the predetermined detection range E in the second captured image G2 for the predetermined detection range E in the first captured image G1, and uses the gain map acquired from the first captured image G1. Gain correction is performed by multiplying the luminance of the predetermined detection range E by the reciprocal of the luminance signal. Thus, image correction is performed to reduce or eliminate a change in luminance due to a foreign object such as a foreign object shadow appearing in a predetermined detection range E in the first captured image G1.

  Here, it is also possible to display the first photographed image G1 subjected to the image correction again on the monitor 31, and allow the user to select whether or not to perform foreign object shadow information detection and image correction processing again. is there.

  As described above, the image processing apparatus 50 according to the present embodiment includes a selection unit (mouse) 34 that can arbitrarily select a specific position where a foreign object such as a foreign object is reflected in a captured image, and a fixed value including the selected position. A range determination unit 61 that determines a range and an image selection unit 62 that selects a second captured image different from the captured image are provided. This eliminates the troublesome trouble of shooting a reference image in which foreign objects such as a foreign object are reflected before normal shooting. For example, the first captured image to be detected displayed on the monitor 31 is displayed. The foreign object detection function can be executed by simply performing a simple operation of clicking (selecting) with a mouse 34 a specific position where a foreign object shadow is reflected.

  The foreign object shadow information acquisition unit 63 in the image processing device 50 includes the second captured image from the range E determined by the range determination unit 61 in the second captured image close to the imaging condition selected by the image selection unit 62. Acquire foreign object shadow information reflected inside. Thereby, the foreign object shadow information which becomes a defect in the first captured image can be detected with high accuracy. Further, it is possible to appropriately correct a defect in the captured image based on the detected foreign object shadow information.

(Deformation)
The present invention is not limited to the above-described embodiment, and various modifications and changes as described below are possible, and these are also within the scope of the present invention.

(1) In the above embodiment, an example in which image data captured by the camera 1 which is an electronic still camera is processed has been described. However, the present invention is not necessarily limited to this content. The present invention can also be applied to image data taken by a video camera that handles moving images. It can also be applied to image data taken with a camera-equipped mobile phone. Furthermore, it can be applied to a copying machine, a scanner, and the like. That is, the present invention can be applied to any image data captured using an image sensor.

(2) In the above-described embodiment, the example in which the image data captured by the camera 1 which is an electronic still camera is processed by the PC (personal computer) 30 to remove the influence of foreign matters has been described. However, the present invention is not limited to this content. Absent. The camera 1 may be provided with such a program. In that case, the defect position on the liquid crystal screen of the camera 1 may be specified by a touch panel method, a cross key provided on the back of the camera, or the like.
Further, such a program may be provided in a printer, a projection device, or the like. That is, the present invention can be applied to any apparatus that handles image data.

(3) The program executed on the PC 30 is not limited to a recording medium such as a CD-ROM, and can be provided through a data signal such as the Internet.
In addition, although embodiment and a deformation | transformation form can also be used in combination as appropriate, detailed description is abbreviate | omitted. Further, the present invention is not limited to the embodiment described above.

  1: Camera, 2: Camera body, 12: Optical filter, 13: Image sensor, 17: Memory card IF, 18: External IF, 30: Personal computer (PC), 33: CD-ROM, 34: Mouse (selection) Part), 36: memory card, 50: image processing device, 51: correction part, 60: foreign object shadow information detection part, 61: range determination part, 62: image selection part, 63: foreign object shadow information acquisition part, G1: first One shot image, G2: second shot image, E: detection range

Claims (11)

A position selection unit capable of selecting a specific position in the first captured image;
In the first captured image, a range determination unit that determines a certain range including the position selected by the position selection unit;
An image selection unit for selecting a second captured image different from the first captured image;
A foreign object shadow information acquisition unit that acquires foreign object shadow information that has moved into the second captured image from a corresponding range corresponding to the certain range of the first captured image in the second captured image; Providing,
An image processing apparatus. The image processing apparatus according to claim 1,
The image selection unit selects, as the second photographed image, an image photographed under a photographing condition close to the photographing condition of the first photographed image;
An image processing apparatus. The image processing apparatus according to claim 2,
The shooting condition is that the camera body is the same as the camera body that shot the first captured image, and at least one of an aperture value, a shooting distance, a lens, a focal length, and a shooting time is the same. Including,
An image processing apparatus. The image processing apparatus according to any one of claims 1 to 3,
The foreign object shadow information acquisition unit acquires the foreign object shadow information using luminance information;
An image processing apparatus. The image processing apparatus according to any one of claims 1 to 4,
The image selection unit has the foreign object shadow information in the corresponding range when the second captured image has the same file type as the first captured image, and brightness around the corresponding range. Select an image that is the same as or closest to the first captured image,
An image processing apparatus. The image processing apparatus according to any one of claims 1 to 4,
The image selection unit has the foreign object shadow information in the corresponding range when the second captured image has the same file type as the first captured image, and the hue around the range is Select the same or closest image as the first captured elephant image,
An image processing apparatus. The image processing apparatus according to any one of claims 1 to 6,
The image selection unit selects, as the second captured image, an image having the foreign object shadow information in the corresponding range and having the smallest standard deviation of luminance around the corresponding range;
An image processing apparatus. The image processing apparatus according to any one of claims 1 to 7,
The image selection unit creates data for correcting the first captured image from the average brightness profile of the corresponding range in the second captured image;
An image processing apparatus. The image processing apparatus according to any one of claims 1 to 8,
The image selection unit selects a photographed image other than the first photographed image with a different shutter speed as the second photographed image;
An image processing apparatus. The image processing apparatus according to any one of claims 1 to 9,
Providing an image correction unit;
An image processing apparatus. Causing a computer to execute the function of the image processing apparatus according to claim 1;
An image processing program characterized by the above.

Images