JP2012044542A - Image processing device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately remove a foreign matter from a cut-out image even when a part of a taken image is cut out.SOLUTION: An image processing device obtains foreign matter information including positional information Di of a foreign matter and size information Ri of the foreign matter which are extracted from a taken image for detecting a foreign matter, the image which is obtained by an imaging element 2, and obtains cut-out positional information T of a cut-out image 201 to be cut out from a taken image taken by the imaging element 2 in normal photographing. Then, on the basis of the positional information Di and the cut-out positional information T, it is determined whether the foreign matter is located within the cut-out image 201. Then, if it is determined that the foreign matter is not located within the cut-out image 201, positional information of the foreign matter is converted to that of the second foreign matter on the basis of the size information Ri of the foreign matter.

Description

  The present invention relates to a technique for suppressing image quality deterioration due to foreign matters adhering to the surface of an optical element such as an optical low-pass filter in an imaging apparatus using an imaging element such as a CCD or a CMOS sensor.

  In a digital camera capable of exchanging lenses such as a single lens reflex system, foreign matters such as dust are likely to be mixed in the optical system due to its structure. In particular, when a foreign object adheres to the surface of an optical element such as a low-pass filter disposed in front of the image sensor, there is a problem in that the shadow of the foreign object appears in the captured image and the quality of the image decreases.

  Therefore, there is a method to make the dust inconspicuous by taking a picture of a white wall etc. with the lens aperture closed and preparing an image showing only dust on the image sensor in combination with a normal shot image. It is known (see Patent Document 1).

  However, in this method, it is necessary to always be aware of the association between the image captured for dust detection and the actual captured image group associated therewith, which is troublesome.

  Therefore, after capturing the position of dust by photographing a white wall etc., hold it on the digital camera and attach a list of dust positions and sizes (dust information) to the image data at the time of normal shooting Can be considered. For example, using a separately prepared image processing device, the dust position on the captured image data is analyzed from the attached dust position, and the analyzed area is interpolated with surrounding pixels to make it inconspicuous ( Patent Document 2).

JP 2004-222231 A JP 2007-189467 A

  On the other hand, there is a function of recording an image with a changed aspect ratio or recording a zoomed up image of a subject by cutting out a part of the captured image at the time of shooting. In an image generated by such a function, since the position of dust stored in advance cannot be applied as it is, the position cut out from the entire image is recorded, and dust position conversion is performed when performing interpolation processing. Yes.

  However, as shown in FIG. 2, when there is dust 100 in the vicinity of the image 201 cut out from the photographed image 200, that is, the center coordinates D (Xi, Yi) of the circle surrounding the dust 100 are outside the image cut-out position. However, when the cutout image 201 is included due to the size of the dust 100 itself, there is a problem in that the included portion cannot be interpolated.

  The present invention has been made in view of the above-described problems, and it is an object of the present invention to perform a foreign substance removal process on a clipped image with high accuracy even when a portion of the captured image is clipped.

  The image processing apparatus of the present invention includes a foreign object information acquisition unit that acquires foreign object information including foreign object position information and foreign object size information extracted from a captured image for foreign object detection obtained by the imaging unit, and the imaging unit. Based on the cutout position information acquisition means for cutting out the cutout position information of the cutout image cut out from the photographed image at the time of normal shooting by means of, the position of the foreign matter is in the cutout image based on the position information of the foreign matter and the cutout position information If the determination means and the determination means determine that the position of the foreign object is not in the cut-out image, the position information of the foreign object is determined based on the size information of the foreign object. Conversion means for converting into position information of a foreign object.

  According to the present invention, even when a part of a photographed image is cut out, the foreign substance removal process for the cut-out image can be performed with high accuracy.

It is a block diagram which shows the circuit structure of the electronic camera which concerns on embodiment. It is a figure explaining the position of an image and dust. It is a flowchart explaining the dust detection process by the electronic camera which concerns on embodiment. It is a figure explaining the position and size of the dust used as dust correction data. 1 is a diagram illustrating a schematic configuration of an image processing apparatus according to an embodiment. 5 is a flowchart for explaining dust removal processing by the image processing apparatus according to the embodiment. It is a flowchart explaining a dust position conversion routine by the image processing apparatus according to the embodiment.

Preferred embodiments of the present invention will be described below with reference to the accompanying drawings.
FIG. 1 is a block diagram showing a circuit configuration of an electronic camera according to an embodiment of the present invention. Reference numeral 1 denotes an imaging optical system including a lens. Reference numeral 2 denotes an image pickup device (photoelectric conversion device) such as a CCDCMOS sensor. Reference numeral 3 denotes an A / D conversion circuit that converts an analog image signal obtained by the image sensor 2 into a digital signal.

  An image processing circuit (process circuit) 4 performs image processing such as gamma correction. Reference numeral 5 denotes a memory control circuit (recording / reproducing unit) that compresses / decompresses image data and exchanges signals from the external memory 7. A frame memory 6 is used for image development and camera control. Reference numeral 7 denotes an external memory for recording an image. Reference numeral 8 denotes an image display memory (buffer memory). Reference numeral 9 denotes an image display unit (monitor) that displays captured images and the like. A system control circuit 10 includes a CPU and the like.

  Reference numeral 11 denotes a lens drive control circuit that controls the focusing drive of the imaging optical system 1. Reference numeral 12 denotes a photographing information display unit that displays a shutter speed, an aperture value, and the like. Reference numeral 13 denotes a photographing instruction switch for instructing the camera to perform photographing. An image display switch 14 instructs the camera to display an image. A setting change switch 15 instructs the camera to display a setting change menu screen.

  Next, the operation of the electronic camera having the above configuration will be described. First, the system control circuit 10 detects a state change of any one of the switches 13, 14, and 15 by the photographer's operation, starts supplying power to the other blocks and the imaging optical system, and sets each block. Make it ready for use.

(Imaging processing)
When the photographer performs a photographing operation by operating the photographing instruction switch 13, the imaging optical system 1 is activated by a control signal from the system control circuit 10, and a subject image is formed on the imaging element 2. The electric signal photoelectrically converted by the image pickup device 2 is converted into a predetermined digital signal through the A / D conversion circuit 3 in order for each pixel and then input to the image processing circuit 4. The image processing circuit 4 generates RGB color signals based on the pixel data of the image sensor 2. Next, each pixel data is stored in the memory 6. The image data stored in the memory 6 is transferred to the image display memory 8 and displayed via the image display unit 9.

(Cut image generation processing)
Further, the system control circuit 10 uses preset image clipping position information from the image data stored in the memory 6, such as the upper left position (Xt, Yt) and lower right position (Xb, Yb) in FIG. Image data is cut out based on the information T. Then, the cut-out image can be stored in the memory 6 as a new image.

(Image saving process)
When saving the shooting result, the image data stored in the memory 6 is compressed by the memory control circuit 5 based on a predetermined compression format, and the result is stored in the external memory 7 (usually a non-volatile memory such as a flash memory). Use). At this time, information such as image cutout position information T (upper left position (Xt, Yt) and lower right position (Xb, Yb) in FIG. 2) can be added as information accompanying the image data.

  When the photographer changes the setting of the camera by operating the setting change switch 15, a setting change screen (configured by a user interface such as a menu) is displayed on the image display unit 9 from the system control circuit 10 through the image display memory 8. Can be displayed.

(Dust detection process)
FIG. 3 is a flowchart for explaining dust detection processing (detection processing of a pixel position where an image defect is caused by dust) by the electronic camera according to the present embodiment. The dust detection process is performed by capturing a dust detection image.

  When performing dust detection processing, a camera is installed with the imaging optical system 1 facing a surface having a uniform color, such as an emission surface of a surface light source device or a white wall, and preparation for dust detection is performed. After the preparation is completed, for example, when the start of dust detection processing is instructed from the photographing instruction switch 13, the system control circuit 10 first sets the aperture. The dust near the image sensor 2 changes its imaging state depending on the aperture value of the lens and changes its position depending on the pupil position of the lens. Accordingly, the dust correction data as foreign matter information needs to hold the aperture value and the lens pupil position at the time of dust detection processing in addition to dust position information and size information. Here, for example, F16 is designated (step S21).

  Next, the system control circuit 10 causes the lens drive control circuit 11 to perform aperture blade control of the imaging optical system 1, and sets the aperture to the aperture value specified in step S21 (step S22). Further, the focus position is set to infinity (step S23).

  When the aperture value and the focus position are set, shooting in the dust detection processing mode is executed (step S24). As a result, a dust detection image, which is a captured image for detecting foreign matter, is obtained.

  When shooting is completed, the aperture value and lens pupil position at the time of shooting are acquired (step S25). Next, data corresponding to each pixel of the captured image stored in the memory 6 is called through the image processing circuit 4 and the memory control circuit 5 (step S26).

  The system control circuit 10 acquires the position and size of the pixel where dust is present (step S27). The dust region acquisition processing in step S27 may use a known method, for example, the method disclosed in Patent Document 2, and a detailed description thereof is omitted here. As shown in FIG. 4, the dust position information and dust size information acquired in the dust region acquisition processing in step S27 are center coordinates D (Xi, Yi) (hereinafter abbreviated as Di) of a circle surrounding the dust 100. ) As position information and the radius Ri of the circle surrounding the dust 100 as size information.

  The position and size of the pixel where dust is present acquired in step S27, and the aperture value and lens pupil position information acquired in step S25 are registered in the external memory 7 as dust correction data (step S28).

(Dust removal (foreign matter removal) processing)
Next, the flow of dust removal processing will be described. The dust removal process is performed not by an electronic camera but by an image processing apparatus such as a personal computer (PC) prepared separately. FIG. 5 is a diagram illustrating a schematic configuration of the image processing apparatus. The CPU 1001 controls the operation of the entire system and executes a program stored in the primary storage unit 1002.

  The primary storage unit 1002 is mainly a memory, and reads and stores programs and the like stored in the secondary storage unit 1003. The secondary storage unit 1003 corresponds to a hard disk, for example. Generally, the capacity of the primary storage unit is smaller than the capacity of the secondary storage unit, and programs and data that cannot be stored in the primary storage unit are stored in the secondary storage unit. Data that must be stored for a long time is also stored in the secondary storage unit. In the present embodiment, the program is stored in the secondary storage unit 1003, read into the primary storage unit 1002 when the program is executed, and the CPU 1001 performs an execution process.

  The input device 1004 corresponds to, for example, a card reader, a scanner, a film scanner, and the like necessary for inputting image data in addition to a mouse and a keyboard used for system control. Examples of the output device 1005 include a monitor and a printer. Various other configurations of the apparatus configuration method are conceivable, but the description thereof is omitted because it is not the main point of the present invention.

  FIG. 6 is a flowchart for explaining dust removal processing by the image processing apparatus according to the present embodiment. First, the image processing apparatus takes in normal photographed image data attached with dust correction data from the external memory 7 in the electronic camera or removed from the electronic camera, and stores it in the primary storage unit 1002 or the secondary storage unit 1003 ( Step S101).

  Next, the dust correction data registered in step S28 is extracted from the normal captured image data in order to acquire foreign substance information (step S102).

  Next, from the dust correction data extracted in step S102, a coordinate row Di (i = 1, 2,..., N) which is dust position information and a radius row Ri (i = 1, 2) which is dust size information. ,..., N), the aperture value f1 and the lens pupil position L1 are acquired (step S103).

Next, an aperture value f2 and a lens pupil position L2 at the time of photographing a normal photographed image are acquired (step S104). Then, the coordinate Di, which is dust position information, and the radius sequence Ri, which is dust size information, are converted by the following equations (1) and (2), and the converted coordinates Di ′ and the converted radius Ri ′ are obtained. Obtain (step S105). Here, d is the distance from the center of the image to the coordinates Di, and H is the distance between the surface of the image sensor 2 and dust. The unit here is a pixel, and “+3” for Ri ′ is a margin amount.
Di ′ = (L2 × (L1-H) × d / ((L2-H) × L1)) × Di (1)
Ri ′ = (Ri × f1 / f2 + 3) (2)

  Next, in order to acquire cutout position information, cutout position information T (upper left position (Xt, Yt) and lower right position (Xb, Yb) in FIG. 2) of the image assigned when the image is stored is acquired. Then, the coordinate Di ′ after the conversion in step S105 is converted based on the cutout position information T of the image and, if necessary, the dust size information Ri, and the converted coordinate Dia (x, y). Or, Div (x, y) is obtained (step S106). Details of the dust position conversion routine will be described later.

  Next, dust in the area indicated by the coordinate Dia (x, y) or Div (x, y) after the conversion in step S106 and the dust size information Ri ′ is detected and interpolated as necessary. Processing is applied (step S107). The interpolation processing in step S107 may use a known method, for example, the method disclosed in Patent Document 2, and detailed description thereof is omitted here.

  Thereafter, it is determined whether or not dust removal processing has been applied to all dust (step S108). If processing has been completed for all dust, the processing ends. Otherwise, the processing returns to step S107.

(Dust position conversion routine)
With reference to FIG. 7, the flow of the dust position conversion routine in step S106 of FIG. 6 will be described. Note that the coordinates handled in this dust position conversion routine are such that the upper left of the clipped image 201 is the origin (0, 0) and the right and lower directions are increasing.

  First, in step S201, it is determined whether or not the coordinate Di ′ after the conversion in step S105 is within the image cutout range T (upper left position (Xt, Yt) and lower right position (Xb, Yb) in FIG. 2). judge.

If the coordinate Di ′ is within the cutout range T of the image in step S201, the process proceeds to step S202, where the coordinate Di ′ is converted into a coordinate based on the cutout image 201 by the following expression (3), and the coordinate Di ′ is converted. Replace with the coordinates Dia (x, y). Thereafter, the process proceeds to step S207.
Dia (x, y) = Di ′ − (Xt, Yt) (3)

On the other hand, if the coordinate Di ′ is not within the cutout range T of the image in step S201, the process proceeds to step S203, where the coordinate Di ′ is converted into a coordinate based on the cutout image 201, and based on the dust size information Ri. Convert. In step S203, the coordinate Di ′ is converted according to any of the following conditions, and the coordinate Di ′ is replaced with the converted coordinate Dib (x, y) (second foreign substance position information).
When Di ′ is above T (upper left position (Xt, Yt) and lower right position (Xb, Yb) in FIG. 2),
Dib (x, y) = Di ′ − (Xt, Yt) + (0, Ri)
When Di ′ is below T (upper left position (Xt, Yt) and lower right position (Xb, Yb) in FIG. 2),
Dib (x, y) = Di ′ − (Xt, Yt) − (0, Ri)
When Di ′ is at a position to the left of T (upper left position (Xt, Yt) and lower right position (Xb, Yb) in FIG. 2),
Dib (x, y) = Di ′ − (Xt, Yt) + (Ri, 0)
When Di ′ is at the right position from T (upper left position (Xt, Yt) and lower right position (Xb, Yb) in FIG. 2),
Dib (x, y) = Di ′ − (Xt, Yt) − (Ri, 0)

  Next, in step S204, as a second determination, it is determined whether or not the coordinates Div (x, y) converted in step S203 are within the cutout position range T of the image. If the coordinate Div (x, y) is within the image cutout range T, the process proceeds to step S205, and the coordinate Di 'is replaced with the converted coordinate Div (x, y). Thereafter, the process proceeds to step S207.

  On the other hand, if the coordinate Dib (x, y) is not within the image cutout range T, the process proceeds to step S206, where the coordinate Di 'is deleted from the coordinate sequence so that it is not subject to dust removal processing. In this case, the position of the dust is outside the cutout image 201 and is not included in the cutout image 201 due to the size of the dust itself, and is excluded from the target of dust removal processing.

  In step S207, it is determined whether the conversion of dust position information has been applied to all coordinates. If the processing has been completed for all coordinates, the process ends. If not, the process returns to step S201.

  As described above, even in the case of an image created by cutting out, in the subsequent dust removal processing, even if the position of the dust is outside the clipping range, Even in such a case, the dust removal process can be applied.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

1001: CPU, 1002: primary storage unit, 1003: secondary storage unit, 1004: input device, 1005: output device

Claims (4)

Foreign matter information acquisition means for acquiring foreign matter information including foreign matter position information and foreign matter size information extracted from a captured image for foreign matter detection obtained by the imaging means;
Cutout position information acquisition means for acquiring cutout position information of a cutout image cut out from a captured image at the time of normal shooting by the imaging means;
Determining means for determining whether or not the position of the foreign object is in the cut-out image based on the position information of the foreign object and the cut-out position information;
Conversion means for converting the position information of the foreign object into position information of the second foreign object based on the size information of the foreign object when the determination unit determines that the position of the foreign object is not in the cut-out image; An image processing apparatus comprising: Based on the position information of the second foreign matter and the cutout position information, further comprising second determination means for determining whether the position of the foreign matter after conversion is in the cutout image;
When the second determination means determines that the position of the foreign matter after conversion is in the cutout image, the cutout image is determined based on the position information of the second foreign matter and the size information of the foreign matter. The image processing apparatus according to claim 1, wherein an interpolation process for removing foreign matter is performed on the image processing apparatus. Acquiring foreign matter information including foreign matter position information and foreign matter size information extracted from a captured image for foreign matter detection obtained by the imaging means;
A step of obtaining cutout position information of a cutout image cut out from a shot image at the time of normal shooting by the imaging means;
Determining whether the position of the foreign object is in the cut-out image based on the position information of the foreign object and the cut-out position information;
Converting the position information of the foreign object into position information of a second foreign object based on the size information of the foreign object when it is determined that the position of the foreign object is not in the cut-out image. Image processing method. Processing for acquiring foreign matter information including foreign matter position information and foreign matter size information extracted from a captured image for foreign matter detection obtained by the imaging means;
A process of obtaining cutout position information of a cutout image cut out from a shot image at the time of normal shooting by the imaging unit;
A process for determining whether or not the position of the foreign object is in the cut-out image based on the position information of the foreign object and the cut-out position information;
In order to cause the computer to execute processing for converting the position information of the foreign object into the position information of the second foreign object based on the size information of the foreign object when it is determined that the position of the foreign object is not in the cut-out image Program.

Images