JP2014220717A - Image processing apparatus and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To apply high-dynamic combination on plural images photographed with different exposures, without giving discomfort even if there is an occlusion region.SOLUTION: The image processing apparatus comprises: input means that receives a first image photographed with exposure adjusted for a main subject and a second image photographed with exposure different from the exposure for the first image; detecting means that detects a non-common region representing different subjects in the first and second images; and combining means that combines the first and second images. The combining means uses a pixel value of the first image for a region of the main subject in a common region which is a region excluding the non-common region, uses a pixel value of the second image for a region which is not the main subject in the first image, uses a pixel value of the first image for the region of the main subject in the first image, and uses a pixel value obtained when the brightness of the first image is matched to the brightness of the second image for a region which is not the main subject in the first image.

Description

  The present invention relates to an image processing apparatus and method, and more particularly, to an image processing apparatus and method for expanding a dynamic range by combining a plurality of images taken at different exposures.

  For example, Patent Document 1 proposes high dynamic range synthesis (HDR) that obtains an image with a wide dynamic range by correcting and synthesizing gradations of a plurality of captured images with different exposures. In HDR, a subject image in an image is aligned between a plurality of images and then combined.

JP 2012-19337 A

  However, due to deformation or movement of the subject, camera shake, etc., an image is formed only on some of the captured images and is not formed on the remaining images (hereinafter referred to as an occlusion region). Description) may be possible. As an example, FIG. 2B shows an image that is aligned and overlapped when the first image 300 as shown in FIG. 2A and the second image 301 are synthesized. In this example, the person image 310 that is the main subject of the first image 300 moves until the second image 301 is photographed, and a person image 310 like the person image 311 of the second image 301. It has become bigger than. Due to this influence, an occlusion area that is a non-common area in which different subjects are captured in the first image and the second image is generated.

  Although there is a problem that if the occlusion area exists, the composition cannot be performed well. However, in Patent Document 1, the existence of the occlusion area is detected by using the reliability of the motion vector, and the composition ratio between a plurality of images is changed. Yes.

  However, in an image having an occlusion area, if the main subject or the non-main subject is aligned based on the influence of the distance between the main subject and the non-main subject, etc., the misalignment occurs on the other (non-main subject or main subject). Remains. Therefore, there is a problem that the outline of the main subject is doubled if simply combined.

  In addition, when the image of the occlusion area is saturated (overexposed) or blackout, even if the composition ratio is changed, overexposure or underexposure appears, and the output image is uncomfortable. was there.

  The present invention has been made in view of the above problems, and an object of the present invention is to enable high dynamic range composition of a plurality of images photographed at different exposures without feeling uncomfortable even if there is an occlusion area.

  In order to achieve the above object, an image processing apparatus according to the present invention inputs a first image taken with exposure of a main subject and a second image taken with an exposure different from the first image. Input means; detection means for detecting non-common areas representing different subjects in the first image and the second image; and synthesis means for synthesizing the first image and the second image. And the compositing means uses a pixel value of the first image in the first image in the common image that is a region excluding the non-common region, and uses the pixel value of the first image in the common region. The region of the first image that is not the main subject uses the pixel value of the second image, and of the non-common region, the region of the main subject of the first image is the region of the first image. Using the pixel value, the first of the non-common areas. A region that is not the main subject in the image is a first that synthesizes the first image and the second image using a pixel value that matches the luminance of the first image with the luminance of the second image. A synthesis process is performed.

  Even if there is an occlusion area, it is possible to synthesize a plurality of images taken at different exposures with a high dynamic range without feeling uncomfortable.

FIG. 2 is a block diagram showing a schematic configuration of an image processing apparatus according to first and second embodiments of the present invention. Illustration of occlusion area Explanatory drawing of the detection method of an occlusion area | region. Explanatory drawing which shows the example of correction | amendment of the occlusion area | region in 1st Embodiment. Explanatory drawing which shows the example of correction | amendment of another occlusion area | region in 1st Embodiment. Explanatory drawing of another detection method of an occlusion area | region. FIG. 10 is a layout diagram of feature points in the second embodiment. Explanatory drawing of the movement of points other than a feature point. Explanatory drawing of the weighting coefficient W. FIG. Explanatory drawing of the movement of the arbitrary points on the image accompanying a deformation | transformation. The block diagram which shows schematic structure of the image processing apparatus in 3rd Embodiment.

  The best mode for carrying out the present invention will be described below in detail with reference to the accompanying drawings.

<First Embodiment>
FIG. 1 is a block diagram illustrating a schematic configuration of an image processing apparatus according to the first embodiment. In the image processing apparatus illustrated in FIG. 1, the first image is input from the first input terminal 100 and the second image is input from the second input terminal 102 and is input to the alignment unit 104.

  In the first embodiment, the first image is an image shot with exposure control so that the main subject is shot with appropriate exposure. For example, in the portrait photography mode, AE (Automatic Exposure) control is performed using a person to be photographed as a main subject and matching the person using face detection or the like. At this time, an area other than the main subject (hereinafter referred to as “non-main subject”) may be overexposed or blackout.

  On the other hand, the second image is an image shot with exposure control so that the non-main subject is shot with appropriate exposure. In a case where the non-main subject area of the first image is overexposed, the main subject area of the second image appears dark and may be blackened. On the other hand, in the case where the non-main subject area of the first image is blacked out, the main subject area of the second image appears bright and may be overexposed.

  The alignment unit 104 inputs two images with different exposures as described above, and performs alignment so that the same subject formed on both images overlap. The alignment is a common technique, and is described in, for example, Patent Document 1 and therefore will not be described here. At the time of alignment, only one of the first image and the second image may be moved, or both may be moved.

  The first cutout unit 101 inputs the first image aligned with the second image by the alignment unit 104, extracts the main subject region from the input first image, and indicates the main subject region. 1 cutout information is output. In addition, the second cutout unit 103 inputs the second image aligned with the first image by the alignment unit 104, extracts a non-main subject area from the input second image, and extracts the non-main subject. Second cutout information indicating the area is output.

  The occlusion area detection unit 105 obtains an occlusion area (non-common area) based on the first cutout information and the second cutout information. Then, the correction unit 106 corrects the detected pixel value of the occlusion area by gain correction. The synthesizing unit 107 generates a synthesized image by synthesizing the first image and the second image. The correction process and the synthesis process performed here will be described in detail later.

  Next, a method for obtaining an occlusion area in the first embodiment will be described. The first cutout unit 101 extracts outline information (first outline information) of the main subject region from the first image as the first cutout information. The second cutout unit 103 extracts contour information (second contour information) of the non-main subject region from the second image as the second cutout information. The occlusion area detection unit 105 detects an occlusion area from the positional deviation between the extracted outline information of the main subject area and the outline information of the non-main subject area.

  Here, with reference to FIG. 3, a case where the first image and the second image are the images shown in FIG. 2 will be described as an example. 3A shows the outline information 400 of the main subject region extracted from the first image 300 shown in FIG. 2A by the first cutout unit 101, and the second image 301 by the second cutout unit 103. FIG. FIG. 6 is a diagram showing contour information 401 of a non-main subject region extracted from

  FIG. 3B shows a state in which the contour information 400 of the main subject region and the contour information 401 of the non-main subject region are overlapped. The occlusion area detection unit 105 obtains a feature point 403 on the outline information 400 of the main subject area and a feature point on the outline information 401 of the non-main subject area corresponding to the feature point 403, and sets the corresponding feature point 403. The misalignment area is taken as the occlusion area. The contour information of the occlusion area is formed by obtaining a plurality of feature points and interpolating between the feature points.

  The correction unit 106 corrects the occlusion region by gain correction based on the contour information of the occlusion region detected by the occlusion region detection unit 105 as described above.

  For the portion of the occlusion area which is the main subject area on the first image, the pixel value of the first image aligned as the pixel value of the occlusion area of the output image is output.

  On the other hand, for a portion that is a non-main subject area on the first image in the occlusion area, a pixel value obtained by multiplying the aligned first image by a gain is used as the pixel value of the occlusion area of the output image. Output. The gain to be multiplied is a gain obtained from an exposure step between the exposure of the first image and the exposure of the second image so that the exposure of the first image matches the exposure of the second image. For example, if the first image and the second image are the images shown in FIG. 2, the background (other than the person image 310) appears in the first image 300 in the region 303 in the occlusion region, and the second image In 301, a part of the person image 310 is shown. In this case, in the first embodiment, as shown in FIG. 4, the background image uses the second image 301 outside the region 303 (outside the dotted line) and the second image 301 inside the region 303 (inside the dotted line). The pixel value of one image 301 is used after gain adjustment. By performing the synthesis process (first synthesis process) in this way, natural results can be obtained.

  Further, for example, as shown in FIG. 5A, even when the human image 610 of the first image 600 is larger than the human image 611 of the second image 601, FIG. The area 603 which is the background image in the first image shown in FIG. By doing so, a natural image as shown in FIG. 5C can be obtained.

  For detection of the occlusion area, not only contour information but also a differential image of pixel values (for example, luminance) as shown in FIG. 6 may be used. First, as shown in FIG. 6A, a gain is applied to the second image 301 so as to fill an exposure step between the first image shooting and the second image shooting. For example, if the second image 301 is darker than the first image 300 by one step, a double gain is multiplied to make the second image 301 brighter by one step. Then, by obtaining the luminance difference between the first image 300 having the same luminance level and the second image 320 after gain correction (FIG. 6B), an occlusion region 321 is obtained (FIG. 6 ( c)).

  As described above, according to the first embodiment, even if there is an occlusion region between images to be synthesized, high dynamic range synthesis can be realized with a relatively simple configuration. It can be realized with a small circuit.

<Second Embodiment>
The method described in the first embodiment described above cannot cope with a case where the occlusion area of the first input image to be output instead of the second input image has whiteout or blackout. In the second embodiment, a method corresponding to such a case will be described.

  The correction unit 106 according to the second embodiment corrects the occlusion area by deforming / magnifying the non-main subject area based on the contour information of the occlusion area detected by the occlusion area detection unit 105. A specific implementation method will be described below.

  In the first embodiment described above, when the occlusion area is obtained, the feature points on the outline information 400 of the main subject area of the first image and the outline of the non-main subject area of the second image as shown in FIG. A correspondence relationship with the feature point 403 on the information 401 is used. In the correction by deformation / magnification described later in the second embodiment, these feature points are used.

  Further, in addition to these feature points, feature points are also arranged at predetermined intervals on the upper, lower, left and right edges of the image, and these movement vectors are set to zero. For example, the feature points 700 at the top, bottom, left and right ends in FIG. 7 are arranged. These feature points 700 at the upper, lower, left and right ends are used as they are in the calculation of the coordinate movement amount, which will be described later.

  A movement vector from each feature point of the first image to a corresponding feature point of the second image is obtained. In the example shown in FIG. 7, the feature point 403 on the first image 300 and the feature point 702 on the second image 301 correspond to each other, but in this case, the feature point 403 is reduced by the movement vector 701. A feature point 703 on the first image 300 and a feature point 704 on the second image 301 correspond to each other, but are reduced by the movement vector 705. Similarly, the movement vectors of all feature points are obtained. Note that, when the magnitude of the maximum vector (coordinate movement amount) in the obtained movement vector exceeds a predetermined magnitude, correction by deformation / magnification is not performed.

  As shown in FIG. 8, an arbitrary point P in the contour information 401 of the non-main subject area on the second image is output to a point Q on the output image. At this time, the coordinate movement amount from the point P to the point Q is obtained by the following equation (1).

  The scalar coefficient W (n) in equation (1) is a weight that increases as the norm decreases according to the norm between the point P and the feature point n, and the amount of movement is determined by the load average. The K represents all feature points. As the norm, for example, a weight table as shown in FIG. 9 is used using the Euclidean norm or the like. By this deformation method, as shown in FIG. 10, an arbitrary point 1100 on the image moves to the destination point 1102 according to the movement vector 1101 obtained by the movement vector calculation by the equation (1), so that the image becomes Deformed and scaled.

  In realization, a method of obtaining the point P from the point Q obtained by reversing the mapping relationship may be used. What is necessary is just to obtain | require by the same method, reversing the direction of the movement vector of each feature point 180 degree | times.

  Then, the pixel value of the main subject area of the first image and the pixel value of the non-main subject area of the second image subjected to the deformation scaling process as described above are synthesized (second synthesis process).

  As described above, according to the second embodiment, even if the occlusion area includes a large number of overexposed or underexposed pixels (saturated pixels and underexposed pixels), it is more natural for high dynamic range synthesis. An output image can be obtained.

<Third Embodiment>
Next, a third embodiment of the present invention will be described. FIG. 11 is a block diagram illustrating a configuration of an image processing apparatus according to the third embodiment. The basic configuration is the same as that described with reference to FIG. 1, but is different from the configuration shown in FIG. 1 in that a correction processing selection unit 206 is added.

  The correction process selection unit 206 selects a correction process according to the state of the occlusion area. First, in the occlusion area of the first image, the pixel of the portion that is a non-main subject area on the first image is examined, and the ratio of overexposed pixels (saturated pixels) and blackout pixels exceeds a predetermined ratio If so, the correction processing described in the second embodiment is selected. On the other hand, when the ratio of the saturated pixels and the blackout pixels does not exceed the predetermined ratio, the correction process described in the first embodiment is selected. That is, the first combining process for combining the images by performing the correction process described in the first embodiment and the correction process described in the second embodiment according to the ratio of the saturated pixels and the blackout pixels. And a second combining process for combining the images.

  This determination may be performed independently for a plurality of individual occlusion areas existing on the image, or may be determined collectively as a whole.

  According to the third embodiment, a more natural output image can be obtained according to the number of whiteout or blackout pixels in the occlusion area.

<Other embodiments>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (12)

An input means for inputting a first image photographed with exposure adjusted to a main subject and a second image photographed with an exposure different from the first image;
Detecting means for detecting non-common areas representing different subjects in the first image and the second image;
Combining means for combining the first image and the second image;
The synthesis means includes
Of the common area that is an area excluding the non-common area, the area of the main subject in the first image uses the pixel value of the first image,
Of the common area, the area that is not the main subject in the first image uses the pixel value of the second image,
Among the non-common areas, the area of the main subject in the first image uses the pixel value of the first image,
Of the non-common area, an area that is not the main subject in the first image uses a pixel value that matches the luminance of the first image with the luminance of the second image,
An image processing apparatus that performs a first combining process for combining the first image and the second image. The synthesizing unit further deforms and changes the second image so that the feature point of the main subject in the second image moves to the position of the feature point of the main subject in the first image. In the first image, the region of the main subject uses the pixel value of the first image, and the region that is not the main subject in the first image undergoes the deformation scaling process. A second combining process for combining the first image and the second image using the pixel values of the second image and the first combining process;
When the saturation pixel and the blackout pixel included in the region that is not the main subject in the first image in the non-common region exceed a predetermined ratio, the second synthesis process is performed, The image processing apparatus according to claim 1, further comprising selection means for selecting to perform the first composition processing when the predetermined ratio is not exceeded. The detection means includes
First cutout means for cutting out the first contour information of the area of the main subject from the first image;
Second cutout means for cutting out the second contour information of the area of the non-main subject from the second image;
3. The image processing apparatus according to claim 1, further comprising an extraction unit that extracts a region corresponding to a positional deviation between the first contour information and the second contour information as the non-common region. .   The detection means is required to eliminate an exposure step between an exposure when the first image is captured and an exposure when the second image is captured in the pixel value of the second image. The non-common area is detected by taking a luminance difference between the second image after gain correction obtained by multiplying the gain and the first image. Image processing apparatus. Input means for inputting a first image taken with exposure adjusted to the main subject and a second image taken with exposure adjusted other than the main subject;
Combining means for combining the first image and the second image;
The synthesizing unit is configured to apply the feature point of the main subject in the second image to the non-main subject region of the second image so that the feature point of the main subject moves to the position of the feature point of the main subject in the first image. A deformation scaling process is performed on the first image, and the region of the main subject in the first image uses the pixel value of the first image, and the region that is not the main subject in the first image An image processing apparatus that combines the first image and the second image using pixel values of the second image that has been subjected to the double processing.   6. The image according to claim 5, wherein when the coordinate movement amount of the pixel of the second image accompanying the deformation scaling process exceeds a predetermined size, the synthesizing unit does not perform synthesis. Processing equipment. An alignment unit for aligning the first image and the second image;
The detection means detects a non-common area using the first image and the second image aligned by the alignment means,
The image processing apparatus according to claim 1, wherein the synthesizing unit synthesizes the first image and the second image aligned by the alignment unit. An input step in which the input means inputs a first image shot with a main subject adjusted for exposure, and a second image shot with an exposure different from the first image;
A detecting step for detecting a non-common area representing different subjects in the first image and the second image;
A synthesis means comprising a synthesis step of synthesizing the first image and the second image;
In the synthesis step,
Of the common area that is an area excluding the non-common area, the area of the main subject in the first image uses the pixel value of the first image,
Of the common area, the area that is not the main subject in the first image uses the pixel value of the second image,
Among the non-common areas, the area of the main subject in the first image uses the pixel value of the first image,
Of the non-common area, an area that is not the main subject in the first image uses a pixel value that matches the luminance of the first image with the luminance of the second image,
An image processing method comprising performing a first combining process for combining the first image and the second image. In the synthesizing step, the second image is further deformed and changed so that the feature point of the main subject in the second image moves to the position of the feature point of the main subject in the first image. In the first image, the region of the main subject uses the pixel value of the first image, and the region that is not the main subject in the first image undergoes the deformation scaling process. A second combining process for combining the first image and the second image using the pixel values of the second image and the first combining process;
When the selection unit includes a saturation pixel and a blackout pixel included in a region that is not the main subject in the first image in the non-common region, the second synthesis is performed. 9. The image processing method according to claim 8, further comprising a selection step of performing processing and selecting to perform the first synthesis processing when the predetermined ratio is not exceeded. An input step in which the input means inputs a first image taken with exposure adjusted to the main subject and a second image taken with exposure adjusted to other than the main subject;
A synthesis means comprising a synthesis step of synthesizing the first image and the second image;
In the synthesizing step, the feature point of the main subject in the second image is moved to the position of the feature point of the main subject in the first image. A deformation scaling process is performed on the first image, and the region of the main subject in the first image uses the pixel value of the first image, and the region that is not the main subject in the first image An image processing method comprising combining the first image and the second image using pixel values of the second image that has been doubled.   A program for causing a computer to execute each step of the image processing method according to any one of claims 8 to 10.   A computer-readable storage medium storing the program according to claim 11.