JP2018050149A - Image processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus which allows generation of a composite image having natural gradation characteristics, even if a subject or a camera moves when synthesizing multiple images.SOLUTION: An image processing apparatus synthesizing multiple images has generation means for generating a composite ratio image for determining the composite ratio of image composition, according to the brilliance of at least one of a first image and a second image, detection means for detecting slippage of the first and second images, and control means for controlling the final composite ratio of the first and second images, according to the composite ratio image and detection results outputted from the detection means. The control means uses the detection results from the detection means only in an area where the first and second images are subjected to weighted synthesis in the composite ratio image.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing apparatus that combines a plurality of images.

  A digital camera or a digital video camera performs image signal processing by photoelectrically converting a captured subject image by a pixel unit in accordance with the intensity of light using an image sensor such as a CMOS sensor or a CCD, and converting it into a video signal. In a scene where the dynamic range of the image sensor is narrow and the brightness difference between a bright subject and a dark subject is large, it is difficult to capture both subjects with appropriate exposure.

  In order to solve such a problem, a method has been proposed in which a plurality of images with different exposure amounts are photographed in the same scene, and a plurality of image data are combined to obtain an image with an expanded dynamic range. .

  In Patent Document 1, the signal level is corrected using an amplification factor corresponding to the exposure amount ratio between the long exposure signal and the short exposure signal, and the level corrected long exposure signal and the short exposure signal are synthesized. A method is disclosed.

Japanese Patent Laid-Open No. 7-75026

  The image synthesizing method proposed in Patent Document 1 is a synthesizing method based on the premise that the positions of two images to be synthesized are not shifted. Accordingly, if the two images to be combined are displaced due to movement of the subject or camera shake of the photographer, an area where the tone characteristics of the combined image are discontinuous occurs. . There is a problem that the discontinuous region appears to be a double image or is blurred in the synthesized image.

  The present invention has been made in view of the above-described problems.When a plurality of images are combined, the subject is slightly moved and the camera is slightly moved due to camera shake of the photographer. It is an object of the present invention to provide an image composition processing apparatus that prevents a problem that the tonal characteristics become discontinuous and results in an unnatural composite image, and makes it possible to generate a natural composite image.

  In order to achieve the above-described object, an embodiment of the present invention is an image processing apparatus that combines a plurality of images, and performs image combining according to the luminance of at least one of the first image and the second image. In accordance with detection results output from the generating means for generating a composite ratio image for determining a composite ratio, detection means for detecting a shift between the first image and the second image, the composite ratio image and the detection means, Control means for controlling the final synthesis ratio of the first image and the second image, and the control means is limited to an area where the first image and the second image are weighted and synthesized in the synthesis ratio image. The detection result of the detection means is used.

  According to the present invention, when a plurality of images are combined, a combined image having continuous gradations can be generated even when the subject moves slightly and the camera moves slightly due to camera shake of the photographer. This makes it possible to improve the quality of the composite image.

1 is a block diagram of an image composition processing apparatus according to an embodiment of the present invention. FIG. 3 is a functional configuration block diagram (without countermeasures against misalignment) according to an embodiment of the present invention. It is an operation | movement flowchart of the image composition processing apparatus concerning embodiment of this invention. It is a figure explaining the gain map and (alpha) image concerning embodiment of this invention. It is a figure explaining the synthetic | combination ratio characteristic concerning embodiment of this invention. FIG. 3 is a functional configuration block diagram (with misalignment countermeasures) according to the embodiment of the present invention. It is an operation | movement flowchart of the image composition processing apparatus concerning embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  In the embodiment described below, when a person is photographed outdoors in fine weather using a digital camera, the sky and clouds in the background will be blown off when the exposure is adjusted to the person. An example will be described in which the present invention is applied to a shooting scene in which a person becomes dark when exposure is adjusted.

  In the present embodiment, two images (low exposure image and high exposure image) with different exposures are taken, tone mapping processing suitable for each is performed, and the low exposure image and high exposure image after the tone mapping processing are performed. When combining a dark subject with a high-exposure image and a bright subject with a low-exposure image to create a composite image with an expanded dynamic range, the positions of two images with different exposures An example will be described in which the present invention is applied to a case where a slight deviation occurs.

  FIG. 1 is a block diagram showing a configuration of a digital camera 100 according to an embodiment of the present invention.

  In FIG. 1, 100 is a digital camera, 101 is a lens group including a zoom lens and a focus lens, 102 is a shutter having a diaphragm function, and 103 is an imaging unit including a CCD, a CMOS element, or the like that converts an optical image into an electrical signal. , 104 is an A / D converter that converts an analog signal into a digital signal, and 105 is an image processing unit that performs various image processing on the image data output from the A / D converter 104.

  The image processing unit 105 performs predetermined calculation processing on the captured image data, and performs exposure control and distance measurement control in the system control unit 50 based on the obtained calculation result. Thereby, AF (autofocus) processing, AE (automatic exposure) processing, and EF (flash pre-emission) processing of the TTL (through-the-lens) method are performed. Further, the image processing unit 105 performs predetermined calculation processing on the captured image data, and performs TTL AWB (auto white balance) processing based on the obtained calculation result.

  Output data from the A / D converter 104 is directly written into the memory 106 via the image processing unit 105 and the memory control unit 107 or via the memory control unit 107. The memory 106 stores image data obtained by converting image data captured by the image capturing unit 103 into digital data by the A / D converter 104 and image data to be displayed on the display unit 109. The memory 106 has a storage capacity sufficient to store a predetermined number of still images, a moving image and sound for a predetermined time. The memory 106 also serves as an image display memory (video memory). The D / A converter 108 converts the image display data stored in the memory 106 into an analog signal and supplies the analog signal to the display unit 109.

  Thus, the display image data written in the memory 106 is displayed on the display unit 109 via the D / A converter 108. The display unit 109 performs display according to an analog signal from the D / A converter 108 on a display such as an LCD. Reference numeral 110 denotes an interface (I / F) with the recording medium 111, reference numeral 111 denotes a recording medium such as a memory card or hard disk, and reference numeral 50 denotes a system control unit that controls the entire system of the digital camera 100.

  The system control unit 50 executes a program recorded in the nonvolatile memory 114. 112 is a system timer that measures the time used for various controls and the time of the built-in clock, 113 is a constant and variable for operating the system control unit 50, a program read from the nonvolatile memory 114 described later, and the like. System memory. The nonvolatile memory 114 can be electrically erased and recorded, and for example, an EEPROM or the like is used. The nonvolatile memory 114 stores constants, programs, and the like for operating the system control unit 50. Reference numeral 115 denotes an operation unit for inputting various operation instructions.

  FIG. 2 shows a functional block diagram relating to the composition processing in the present embodiment. FIG. 2 shows a functional configuration block diagram in the case where positional deviation between two images (low exposure image and high exposure image) having different exposures is not taken into consideration. A functional configuration block diagram in consideration of misalignment will be described in detail later.

In FIG. 2, reference numeral 201 denotes a gain map generation circuit. Gain map generator circuit 201, to the low-exposure image taken with short exposure signal S _S, to generate a gain map for tone mapping (tone correction).

A gain processing circuit 202 multiplies the input long-time exposure signal _SL by the gain map for the high-exposure image generated by the gain map generation circuit 201 and outputs the multiplication result. 203 is a gain processing circuit, the short exposure signal S _S input, multiplied by the gain map for the low-exposure image generated by the gain map generator circuit 201, and outputs the multiplication result.

  Reference numeral 204 denotes a composition ratio calculation circuit that generates a composition ratio image (hereinafter referred to as an α image) based on the gain map for the low exposure image generated by the gain map generation circuit 201. An image synthesis circuit 205 synthesizes the low-exposure image and the high-exposure image after gain processing output from the gain processing circuits 202 and 203 at a synthesis ratio based on the α image calculated by the synthesis ratio calculation circuit 204. .

  The synthesizing process operation with the above configuration will be described with reference to the flowchart of FIG.

  In step S301, the values possessed by the system control unit 50 are initialized.

In step S302, the system control unit 50 of the exposed image by the exposed by long exposure signal S _L that is input image short exposure signal Ss, either the image and the reference image at the time of gain map generator Select whether to do. In the present embodiment, short image exposed by the exposure signal S _S selects a reference image, for a long time an example of an exposure signal S _L were selected as non-reference image will be described.

  In step S303, the gain map generation circuit 201 generates a gain map image based on the reference image selected in step S302. The gain map has the same size as the image size, and the pixel value represents a gain multiplication value to be multiplied. How much tone mapping processing is applied to the image may be determined by any method. For example, the luminance distribution for each region of the low-exposure image can be analyzed and determined. Specifically, when shooting a scene including sky, people, and background as shown in FIG. 4A, a low-exposure image that is not overexposed is shown in FIG. 4B as n × m. The block is divided into blocks, and each of the blocks corresponds to which area of the sky, the person, and the background area, and the area is determined to calculate a luminance histogram for each area.

  As an area determination method, an existing method may be used. For example, for an empty area, a block in which both the luminance value and color difference BY of each block are larger than a predetermined value is set as an empty area. For example, for the human area, the human face is detected by wavelet transforming the image and pattern matching, or the human face to be recognized as the main subject is imaged in advance and stored in the nonvolatile memory 114 before imaging. Thus, the facial feature amount detected by the above-described method is specified by comparing it with the facial feature amount of the person to be recognized as the main subject.

  Region determination is performed using these existing methods, and a luminance histogram for each region is calculated. As shown in FIG. 4B, the pixel value of the gain map of the sky region block with high luminance is set to a small gain value based on the result of the luminance histogram, as shown in FIG. The gain map is generated by controlling the gain amount so that the pixel value of the gain map in the low luminance region such as the face region of the backlight scene is a large gain value, and the intermediate gain value if the luminance value is an intermediate luminance value.

  Further, the gain map for multiplying the high exposure image is generated by multiplying the gain map generated for the low exposure image by the gain corresponding to the exposure step. For example, if the low-exposure image and the high-exposure image have an exposure step of one step, the gain map obtained by multiplying the gain map to be multiplied by the low-exposure image by a factor of x1 / 2 is the gain of the high-exposure image. Map.

  In step S304, the composition ratio calculation circuit 204 generates an α image. The α image has the same size as the image size, and the pixel value represents a composite ratio value of either the low exposure image or the high exposure image. For example, the ratio value to be combined is calculated based on the combination ratio characteristic as shown in FIG. The horizontal axis in FIG. 5 indicates the gain value of the gain map for the low exposure image generated by the gain map generation circuit 201, and the vertical axis indicates the composition ratio.

  In the range of C0 having a low gain value (a region where the luminance value is high and the gain amount of the gain map is small because the luminance value of the low-exposure image is high), the composition ratio of the low-exposure image is 100% and the gain value is high. In the range of (a region where the luminance value of the low exposure image is low and the gain value of the gain map is large because the luminance value is low), the composition ratio of the high exposure image is 100%. The range of C1, which is worth the middle, has a characteristic of blending a low exposure image and a high exposure image.

  In step S305, the gain processing circuits 202 and 203 perform gain processing for multiplying the low exposure image and the high exposure image by the gain map generated by the gain map generation circuit 201, respectively.

  In step S306, the image composition circuit 205 synthesizes the low exposure image and the high exposure image based on the α image generated in step S304. As shown in FIG. 4C, the synthesized image has a high composition ratio of the low-exposure image for the sky area, a high composition ratio of the high-exposure image for the human area, and a low exposure for the background area. The result is a blended image / high-exposure image.

  In the foregoing, the processing operation of the digital camera 100 when the countermeasure for misalignment is not considered has been described.

  Next, when misalignment occurs between two images with different exposures (low exposure image and high exposure image), the misalignment between the two images is detected, and the final composition ratio is controlled based on the detection result. An example of synthesizing processing will be described.

  FIG. 6 shows a functional block diagram relating to the composition processing in the present embodiment. 6 has a configuration in which a motion detection circuit 601 and a final synthesis ratio calculation circuit 602 are added to the functional blocks described above with reference to FIG. Therefore, in the following description, the same reference numerals are used for blocks common to those in FIG. 2, and only the description of the motion detection circuit 601 and the final composite ratio calculation circuit 602, which are the differences, and the operation flowchart will be described in detail.

In FIG. 6, reference numeral 201 denotes a gain map generation circuit. Gain map generator circuit 201, to the low-exposure image taken with short exposure signal S _S, to generate a gain map for tone mapping (tone correction). A gain processing circuit 202 multiplies the input long-time exposure signal _SL by the gain map for the high-exposure image generated by the gain map generation circuit 201 and outputs the multiplication result. 203 is a gain processing circuit, the short exposure signal S _S input, multiplied by the gain map for the low-exposure image generated by the gain map generator circuit 201, and outputs the multiplication result.

  The composition ratio calculation circuit 204 generates a composition ratio image (α image) based on the gain map for the low exposure image generated by the gain map generation circuit 201. Reference numeral 601 denotes a motion detection circuit which receives a high-exposure image after gain processing from the gain processing circuit 202 and a low-exposure image after gain processing from the gain processing circuit 203. The motion detection circuit 601 detects the motion of the input high-exposure image and low-exposure image after gain processing.

  The final composite ratio calculation circuit 602 generates a final composite ratio image (hereinafter referred to as an α ′ image) based on the α image generated by the input composite ratio calculation circuit 204 and the motion detection result from the motion detection circuit 601. To do. Reference numeral 205 denotes an image composition circuit, which is based on the calculation result (α ′ image) calculated by the final composition ratio calculation circuit 602 for the low-exposure image and the high-exposure image after gain processing output from the gain processing circuits 202 and 203. Synthesize at the synthesis ratio.

  The composition processing operation with the above configuration will be described with reference to the flowchart of FIG.

  In step S701, the value possessed by the system control unit 50 is initialized.

In step S702, the system control unit 50 of the exposed image by the exposed by long exposure signal S _L that is input image short exposure signal Ss, either the image and the reference image at the time of gain map generator Select whether to do. In the present embodiment, short image exposed by the exposure signal S _S selects a reference image, for a long time an example of an exposure signal S _L were selected as non-reference image will be described.

  In step S703, the gain map generation circuit 201 generates a gain map image based on the reference image selected in step S702. The gain map has the same size as the image size, and the pixel value represents a gain multiplication value to be multiplied.

  Regarding the method of generating the gain map, as described above, the reference image is divided into n × m blocks as shown in FIG. 4B, and the luminance histogram of each block is calculated. In addition, an area is determined by analyzing which of the sky, human, and background areas each block corresponds to, and a luminance histogram for each area is calculated.

  Based on the result of the brightness histogram for each area, the pixel value of the gain map of the block in the sky area with high brightness is set to a small gain value, and the pixel value of the gain map in the area of low brightness such as the face area of the backlight scene is A gain map is generated by controlling the amount of gain so that the gain value is an intermediate gain value if it is a large gain value or an intermediate luminance value. Further, the gain map for multiplying the high exposure image is generated by multiplying the gain map generated for the low exposure image by the gain corresponding to the exposure step.

  In step S704, the composition ratio calculation circuit 204 generates an α image. As described above, a ratio value to be synthesized is calculated based on a synthesis ratio characteristic as shown in FIG.

  In step S705, the gain processing circuits 202 and 203 perform gain processing for multiplying the low exposure image and the high exposure image by the gain map generated by the gain map generation circuit 201, respectively.

  In step S706, the motion detection circuit 601 detects motion between the reference image and the non-reference image after gain multiplication in step S705. Specifically, a luminance difference value between the high-exposure image and the low-exposure image after gain processing is calculated, and an area where the difference value is equal to or greater than a predetermined threshold is detected as a motion area.

  In step S707, the final composition ratio calculation circuit 602 calculates the final composition ratio based on the α image generated in step S704 and the motion detection result detected in step S706. Specifically, the detection result of the motion detection circuit 601 is reflected in the final composition ratio only when the pixel value of the α image is a ratio value for blending the high exposure image and the low exposure image. For example, in a region where the pixel value of the α image is a combination of the high exposure image or the low exposure image at a ratio of 100%, the pixel value of the α image is low compared to the high exposure image without using the result of the motion detection circuit 601. The area that is a ratio value for blending the exposure images and that is detected as a motion area in the motion detection circuit 601 controls the final composition ratio so that the composition ratio of the low-exposure image is 100%. In this description, an example in which the composition ratio of the low-exposure image is set to 100% has been described. However, the result of the motion detection circuit 601 is obtained in the region where either image is output at 100% in the α image. If not used, it is not limited to this.

  In step S708, the image composition circuit 205 synthesizes the low exposure image and the high exposure image based on the final composition ratio calculated in step S707.

  As described above, the composition processing operation of the image processing apparatus considering the positional deviation of two images when the positions of the two images having different exposures are slightly shifted has been described. As a result, even if the subject moves slightly or the camera moves slightly due to the camera shake of the photographer, the α image generated based on the gain map in the synthesis ratio calculation circuit 204 is either a low exposure image or a high exposure image. If one is within the region where 100% is output, the final composition ratio is controlled so that the motion detection result is not used, so that the image quality of the composite image is improved.

  In the present embodiment, the luminance distribution is analyzed based on the sky, people, background, and the like to generate an α image. Therefore, an α image having an α value within a certain range is generated for each region. It tends to be. Therefore, for example, in an area of a person who becomes dark in a backlight scene, an α image is generated in which the composition ratio of the high-exposure image is set to 100%. However, in this region, control is performed so that the high-exposure image is synthesized at a ratio of 100% without using the motion detection result, and it is possible to prevent the adverse effect that the hand in the human body becomes a double image.

  In the present embodiment, only when the pixel value of the α image generated by the composition ratio calculation circuit 204 is a ratio value for blending the high exposure image and the low exposure image in the final composition ratio calculation circuit 602. Although the example in which the detection result of the motion detection circuit 601 is reflected in the final composition ratio has been described, the present invention is not limited to this. For example, whether or not to use the result of the motion detection result may be configured to provide a predetermined threshold range in the pixel value of the α image. For example, the configuration may be such that the motion detection result is not used when the pixel value of the α image is a predetermined level or lower (first level or lower), or higher than a predetermined level (second level or higher).

  In the present embodiment, an example in which two low-exposure images and high-exposure images are combined has been described. However, a configuration in which three or more images with different exposures are combined may be used.

DESCRIPTION OF SYMBOLS 50 System control part 100 Digital camera 103 Image pick-up part 105 Image processing part 106 Memory 107 Memory control part 109 Display part 111 Recording medium 115 Operation part

Claims (6)

An image processing apparatus for combining a plurality of images,
Generating means for generating a composition ratio image for determining a composition ratio of image composition in accordance with the luminance of at least one of the first image and the second image;
Detecting means for detecting a shift between the first image and the second image;
Control means for controlling the final composition ratio of the first image and the second image according to the composition ratio image and the detection result output from the detection means;
The image processing apparatus according to claim 1, wherein the control unit uses a detection result of the detection unit only in a region where the first image and the second image are weighted and combined in the combination ratio image. An image processing apparatus for combining a plurality of images,
Generating means for generating a composite ratio image of image synthesis according to the luminance of at least one of the first image and the second image;
Detecting means for detecting a shift between the first image and the second image;
Control means for controlling the final composition ratio of the first image and the second image in accordance with the composition ratio image generated by the generation means and the detection result output from the detection means;
The control means controls whether or not to use the motion detection result of the detection means in accordance with the composition ratio of the composition ratio image.   The control means controls the final composition ratio without using the detection result of the detection means when the composition ratio of the composition ratio image is not more than a predetermined first level or not less than a predetermined second level. The image processing apparatus according to claim 2. Tone mapping processing means for performing tone mapping processing on an image;
For the first image and the second image after the tone mapping process, the control means determines the first ratio according to the composite ratio image generated by the generation means and the detection result output from the detection means. The image processing apparatus according to any one of claims 1 to 3, wherein a final composition ratio of the image and the second image is controlled.   The generation unit calculates a luminance distribution for each region of the subject in the first image or the second image, and generates a composite ratio image based on the calculation result. An image processing apparatus according to any one of the above.   6. The image processing apparatus according to claim 5, wherein the composition ratio image generated by the generation unit is a composition ratio within a certain range for each region of the subject.

Images