JP2012044377A - Image compositing device, image compositing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a composite image having little halation and little black saturation even when a dynamic body exists in an imaging field angle.SOLUTION: When push of a shutter key is detected, proper exposure is detected. An image is picked up with -1EV achieved by decrementing 0EV of this proper exposure by "1" to obtain an underexposure image (step S102), and an image is picked up with +1EV achieved by incrementing 0EV by "1" to obtain an overexposure image (step S103). The gain of the underexposure image is increased so that the exposure condition is equal to that of the overexposure image, thereby generating a pseudo overexposure image (step S104). Dynamic body determination/α blending processing is executed by using the pseudo overexposure image and the overexposure image to generate a blending image (step S105), and the blending image and the underexposure image are subjected to pixel addition composition (step S107).

Description

  The present invention relates to an image composition device, an image composition method, and a program.

2. Description of the Related Art Conventionally, in the technical field related to imaging, a technique has been devised to eliminate lack of gradation such as whiteout or blackout in a photographed image.
For example, in the following patent document, a device has been devised that continuously captures images with different exposure amounts, appropriately selects and combines them, thereby solving problems such as overexposure and blackout and expanding the dynamic range. Yes.

JP 2003-264738 A

However, in the case of the above image composition technique, the effect is limited to the case where the main subject does not move with respect to the shooting angle of view, that is, there is no moving object. If there is a moving object at the shooting angle of view, the main subject will be misaligned during continuous shooting, making it difficult to synthesize images continuously shot with different exposure amounts.
Therefore, in such a case, there is a problem that the dynamic range cannot be expanded by the synthesis process.

  The present invention has been made in view of such a conventional problem, and an object thereof is to obtain a composite image with little overexposure and underexposure even when a moving object is present at a shooting angle of view. To do.

  In order to solve the above-mentioned problem, the invention according to claim 1 is characterized in that image acquisition means for acquiring a plurality of images having substantially the same shooting angle of view and different exposure conditions, and images of other exposure conditions in the plurality of images. And gain control means for controlling to increase the gain of the brightness of the image with the lower exposure amount among the plurality of images acquired by the image acquisition means so that the exposure conditions are equal, and the gain control means A moving body region detecting means for detecting a moving body region at the shooting angle of view by comparing the image with the increased gain and the image with the same exposure condition, and the image with the gain increased. The image of the moving object area detected by the moving object area detecting means in the image having the same exposure condition as that of the image is replaced with the image of the same area in the image having the increased gain. For the moving object region detected by the replacing means and the moving object region detecting means, the maximum value of the difference for each pixel between the image with the increased gain, the image with the increased gain, and the image with the same exposure condition is acquired. A difference acquisition unit, a blending unit that averages the information acquired by the difference acquisition unit and blends the information as transmission intensity information into the image after replacement by the replacement unit, the image blended by the blending unit, and the And generating means for generating a pixel addition combined image with an image having a lower exposure amount.

  The invention according to claim 2 is the invention according to claim 1, wherein the blending means averages the information obtained by the difference obtaining means, and executes a process for further increasing the contrast of the averaged information. It is characterized by doing.

  According to a third aspect of the invention, in the invention of the first or second aspect, the blending means averages the information acquired by the difference acquisition means, and further smoothes the averaged information. It is characterized by performing.

  The invention according to claim 4 is the invention according to any one of claims 1 to 3, wherein the image acquisition means includes an imaging means.

  The invention according to claim 5 is characterized in that, in the invention according to any one of claims 1 to 4, the image is acquired at substantially the same imaging angle of view.

  According to a sixth aspect of the present invention, the image acquisition step of acquiring a plurality of images having substantially the same shooting angle of view and different exposure conditions, and the exposure conditions of the images of other exposure conditions in the plurality of images are equal. The gain control step for controlling to increase the gain of the brightness of the image having the lower exposure amount among the plurality of images acquired by the image acquisition step, and the image whose gain has been increased by the gain control step And a moving object region detecting step for detecting a moving object region at the shooting angle of view by comparing the image with the increased gain and the image with the same exposure condition, and the exposure condition is equal to the image with the increased gain. The image of the moving object area detected by the moving object area detecting step in the image is changed to the image of the same area in the image with the gain increased. For the moving object region detected by the replacing step and the moving object region detecting step, the maximum value of the difference for each pixel between the image with the increased gain, the image with the increased gain, and the image with the same exposure condition. The difference acquisition step to be acquired, the information acquired by the difference acquisition step is averaged, and the blended step is blended to the image after the replacement by the replacement step as transmission intensity information, and the image blended by the blending step And a generation step of generating a pixel addition composite image of the image with the lower exposure amount.

  According to a sixth aspect of the present invention, there is provided an image acquisition means for acquiring a plurality of images having substantially the same shooting angle of view and different exposure conditions, and images of other exposure conditions in the plurality of images. Gain control means for controlling the gain of the image with the lower exposure amount among the plurality of images acquired by the image acquisition means so that the exposure conditions are equal, and the gain control means controls the gain. A moving object region detecting means for detecting a moving object region at the shooting angle of view by comparing the raised image with an image having the same exposure condition as that of the image having the increased gain, and the image having the increased gain and the exposure An image of the moving object region detected by the moving object region detection unit in an image having the same condition is placed on the image of the same region in the image with the gain increased. For the moving body region detected by the moving body region detecting unit, the maximum value of the difference for each pixel between the image with the increased gain, the image with the increased gain, and the image with the same exposure condition is acquired. Difference obtaining means, information obtained by the difference obtaining means is averaged, and blending means for blending the obtained information as transmission intensity information on the image after replacement by the replacing means, the image blended by the blending means and the exposure amount And generating means for generating a pixel-added combined image with the lower image.

  According to the present invention, it is possible to obtain a composite image with little overexposure and underexposure even when a moving object is present at the shooting angle of view.

It is a circuit block diagram of an imaging device to which an embodiment of the present invention is applied. It is a flowchart which shows the process sequence in the imaging device. It is a flowchart which shows the process sequence of a moving body determination and blending process. It is a figure which shows an example of the image continuously taken. (A) is a figure which shows the process result by a smoothing process, (b) is a figure which shows a process result by an emphasis process. (A) is a figure which shows the synthesized image by addition composition, (b) is a figure which shows the synthesized image by this Embodiment. It is a flowchart which shows the process sequence in other embodiment of this invention.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a circuit configuration diagram of an imaging apparatus including an image synthesis apparatus according to an embodiment of the present invention. The imaging apparatus includes an imaging unit 1, a drive control unit 2, a CDS / ADC 3, a key input unit 4, a display unit 5, an image recording unit 6, a program memory 7, a RAM 8, a control unit 9, and an image processing unit 10. Yes. These are connected via a bus line. The key input unit 4 includes a shutter key 41 for detecting a recording instruction of a photographer, and the image processing unit 10 includes a synthesis unit 11.

  In the figure, the imaging unit 1 holds the light intensity as a charge accumulation for a certain period of time by controlling from an image sensor such as a CMOS, an RGB color filter provided on the image sensor, and the drive control unit 2. The CDS / ADC 3 includes a driver that outputs these as analog imaging signals. Then, by detecting the shooting instruction of the photographer via the shutter key 41, the control unit 9, and the drive control unit 2, while changing the exposure condition (shutter speed or aperture value), underexposure, proper exposure, overexposure A plurality of images (color images) including are acquired.

  The CDS / ADC 3 is a circuit to which an analog imaging signal corresponding to the optical image of the subject output from the imaging unit 1 is input. The CDS / ADC 3 holds the input imaging signal and gain adjustment for amplifying the imaging signal. An amplifier (AGC), an A / D converter (ADC) that converts the amplified imaging signal into a digital imaging signal, and the like are configured. Note that control related to the adjustment of the gain adjustment amplifier is also performed based on an instruction from the drive control unit 2. Therefore, even if multiple images are acquired with the same exposure conditions (shutter speed or aperture value), multiple images with different conditions can be generated by sequentially changing the RGB gain adjustment amplifier and the color of the image. can do.

  In the present embodiment, the drive control unit 2 performs control related to the gain adjustment amplifier. However, the present invention is not limited to this. For example, the above control may be performed by the control unit 9.

  In addition to the shutter key 41 described above, the key input unit 4 includes various keys for detecting switching to a shooting mode for the purpose of acquiring and recording an image according to the present invention, display switching, and the like.

  The display unit 5 has a function of displaying the synthesized image. The image recording unit 6 stores and stores image data (image file) encoded by the JPEG method after the composition processing according to the present invention is executed. The program memory 7 stores a program executed by the control unit 9 and the image processing unit 10, and this program is read by the control unit 9 as necessary. The RAM 8 has a function of temporarily holding data being processed generated by each process. The control unit 9 controls the processing operation of the entire imaging apparatus. The image processing unit 10 includes a synthesizing unit 11 corresponding to the characteristic configuration of the present invention, in addition to image data encoding / decoding processing.

  That is, the synthesizing unit 11 performs synthesis of an overexposed image and a pseudo overexposed image, which will be described later, and uses a moving object determination map, which will be described later, for this synthesized image as a transmission intensity map (α map) corresponding to the difference area. This reduces the afterimage caused by. Further, the dynamic range is expanded by adding and synthesizing the synthesized image and the underexposed image with pixels.

  Next, the operation of the present embodiment will be described. When the user operates the mode button provided in the key input unit 4 to set the shooting mode according to the present embodiment, the control unit 9 reads the program from the program memory 7 and as shown in the flowchart of FIG. Start processing.

  That is, the control unit 9 determines whether or not the pressing of the shutter key 41 is detected (step S101). If the pressing of the shutter key 41 is detected, the proper exposure is detected, and the drive control unit 2 is instructed to execute photographing by −1EV, which is obtained by reducing the exposure value 0EV of the proper exposure by “1”. The underexposed image obtained by photographing at -1 EV is temporarily stored in the RAM 8 (step S102). Subsequently, the drive control unit 2 is instructed to execute photographing with + 1EV, which is obtained by increasing the exposure value 0EV of the appropriate exposure by “1”, and the overexposed image obtained by the photographing with + 1EV is temporarily stored in the RAM 8 (Step S1). S103).

  Therefore, by the processing in step S102 and step S103, the underexposed image U made up of the YUV image shown in FIG. 4A and the overexposed image O made up of the YUV image shown in FIG. 4B are stored in the RAM 8. Is done. In FIG. 4, an arrow t is a time axis, and the continuously exposed underexposed image U and overexposed image O are in a state where the hand H as a moving object is shaken up and down in the background B indicating the entire imaging field angle. Thus, continuous shooting is performed with substantially the same angle of view. In both images U and O, the background B is the same, but as the time t advances, the image of the hand H is displaced relatively downward in the background B, and the image of the hand H of the overexposed image O is underexposed. It is located below the image of the hand H of the image U.

  In the continuous shooting of these continuous shot images U and O, it is assumed that image blur due to camera shake is corrected by a known technique (view angle correction technique using CCD shift or lens shift). However, the image of the hand H in each of the continuous shot images U and O is considered to be a cause of motion blur.

  Next, by increasing the gain of the underexposed image U so that the exposure condition is equal to the overexposed image O, this pseudo overexposed image is generated and temporarily stored in the RAM 8 (step S104). That is, the underexposed image U captured at -1EV and the overexposed image O captured at + 1EV are four times brighter than the underexposed image U, and the overexposed image O is four times brighter than the underexposed image U. . Therefore, by processing the underexposed image U with a digital gain of 4 times, a pseudo overexposed image having the same exposure condition as the overexposed image O is generated.

Next, a moving object determination / α blending process is performed (step S105). FIG. 3 is a flowchart showing a processing procedure of the moving object determination / α blending process (step S105). In the description of the flowchart shown in FIG. 3, the following equations, symbols, or functions are used.
Y_Under_4, U_ Under_4, V_ Under_4 ： Pseudo overexposed image YUV
Y_Over, U_Over, V_Over: Overexposed image YUV
fMax (): Maximum value calculation function
fLpf (): smoothing function
fEmphasis (): Map enhancement function

In the flowchart of FIG. 3, the control unit 9 obtains an absolute difference value between the pseudo overexposed image and the overexposed image for each pixel of both images as shown in the following formulas [1] to [3] (step S <b> 201). .
In other words,
Diff_Y = | Y_Under_4 − Y_Mid | ・ ・ ・ Formula [1] // Difference absolute value of Y
Diff_U = | U_Under_4 − U_Mid | ・ ・ ・ Equation [2] // Difference absolute value of U
Diff_V = | V_Under_4−V_Mid | Expression (3) // Absolute absolute value of V is obtained for each pixel of the both images.

The larger the difference, the higher the possibility that the subject has moved. Therefore, as shown in the following formula [4], the difference is the largest (the difference is the largest for each corresponding pixel of both images). ) Is selected (step S202).
Diff = fMax (Diff_Y, Diff_U, Diff_V) Expression [4]

Next, as shown by the following formula [5], a process of averaging the maximum difference between the two images is executed (step S203).
Ave_Diff = (Diff [0] + Diff [1] + ... + Diff [N]) / N-1 ... Equation [5]

  Furthermore, as shown by the following formula [6], a smoothing process is executed (step S204).

Diff_Lpf = fLpf (Diff) (6)
As a result of the smoothing process in step S204, a moving object determination map M is generated as shown in FIG. In this moving object determination map M, the hatched portion is a region where α = 0 and is totally transmitted, and the white portion is a region where α = 255 and is not transmitted.

Subsequently, as shown by the following equation [7], enhancement processing is performed to increase the contrast of the moving object determination map M (step S105).
Map_Move = fEmphasis (Diff_Lpf) (7)
By the smoothing process in step S205, as shown in FIG. 5B, the contrast in the region of the image of the hand H that is a moving object in the moving object determination map M is increased.

Thereafter, using the moving object determination map M ((Map_Move [0 to 255]: 0 = no moving object) shown in FIG. 5B as an α map, the above-described pseudo overexposed image and overexposed image are obtained. Combining (step S206).
The YUV parameters (Y_result_Over, U_result_Over, V_result_Over) of each pixel of the synthesized image (hereinafter referred to as a blended image) are expressed by the following equations [8] to [10].
Y_result_Over = (Y_Over × (255 − Map_Move) + Y_Under_4 × Map_Move) / 255
... Formula [8]
U_result_Over = (U_Over × (255 − Map_Move) + U_Under _4 × Map_Move) / 255
... Formula [9]
V_result_Over = (V_Over × (255 − Map_Move) + V_Under _4 × Map_Move) / 255
... Formula [10]

  Then, in step S106 following step S105 in the flowchart of FIG. 2, the blended image is temporarily stored in the RAM 8. Subsequently, the blended image once stored in the RAM 8 and the underexposed image U are added and combined (step S107). Thereby, as shown in FIG. 6B, YUV image data representing the composite image PM composed of the background B and the hand H is generated.

Here, FIG. 6A is a composite image PN obtained by simply averaging the underexposed image U and the overexposed image O. As described above, in the composite image PN obtained by simply averaging the continuously shot images, a large blurring occurs in the image of the hand H that is a moving body image. On the other hand, according to the present embodiment, even when the image of the hand H is moving, as shown in FIG. 6B, it is possible to obtain a composite image PM with less moving body shake.
Further, even when a moving object is present at the shooting angle of view, a composite image PM with little whiteout or blackout can be obtained.

  Moreover, as described above, since the smoothing process is performed in step S204, the boundary line between the moving object (hand H) and the background by synthesis can be made more natural.

  Further, in the above-described step S205, since the process of increasing the contrast of the map M is executed, it is possible to reduce the afterimage (ghost) of the subject blur in the synthesis process.

  When the YUV image data representing the composite image PM is generated as described above, the YUV image data of the composite image PM that is the completed image is encoded and filed by the JPEG method, and is recorded and stored in the image recording unit 6. (Step S110), the process ends.

  In this embodiment, YUV image data is used, but RGB image data may be used. In this case, the difference square is similarly calculated for each of RGB.

  FIG. 7 is a flowchart showing a processing procedure in another embodiment of the present invention, in which three images taken with substantially the same angle of view, an underexposed image, a normal exposed image, and an overexposed image are used. It is. When the user operates the mode button provided in the key input unit 4 to set the shooting mode according to the present embodiment, the control unit 9 reads the program from the program memory 7 and performs processing as shown in this flowchart. To start.

  That is, the control unit 9 determines whether or not the pressing of the shutter key 41 is detected (step S301). If the pressing of the shutter key 41 is detected, the appropriate exposure is detected, and the appropriate exposure value 0EV is decreased by “2”. The underexposed image obtained by photographing at −2 EV is temporarily stored in the RAM 8 (step S302).

  Further, the drive control unit 2 is instructed to execute photographing at an exposure value 0 EV of appropriate exposure, and the exposure normal image obtained by photographing at this 0 EV is temporarily stored in the RAM 8 (step S303). Furthermore, the drive control unit 2 is instructed to perform shooting at + 2EV, which is obtained by increasing the exposure value 0EV of the appropriate exposure by “2”, and the overexposed image obtained by the shooting at + 2EV is temporarily stored in the RAM 8 (step S1). S304).

  Therefore, an underexposed image, a normal exposed image, and an overexposed image made up of YUV images are stored in the RAM 8 by the processing in steps S302 and S303.

  Next, by increasing the gain of the underexposed image so that the exposure condition is equal to the normal exposure image, a first pseudo overexposure image is generated, and the overexposure image and the exposure condition are equalized. By increasing the gain of the underexposed image, a second pseudo overexposed image is generated and temporarily stored in the RAM 8 (step S305). That is, the underexposure image captured at −2 EV and the dew normal image captured at 0 EV are 8 times different in brightness, and the normal exposure image is 8 times brighter than the underexposure image. Accordingly, by processing the underexposed image with a digital gain of 8 times, a first pseudo overexposed image having the same exposure condition as that of the normal exposure image is generated.

  Also, the underexposed image captured at -2EV and the overexposed image captured at + 2EV are 16 times brighter than the underexposed image, and the overexposed image is 16 times brighter than the underexposed image. Therefore, by processing the underexposed image with a digital gain of 16 times, a second pseudo overexposed image having the same exposure condition as the overexposed image is generated. The first pseudo overexposure image and the second pseudo overexposure image are temporarily stored in the RAM 8.

  Next, it is determined whether or not the first moving object determination / blending process has been completed (step S306). If the moving object determination / blending process has not yet been executed and this time is the first process, the determination in step S306 is NO, and the process proceeds to step S307. Then, the moving object determination / α blending process is performed using the first pseudo overexposure image and the normal exposure image having the same pseudo exposure value. The processing procedure of the moving object determination / α blending process (step S307) is the same as that of the embodiment described based on the flowchart of FIG. When the moving object determination / α blending process using the first pseudo overexposed image and the normal exposure image is completed, the first blended image obtained thereby is temporarily stored in the RAM 8 (step S308), and the process proceeds to step S306. Return.

  Then, since the first moving object determination / α blending process is completed, the determination in step S306 becomes YES. Accordingly, the process proceeds to step S309, and the moving object determination / α blending process is performed using the second pseudo overexposure image and the overexposure image that have the same pseudo exposure value. The processing procedure of the moving object determination / α blending process (step S309) is also the same as that of the embodiment described based on the flowchart of FIG. When the moving object determination / α blending process using the second pseudo overexposed image and the overexposed image is completed, the second blended image obtained thereby is temporarily stored in the RAM 8 (step S310).

  Subsequently, the first and second blended images once stored in the RAM 8 and the underexposed image are subjected to pixel addition synthesis to generate YUV image data representing the synthesized image (step S311). Thereafter, the YUV image data of the composite image PM, which is a completed image, is encoded and filed by the JPEG method, recorded and saved in the image recording unit 6 (step S312), and the process is terminated.

  In this embodiment, in step S311, the first blended image, the second blended image, and the underexposed image are subjected to pixel addition synthesis to obtain a completed image. However, a completed image may be obtained by adding and synthesizing only the first blended image and the underexposed image, or a completed image may be obtained by adding and synthesizing the second blended image and the underexposed image.

1 Imaging unit 2 Drive control unit 3 CDS / ADC
4 Key input section 5 Display section 6 Image recording section 7 Program memory 8 RAM
9 Control unit 10 Image processing unit 11 Composition unit 41 Shutter key

Claims (7)

Image acquisition means for acquiring a plurality of images having substantially the same shooting angle of view and different exposure conditions;
Control is performed to increase the brightness gain of the image with the lower exposure amount among the plurality of images acquired by the image acquisition unit so that the exposure condition is equal to the image of the other exposure conditions in the plurality of images. Gain control means;
A moving object region detecting means for detecting a moving object region at the shooting angle of view by comparing an image whose gain has been increased by the gain control unit with an image having the same exposure condition as that of the image having the increased gain;
Replacement means for replacing the image of the moving object region detected by the moving object region detection means in the image having the same exposure condition as that of the image with the increased gain with an image of the same area in the image with the increased gain;
Difference acquisition means for acquiring a maximum value of a difference for each pixel between the image with the increased gain, the image with the increased gain, and an image with the same exposure condition for the moving object region detected by the moving object region detection means; ,
A blending unit that averages the information acquired by the difference acquisition unit and blends the information as transmission intensity information into the image after replacement by the replacement unit;
Generating means for generating a pixel addition composite image of the image blended by the blending means and the image having the lower exposure amount;
An image composition device comprising:   2. The image synthesizing apparatus according to claim 1, wherein the blending means averages the information acquired by the difference acquisition means, and executes a process for further increasing the contrast of the averaged information.   The image blending apparatus according to claim 1, wherein the blending unit averages the information acquired by the difference acquisition unit, and further executes a smoothing process on the averaged information.   The image synthesizing apparatus according to claim 1, wherein the image acquisition unit includes an imaging unit.   The image synthesizing apparatus according to claim 1, wherein the image is acquired at substantially the same imaging angle of view. An image acquisition step of acquiring a plurality of images having substantially the same shooting angle of view and different exposure conditions;
Control is performed to increase the brightness gain of the image with the lower exposure amount among the plurality of images acquired by the image acquisition step so that the exposure condition is equal to the image of the other exposure conditions in the plurality of images. A gain control step;
A moving object region detecting step for detecting a moving object region at the shooting angle of view by comparing an image whose gain has been increased by the gain control step with an image having the same exposure condition as that of the image having the increased gain;
A replacement step of replacing an image of the moving object region detected by the moving object region detection step in an image having the same exposure condition as that of the image having the increased gain with an image of the same region in the image having the increased gain;
A difference obtaining step for obtaining a maximum value of a difference for each pixel between the image with the increased gain, the image with the increased gain, and an image with the same exposure condition for the moving object region detected by the moving object region detecting step; ,
A blending step of averaging the information acquired by the difference acquisition step, and blending the information as transmission intensity information to the image after replacement by the replacement step;
A generation step of generating a pixel addition composite image of the image blended by the blending step and the image with the lower exposure amount;
An image composition method comprising: The computer that the device has
Image acquisition means for acquiring a plurality of images having substantially the same shooting angle of view and different exposure conditions;
Control is performed to increase the brightness gain of the image with the lower exposure amount among the plurality of images acquired by the image acquisition unit so that the exposure condition is equal to the image of the other exposure conditions in the plurality of images. Gain control means,
A moving object region detecting means for detecting a moving object region at the shooting angle of view by comparing an image whose gain has been increased by the gain control unit with an image having the same exposure condition as that of the image having the increased gain;
Replacement means for replacing the image of the moving object region detected by the moving object region detection means in the image with the same exposure condition as that of the image with the increased gain, with the image of the same area in the image with the increased gain;
A difference acquisition unit that acquires a maximum value of a difference for each pixel between the image with the increased gain and the image with the increased gain and an image having the same exposure condition for the moving region detected by the moving region detection unit;
A blending unit that averages the information acquired by the difference acquisition unit and blends the information as transmission intensity information into the image after replacement by the replacement unit;
Generating means for generating a pixel-added synthesized image of the image blended by the blending means and the image with the lower exposure amount;
A program characterized by functioning as

Images