JP5146500B2 - Image composition apparatus, image composition method, and program - Google Patents

Description

  The present invention relates to an image composition device, an image composition method, and a program.

2. Description of the Related Art Conventionally, in the technical field related to imaging, a technique has been devised to eliminate lack of gradation such as whiteout or blackout in a photographed image.
For example, in the following patent document, a device has been devised that continuously captures images with different exposure amounts, appropriately selects and combines them, thereby solving problems such as overexposure and blackout and expanding the dynamic range. Yes.

JP 2003-264738 A

However, in the case of the above image composition technique, the effect is limited to the case where the main subject does not move with respect to the shooting angle of view, that is, there is no moving object. If there is a moving object at the shooting angle of view, the main subject will be misaligned during continuous shooting, making it difficult to synthesize images continuously shot with different exposure amounts.
Therefore, in such a case, there is a problem that the dynamic range cannot be expanded by the synthesis process.

  The present invention has been made in view of such a conventional problem, and an object thereof is to obtain a composite image with little overexposure and underexposure even when a moving object is present at a shooting angle of view. To do.

In order to solve the above-mentioned problem, the invention according to claim 1 is characterized in that image acquisition means for acquiring a plurality of images having substantially the same shooting angle of view and different exposure conditions, and images of other exposure conditions in the plurality of images, Gain control means for controlling the gain of the image with the lower exposure amount among the plurality of images acquired by the image acquisition means so that the exposure conditions are equal, and the gain by the gain control means an image that has been raised, the image and the exposure condition image the gain is raised is equal compared for each pixel, a motion region detection means for detecting a region in which the difference is present as the moving object region in the photographing field angle, this Using the moving object region detected by the moving object region detection means as the non-transparent region, the image with the increased gain and the image with the same exposure condition as the image with the increased gain are blocked. And blending means for bindings, characterized by comprising a generating means for generating a pixel addition composite image of the person in the image as low as blending images with the exposure amount by the blending unit.

According to a second aspect of the present invention, in the first aspect of the invention, the blending means averages the maximum value of the difference value for each pixel of the area detected by the moving object area detecting means , and this averaging is performed. It is characterized in that a process for further increasing the contrast is executed for the difference value .

According to a third aspect of the present invention, in the first or second aspect of the present invention, the image acquisition means includes an imaging means.

The invention according to claim 4 is the invention according to any one of claims 1 to 3 , characterized in that the image is acquired at substantially the same angle of view.

According to a fifth aspect of the present invention, there is provided an image acquisition step of acquiring a plurality of images having substantially the same shooting angle of view and different exposure conditions, and an exposure condition equal to an image of another exposure condition in the plurality of images. The gain control step for controlling to increase the gain of the brightness of the image having the lower exposure amount among the plurality of images acquired by the image acquisition step, and the image whose gain has been increased by the gain control step And a moving object region detecting step for comparing the image with the increased gain with an image having the same exposure condition, and detecting a region where the difference exists as a moving object region at the shooting angle of view, and the moving object region detecting step. The moving object region is set as a non-transparent region, and the image with the increased gain and the image with the same exposure condition as the image with the increased gain are blocked. A blending step of bindings, characterized in that it comprises a generation step of generating a pixel addition composite image of the images of the lower and blending images with the exposure amount by this blending step.

According to a sixth aspect of the present invention, there is provided an image acquisition means for acquiring a plurality of images having substantially the same shooting angle of view and different exposure conditions, and images of other exposure conditions in the plurality of images. Gain control means for controlling the gain of the image with the lower exposure amount among the plurality of images acquired by the image acquisition means so that the exposure conditions are equal, and the gain control means controls the gain. The moving object region detecting means for comparing the raised image with the image having the increased gain and the image having the same exposure condition for each pixel, and detecting a region where a difference exists as a moving object region at the shooting angle of view, and the moving object The moving object area detected by the area detecting means is set as a non-transparent area, and the image with the same gain as the image with the increased gain and the image with the same exposure condition. Blending means for blending the, characterized in that to function as generating means, for generating a pixel addition composite image of the images of the lower and blending images with the exposure amount by the blending unit.

  According to the present invention, it is possible to obtain a composite image with little overexposure and underexposure even when a moving object is present at the shooting angle of view.

It is a circuit block diagram of an imaging device to which an embodiment of the present invention is applied. 3 is a flowchart illustrating a processing procedure in the imaging apparatus. It is a flowchart which shows the process sequence of a moving body determination and blending process. It is a figure which shows an example of the image continuously taken. (A) is a figure which shows the process result by a smoothing process, (b) is a figure which shows a process result by an emphasis process. (A) is a figure which shows the synthesized image by addition composition, (b) is a figure which shows the synthesized image by this Embodiment. It is a flowchart which shows the process sequence in other embodiment of this invention.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a circuit configuration diagram of an imaging apparatus including an image synthesis apparatus according to an embodiment of the present invention. The imaging apparatus includes an imaging unit 1, a drive control unit 2, a CDS / ADC 3, a key input unit 4, a display unit 5, an image recording unit 6, a program memory 7, a RAM 8, a control unit 9, and an image processing unit 10. Yes. These are connected via a bus line. The key input unit 4 includes a shutter key 41 for detecting a recording instruction of a photographer, and the image processing unit 10 includes a synthesis unit 11.

  In the figure, the imaging unit 1 holds the light intensity as a charge accumulation for a certain period of time by controlling from an image sensor such as a CMOS, an RGB color filter provided on the image sensor, and the drive control unit 2. The CDS / ADC 3 includes a driver that outputs these as analog imaging signals. Then, by detecting the shooting instruction of the photographer via the shutter key 41, the control unit 9, and the drive control unit 2, while changing the exposure condition (shutter speed or aperture value), underexposure, proper exposure, overexposure A plurality of images (color images) including are acquired.

  The CDS / ADC 3 is a circuit to which an analog imaging signal corresponding to the optical image of the subject output from the imaging unit 1 is input. The CDS / ADC 3 holds the input imaging signal and gain adjustment for amplifying the imaging signal. An amplifier (AGC), an A / D converter (ADC) that converts the amplified imaging signal into a digital imaging signal, and the like are configured. Note that control related to the adjustment of the gain adjustment amplifier is also performed based on an instruction from the drive control unit 2. Therefore, even if multiple images are acquired with the same exposure conditions (shutter speed or aperture value), multiple images with different conditions can be generated by sequentially changing the RGB gain adjustment amplifier and the color of the image. can do.

  In the present embodiment, the drive control unit 2 performs control related to the gain adjustment amplifier. However, the present invention is not limited to this. For example, the above control may be performed by the control unit 9.

  In addition to the shutter key 41 described above, the key input unit 4 includes various keys for detecting switching to a shooting mode for the purpose of acquiring and recording an image according to the present invention, display switching, and the like.

  The display unit 5 has a function of displaying the synthesized image. The image recording unit 6 stores and stores image data (image file) encoded by the JPEG method after the composition processing according to the present invention is executed. The program memory 7 stores a program executed by the control unit 9 and the image processing unit 10, and this program is read by the control unit 9 as necessary. The RAM 8 has a function of temporarily holding data being processed generated by each process. The control unit 9 controls the processing operation of the entire imaging apparatus. The image processing unit 10 includes a synthesizing unit 11 corresponding to the characteristic configuration of the present invention, in addition to image data encoding / decoding processing.

  That is, the synthesizing unit 11 performs synthesis of an overexposed image and a pseudo overexposed image, which will be described later, and uses a moving object determination map, which will be described later, for this synthesized image as a transmission intensity map (α map) corresponding to the difference area. This reduces the afterimage caused by. Further, the dynamic range is expanded by adding and synthesizing the synthesized image and the underexposed image with pixels.

  Next, the operation of the present embodiment will be described. When the user operates the mode button provided in the key input unit 4 to set the shooting mode according to the present embodiment, the control unit 9 reads the program from the program memory 7 and as shown in the flowchart of FIG. Start processing.

  That is, the control unit 9 determines whether or not the pressing of the shutter key 41 is detected (step S101). If the pressing of the shutter key 41 is detected, the proper exposure is detected, and the drive control unit 2 is instructed to execute photographing by −1EV, which is obtained by reducing the exposure value 0EV of the proper exposure by “1”. The underexposed image obtained by photographing at -1 EV is temporarily stored in the RAM 8 (step S102). Subsequently, the drive control unit 2 is instructed to execute photographing with + 1EV, which is obtained by increasing the exposure value 0EV of the appropriate exposure by “1”, and the overexposed image obtained by the photographing with + 1EV is temporarily stored in the RAM 8 (Step S1). S103).

Therefore, by the processing in step S102 and step S103, the underexposed image U made up of the YUV image shown in FIG. 4A and the overexposed image O made up of the YUV image shown in FIG. 4B are stored in the RAM 8. Is done . The continuously exposed underexposed image U and overexposed image O are obtained by continuously shooting at substantially the same imaging angle of view while shaking the hand H as a moving object up and down in the background B showing the entire imaging angle of view. It is. In both images U and O, the background B is the same, but as the time t advances, the image of the hand H is displaced relatively downward in the background B, and the image of the hand H of the overexposed image O is underexposed. It is located below the image of the hand H of the image U.

  In the continuous shooting of these continuous shot images U and O, it is assumed that image blur due to camera shake is corrected by a known technique (view angle correction technique using CCD shift or lens shift). However, the image of the hand H in each of the continuous shot images U and O is considered to be a cause of motion blur.

  Next, by increasing the gain of the underexposed image U so that the exposure condition is equal to the overexposed image O, this pseudo overexposed image is generated and temporarily stored in the RAM 8 (step S104). That is, the underexposed image U captured at -1EV and the overexposed image O captured at + 1EV are four times brighter than the underexposed image U, and the overexposed image O is four times brighter than the underexposed image U. . Therefore, by processing the underexposed image U with a digital gain of 4 times, a pseudo overexposed image having the same exposure condition as the overexposed image O is generated.

Next, a moving object determination / α blending process is performed (step S105). FIG. 3 is a flowchart showing a processing procedure of the moving object determination / α blending process (step S105). In the description of the flowchart shown in FIG. 3, the following equations, symbols, or functions are used.
Y_Under_4, U_ Under_4, V_ Under_4 ： Pseudo overexposed image YUV
Y_Over, U_Over, V_Over: Overexposed image YUV
fMax (): Maximum value calculation function
fLpf (): smoothing function
fEmphasis (): Map enhancement function

In the flowchart of FIG. 3, the control unit 9 obtains an absolute difference value between the pseudo overexposed image and the overexposed image for each pixel of both images as shown in the following formulas [1] to [3] (step S <b> 201). .
In other words,
Diff_Y = | Y_Under_4 − Y_Mid | ・ ・ ・ Formula [1] // Difference absolute value of Y
Diff_U = | U_Under_4 − U_Mid | ・ ・ ・ Equation [2] // Difference absolute value of U
Diff_V = | V_Under_4−V_Mid | Expression [3] // The absolute difference value of V is obtained for each pixel of both images.

The larger the difference, the higher the possibility that the subject has moved. Therefore, as shown in the following formula [4], the difference is the largest (the difference is the largest for each corresponding pixel of both images). ) Is selected (step S202).
Diff = fMax (Diff_Y, Diff_U, Diff_V) Expression [4]

Next, as shown by the following formula [5], a process of averaging the maximum difference between the two images is executed (step S203).
Ave_Diff = (Diff [0] + Diff [1] + ... + Diff [N]) / N-1 ... Equation [5]

  Furthermore, as shown by the following formula [6], a smoothing process is executed (step S204).

Diff_Lpf = fLpf (Diff) (6)
As a result of the smoothing process in step S204, a moving object determination map M is generated as shown in FIG. In this moving object determination map M, the hatched portion is a region where α = 0 and is totally transmitted, and the white portion is a region where α = 255 and is not transmitted.

Subsequently, as shown by the following equation [7], enhancement processing is performed to increase the contrast of the moving object determination map M (step S205 ).
Map_Move = fEmphasis (Diff_Lpf) (7)
By the smoothing process in step S205, as shown in FIG. 5B, the contrast in the region of the image of the hand H that is a moving object in the moving object determination map M is increased.

Thereafter, using the moving object determination map M ((Map_Move [0 to 255]: 0 = no moving object) shown in FIG. 5B as an α map, the above-described pseudo overexposed image and overexposed image are obtained. Combining (step S206).
The YUV parameters (Y_result_Over, U_result_Over, V_result_Over) of each pixel of the synthesized image (hereinafter referred to as a blended image) are expressed by the following equations [8] to [10].
Y_result_Over = (Y_Over × (255 − Map_Move) + Y_Under_4 × Map_Move) / 255
... Formula [8]
U_result_Over = (U_Over × (255 − Map_Move) + U_Under _4 × Map_Move) / 255
... Formula [9]
V_result_Over = (V_Over × (255 − Map_Move) + V_Under _4 × Map_Move) / 255
... Formula [10]

  Then, in step S106 following step S105 in the flowchart of FIG. 2, the blended image is temporarily stored in the RAM 8. Subsequently, the blended image once stored in the RAM 8 and the underexposed image U are added and combined (step S107). Thereby, as shown in FIG. 6B, YUV image data representing the composite image PM composed of the background B and the hand H is generated.

Here, FIG. 6A is a composite image PN obtained by simply averaging the underexposed image U and the overexposed image O. As described above, in the composite image PN obtained by simply averaging the continuously shot images, a large blurring occurs in the image of the hand H that is a moving body image. On the other hand, according to the present embodiment, even when the image of the hand H is moving, as shown in FIG. 6B, it is possible to obtain a composite image PM with less moving body shake.
Further, even when a moving object is present at the shooting angle of view, a composite image PM with little whiteout or blackout can be obtained.

  Moreover, as described above, since the smoothing process is performed in step S204, the boundary line between the moving object (hand H) and the background by synthesis can be made more natural.

  Further, in the above-described step S205, since the process of increasing the contrast of the map M is executed, it is possible to reduce the afterimage (ghost) of the subject blur in the synthesis process.

When the YUV image data representing the composite image PM is generated as described above, the YUV image data of the composite image PM that is the completed image is encoded and filed by the JPEG method, and is recorded and stored in the image recording unit 6. (Step S108 ), the process is terminated.

  In this embodiment, YUV image data is used, but RGB image data may be used. In this case, the difference square is similarly calculated for each of RGB.

  FIG. 7 is a flowchart showing a processing procedure in another embodiment of the present invention, in which three images taken with substantially the same angle of view, an underexposed image, a normal exposed image, and an overexposed image are used. It is. When the user operates the mode button provided in the key input unit 4 to set the shooting mode according to the present embodiment, the control unit 9 reads the program from the program memory 7 and performs processing as shown in this flowchart. To start.

  That is, the control unit 9 determines whether or not the pressing of the shutter key 41 is detected (step S301). If the pressing of the shutter key 41 is detected, the appropriate exposure is detected, and the appropriate exposure value 0EV is decreased by “2”. The underexposed image obtained by photographing at −2 EV is temporarily stored in the RAM 8 (step S302).

  Further, the drive control unit 2 is instructed to execute photographing at an exposure value 0 EV of appropriate exposure, and the exposure normal image obtained by photographing at this 0 EV is temporarily stored in the RAM 8 (step S303). Further, the drive control unit 2 is instructed to perform photographing at + 2EV, which is obtained by increasing the exposure value 0EV of the appropriate exposure by “2”, and the overexposed image obtained by photographing at + 2EV is temporarily stored in the RAM 8 (step S1). S304).

  Therefore, an underexposed image, a normal exposed image, and an overexposed image made up of YUV images are stored in the RAM 8 by the processing in steps S302 and S303.

  Next, by increasing the gain of the underexposed image so that the exposure condition is equal to the normal exposure image, a first pseudo overexposure image is generated, and the overexposure image and the exposure condition are equalized. By increasing the gain of the underexposed image, a second pseudo overexposed image is generated and temporarily stored in the RAM 8 (step S305). That is, the underexposure image captured at −2 EV and the dew normal image captured at 0 EV are 8 times different in brightness, and the normal exposure image is 8 times brighter than the underexposure image. Accordingly, by processing the underexposed image with a digital gain of 8 times, a first pseudo overexposed image having the same exposure condition as that of the normal exposure image is generated.

  Also, the underexposed image captured at -2EV and the overexposed image captured at + 2EV are 16 times brighter than the underexposed image, and the overexposed image is 16 times brighter than the underexposed image. Therefore, by processing the underexposed image with a digital gain of 16 times, a second pseudo overexposed image having the same exposure condition as the overexposed image is generated. The first pseudo overexposure image and the second pseudo overexposure image are temporarily stored in the RAM 8.

  Next, it is determined whether or not the first moving object determination / blending process has been completed (step S306). If the moving object determination / blending process has not yet been executed and this time is the first process, the determination in step S306 is NO, and the process proceeds to step S307. Then, the moving object determination / α blending process is performed using the first pseudo overexposure image and the normal exposure image having the same pseudo exposure value. The processing procedure of the moving object determination / α blending process (step S307) is the same as that of the embodiment described based on the flowchart of FIG. When the moving object determination / α blending process using the first pseudo overexposed image and the normal exposure image is completed, the first blended image obtained thereby is temporarily stored in the RAM 8 (step S308), and the process proceeds to step S306. Return.

  Then, since the first moving object determination / α blending process is completed, the determination in step S306 becomes YES. Accordingly, the process proceeds to step S309, and the moving object determination / α blending process is performed using the second pseudo overexposure image and the overexposure image that have the same pseudo exposure value. The processing procedure of the moving object determination / α blending process (step S309) is also the same as that of the embodiment described based on the flowchart of FIG. When the moving object determination / α blending process using the second pseudo overexposed image and the overexposed image is completed, the second blended image obtained thereby is temporarily stored in the RAM 8 (step S310).

  Subsequently, the first and second blended images once stored in the RAM 8 and the underexposed image are subjected to pixel addition synthesis to generate YUV image data representing the synthesized image (step S311). Thereafter, the YUV image data of the composite image PM, which is a completed image, is encoded and filed by the JPEG method, recorded and saved in the image recording unit 6 (step S312), and the process is terminated.

  In this embodiment, in step S311, the first blended image, the second blended image, and the underexposed image are subjected to pixel addition synthesis to obtain a completed image. However, a completed image may be obtained by adding and synthesizing only the first blended image and the underexposed image, or a completed image may be obtained by adding and synthesizing the second blended image and the underexposed image.

1 Imaging unit 2 Drive control unit 3 CDS / ADC
4 Key input section 5 Display section 6 Image recording section 7 Program memory 8 RAM
9 Control unit 10 Image processing unit 11 Composition unit 41 Shutter key

Claims (6)

Image acquisition means for acquiring a plurality of images having substantially the same shooting angle of view and different exposure conditions;
Control is performed to increase the brightness gain of the image with the lower exposure amount among the plurality of images acquired by the image acquisition unit so that the exposure condition is equal to the image of the other exposure conditions in the plurality of images. Gain control means;
And image raised gain by the gain control means, and an image with the exposure condition image the gain is raised is equal compared for each pixel, detects an area where the difference is present as the moving object region in the photographing field angle Moving object region detection means ;
Blending means for blending the image with the increased gain and the image with the same exposure condition with the gain increased, with the moving object area detected by the moving object area detecting means as the opaque area ;
Generating means for generating a pixel addition composite image of the image blended by the blending means and the image having the lower exposure amount;
An image composition device comprising: The blending means averages the maximum value of the difference values for each pixel in the area detected by the moving object area detection means, and executes a process of further increasing the contrast for the averaged difference value. The image composition device according to claim 1. The image synthesizing apparatus according to claim 1 , wherein the image acquisition unit includes an imaging unit . The image synthesizing apparatus according to any one of claims 1 to 3, wherein the image is acquired at substantially the same imaging angle of view . An image acquisition step of acquiring a plurality of images having substantially the same shooting angle of view and different exposure conditions;
Control is performed to increase the brightness gain of the image with the lower exposure amount among the plurality of images acquired by the image acquisition step so that the exposure condition is equal to the image of the other exposure conditions in the plurality of images. A gain control step;
A moving object region detection that compares an image whose gain has been increased by the gain control step with an image having the same exposure condition as that of the image whose gain has been increased, and detects an area where a difference exists as a moving object region at the shooting angle of view. Steps,
A blending step for blending an image with an increased gain and an image with the same exposure condition as the gain increased with the moving body region detected by the moving body region detecting step as an opaque region;
A generation step of generating a pixel addition composite image of the image blended by the blending step and the image with the lower exposure amount;
An image composition method comprising: The computer that the device has
Image acquisition means for acquiring a plurality of images having substantially the same shooting angle of view and different exposure conditions;
Control is performed to increase the brightness gain of the image with the lower exposure amount among the plurality of images acquired by the image acquisition unit so that the exposure condition is equal to the image of the other exposure conditions in the plurality of images. Gain control means,
The image with the gain increased by the gain control means and the image with the same exposure condition as that of the image with the increased gain are compared for each pixel, and an area where there is a difference is detected as a moving object area at the shooting angle of view. Moving body region detection means,
Blending means for blending an image with an increased gain and an image with the same exposure condition as the gain increased with the moving body area detected by the moving body area detecting means as an opaque area;
Generating means for generating a pixel-added synthesized image of the image blended by the blending means and the image with the lower exposure amount;
A program characterized by functioning as

Images