JP6118118B2 - Imaging apparatus and control method thereof - Google Patents

Description

  The present invention relates to an imaging apparatus and a control method thereof, and more particularly to an imaging apparatus having a function of combining a plurality of images and a control method thereof.

  2. Description of the Related Art An imaging device equipped with HDR (High Dynamic Range) technology that expands a dynamic range of an image by generating a single image by combining a plurality of continuously captured image outputs with different exposure amounts is known. . The HDR technology can be applied not only to still images but also to moving images. However, since one frame is generated from a plurality of images, the recording frame rate is lower than the shooting frame rate. For example, in the case of a shooting frame rate of 60 FPS, when two frames of images shot alternately with a long exposure time and a short exposure time are combined to generate a recorded image of one frame, an HDR video with a recording frame rate of 30 FPS may be generated. Is possible. Such an HDR technique that combines a plurality of images taken continuously is referred to as inter-plane HDR in this specification.

  On the other hand, by performing readout by varying the exposure time (charge accumulation time) for each predetermined unit area (for example, row), a plurality of images with different exposure amounts can be obtained by shooting for one frame, and the dynamic range can be increased. A technique for obtaining an enlarged image has been proposed (Patent Document 1). This technique is also common to the inter-plane HDR in that a plurality of images are used, but since a plurality of images are obtained from pixel information obtained by shooting for one frame, this technique is referred to as in-plane HDR.

  In the in-plane HDR, images with different exposure amounts are obtained by photographing one frame. Therefore, when the photographer or the main subject is moving, images between a plurality of images used for composition compared to the in-plane HDR. It can be said that the amount of blurring is small and advantageous.

JP 2012-105225 A

  However, in the in-plane HDR, since two images are generated by shooting for one screen (one frame), the resolution of each image is lower than that obtained by normal shooting. For example, when shooting is performed by switching the charge accumulation time in two stages, long and short, in units of rows, the vertical resolution of an image obtained from a row having a long charge accumulation time and an image obtained from a row having a short charge accumulation time can be obtained by normal photographing. It becomes 1/2 of the image. Therefore, if pixel interpolation or the like is not performed, the maximum resolution of the composite image that can be obtained by the in-plane HDR is lower than the maximum resolution of the composite image that can be obtained by the inter-plane HDR. In addition, since the resolution of the source image is low, depending on the luminance level of the subject, the resolution of the post-combination image may be lower than that obtained with the inter-plane HDR.

  As described above, the inter-plane HDR and the in-plane HDR have advantages and disadvantages, respectively, but a technique for appropriately switching between them has not been proposed. The present invention has been made in view of the above-described problems of the prior art, and an object thereof is to provide an imaging apparatus capable of appropriately switching between the in-plane HDR and the in-plane HDR and a control method thereof. .

The above-described object is an image pickup apparatus, which combines an image pickup device capable of setting a charge accumulation time for each predetermined unit area and a plurality of images obtained by shooting under different exposure conditions to form one image Synthesis means for generating an image, acquisition means for acquiring a composite ratio of a plurality of images related to a predetermined region of the composite image, and control means for controlling the operation of the image sensor in accordance with the composite ratio acquired by the acquisition means And (a) when it is determined that there is a bias in the composite ratio, a plurality of frames are photographed by photographing a plurality of frames using the first exposure control for setting a charge accumulation time common to the image sensor. The operation of the image sensor is controlled so as to obtain an image, and (b) second exposure control is used to set a plurality of different charge accumulation times according to unit areas when it is not determined that the composition ratio is biased . For shooting one frame Ri, controls the operation of the image sensor so that a plurality of images are obtained, are achieved by an imaging apparatus, characterized in that.

  With such a configuration, according to the present invention, it is possible to obtain an imaging apparatus capable of appropriately switching between the in-plane HDR and the in-plane HDR and a control method thereof.

1 is a block diagram of a moving image capturing apparatus according to a first embodiment. Flow chart of HDR moving image recording in the moving image capturing apparatus of the first embodiment Flowchart of inter-plane HDR moving image recording in the moving image capturing apparatus of the first embodiment Flow chart of in-plane HDR moving image recording in moving image imaging apparatus of first embodiment Example of exposure / reading control of in-plane HDR moving image recording in moving image imaging apparatus of first embodiment Display example of captured HDR image Flow chart of HDR moving image recording in the moving image capturing apparatus of the second embodiment

(First embodiment)
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.
FIG. 1 is a block diagram illustrating a functional configuration example of an imaging apparatus according to the first embodiment of the present invention. Here, the imaging apparatus 100 is assumed to be a digital still camera having a moving image shooting function. However, the present invention can be applied to an arbitrary imaging apparatus having a moving image shooting function and an arbitrary electronic device including such an imaging apparatus. . Non-limiting examples of such electronic devices include game machines, mobile phones, personal computers, tablet terminals, drive recorders, and the like. For convenience, the imaging apparatus 100 is described as a lens-integrated type, but a configuration in which a lens can be replaced may be used.

  The imaging apparatus 100 includes an imaging lens including a zoom lens 110, a shift lens 111 that corrects apparatus shake, and a focus lens 112 for focus adjustment. The mechanical shutter 113 and the diaphragm 114 adjust the exposure amount of the image sensor 115. The timing generator 116 generates timing pulses necessary for driving the imaging element 115 and sampling.

  The image processing unit 117 performs various types of image processing such as various types of processing on the image signal output from the image sensor 115, compression encoding and decoding processing, AF evaluation value calculation, and HDR image generation. In FIG. 1, an image processing unit 117 has the following circuits. An SSG circuit that generates a synchronization signal used to drive the image sensor 115, a preprocessing circuit that applies preprocessing to an output image signal of the image sensor 115, an AF evaluation value calculation circuit, a luminance integration circuit, a signal processing circuit, an image synthesis circuit, Reduction circuit, raster block conversion circuit, and compression circuit. These various circuits may be realized by dedicated hardware or may be realized by software.

  The SSG circuit receives the drive clock for the image sensor 115 from the timing generator 116, generates horizontal and vertical synchronization signals, and outputs them to the timing generator 116 and the image sensor 115. The preprocessing circuit distributes the image signal to the luminance integration circuit and the signal processing circuit in units of rows. The preprocessing circuit further performs data correction between channels.

  The AF evaluation value calculation circuit performs horizontal filtering on the luminance component of the image signal in the set focus detection areas (AF areas), and selects the maximum value while extracting a predetermined frequency representing contrast. Then, integral calculation is performed in the vertical direction. In the luminance integration circuit, luminance components are mixed and generated from RGB signals, the input image is divided into a plurality of regions, and a luminance component is generated for each region.

  In the signal processing circuit, the image signal output from the image sensor 115 is subjected to color carrier removal, aperture correction, gamma correction processing, and the like to generate a luminance signal (Y), color interpolation, matrix conversion, gamma processing, gain adjustment, etc. To generate a color difference signal (UV). The signal processing circuit writes the generated YUV format image data in the memory unit 123. The signal processing circuit adds up the luminance signals (Y) of the generated YUV format image data for each level to generate luminance distribution data for each image data.

  The image composition circuit acquires a plurality of RGB image data stored in the memory unit 123 or YUV format image data for the captured image and multiplies them by a coefficient set in units of pixels. Thus, a composite image is generated.

  The reduction circuit receives the output of the signal processing circuit, cuts out input pixel data, performs thinning, linear interpolation processing, and the like, and performs pixel data reduction processing in the horizontal and vertical directions. The raster block conversion circuit converts the raster scan image data scaled by the reduction circuit into block scan image data. Such a series of image processing is realized by using the memory unit 123 as a buffer memory. The compression circuit compresses the YUV image data converted into the block scan data by the buffer memory according to the moving image compression method, and outputs a moving image bit stream.

  The exposure control unit 118 controls the operation of the mechanical shutter 113 and the diaphragm 114 according to the control of the system control unit 121. The lens control unit 119 controls the magnification of the zoom lens 110 and the position of the focus lens 112 according to the control of the system control unit 121, and forms an image of the subject included in the field of view of the zoom lens 110 on the image sensor 115. The acceleration sensor 181 detects the movement of the imaging device 100 and outputs it to the system control unit 121. Based on the output of the acceleration sensor 181, the system control unit 121 drives the shift lens 111 through the lens control unit 119 in a direction that cancels the movement of the imaging device 100, and optically corrects the image blur.

  The release switch 120 is included in the input device group included in the imaging apparatus 100 and inputs a recording start instruction to the system control unit 121. The system control unit 121 includes, for example, a CPU and its interface circuit, a DMAC (Direct Memory Access Controller), a bus arbiter, and the like, and controls the entire operation of the imaging apparatus 100. A program executed by the CPU of the system control unit 121 is stored in a flash memory 122 as an example of a nonvolatile storage device. The memory unit 123 includes a DRAM or the like, and is used for developing a program executed by the CPU of the system control unit 121 or temporarily storing data being processed by the CPU.

  The interface unit 124 is an interface with the recording medium 200, and the connector 125 is physically and electrically connected to the connector 203 of the recording medium 200. The recording medium attachment / detachment detection switch 126 detects attachment / detachment of the recording medium 200 to / from the connector 125.

  The recording medium 200 is, for example, a semiconductor memory card, and includes a recording unit 201 that is a semiconductor memory, an interface 202 with the imaging apparatus 100, a connector 203 corresponding to the connector 125, and a medium recording prohibition switch 204.

  The microphone 127 converts the input sound into an audio signal. The A / D converter 128 converts the audio signal output from the microphone 127 into a digital audio signal. The audio processing unit 129 performs predetermined audio processing on the digital audio signal and outputs an audio bitstream.

  The power supply control unit 130 controls the battery detection circuit and the DC-DC converter, and supplies a necessary voltage to each unit including the recording medium 200 for a necessary period. The power connector 131 is electrically connected to a power source 132 composed of a primary battery, a secondary battery, an AC adapter, or the like. The moving image button 180 is a button for the user to instruct the imaging apparatus 100 to start and pause moving image recording.

  The reproduction circuit 150 converts the image data generated by the image processing unit 117 and stored in the memory unit 123 into a display image and transfers it to the monitor 151 which is a display device. The reproduction circuit 150 separates the YUV format image data into a luminance component signal Y and a modulated chrominance component C, and applies LPF to the analog Y signal subjected to D / A conversion. Further, the reproduction circuit 150 performs BPF on the analog C signal that has been subjected to D / A conversion, and extracts only the frequency component of the modulated color difference component. Based on the signal component thus generated and the subcarrier frequency, the reproduction circuit 150 generates a Y signal and an RGB signal and outputs them to the monitor 151. By sequentially processing moving image data continuously captured by the image sensor 115 and displaying it on the monitor 151, the monitor 151 can function as an electronic viewfinder (EVF).

  In the imaging apparatus 100 of the present embodiment, it is assumed that the imaging element 115 is a CMOS image sensor on which a color filter having a general primary color Bayer arrangement is mounted. The Bayer arrangement is a regular arrangement of R (red) 1 pixel, G (green) 2 pixels, and B (blue) 1 pixel, with 4 pixels of 2 × 2 in length and width as one unit.

  In the in-plane HDR mode, the image sensor 115 accumulates charges while alternately switching the charge accumulation time for each predetermined unit region (here, two lines (rows)) to be longer and shorter than appropriate exposure, for example. In addition, it has a configuration capable of reading and transferring.

  In this embodiment, the recording frame rate is set to 60 FPS, and two images obtained with an exposure amount that is one step over and one step under the appropriate exposure amount are combined during HDR moving image recording. In the automatic HDR moving image capturing mode, a mode capable of high-speed reading of 120 FPS or higher is used. However, the shooting frame rate is variably set to 120 FPS or 60 FPS. When inter-plane HDR is used, the exposure frame rate is set to 120 FPS, and the exposure control (first exposure control) for setting the charge accumulation time common to the image sensor 115 is used to expose one step over and one step under. Acquire two-frame images that were taken continuously under conditions. Then, these two frames of images are combined to generate one frame of the HDR video. Note that the number of frames that are continuously shot varies depending on the number of combined frames.

  On the other hand, when in-plane HDR is used, the shooting frame rate is switched to 60 FPS, and exposure control (second exposure control) for setting the charge accumulation time according to the unit area of the image sensor 115 is used to capture 1 by shooting one frame. An image with an exposure amount of step over and under one step is acquired. In this case, an image having a one-stage overexposure amount from a plurality of unit areas in which one-stage overcharge accumulation time is set, and a one-stage under-exposure from a plurality of unit areas in which one-stage undercharge accumulation time is set Get a quantity image. In this way, one frame of an HDR video is generated by combining two images obtained by shooting one frame.

  As described above, it is possible to perform image blur correction by detecting the movement of the imaging device 100 obtained from the acceleration sensor 181 and controlling the shift lens 111 so as to cancel the movement. However, in the case of a face-to-face HDR moving image, even though continuous shooting is performed, the images are captured at different times, and thus are more greatly affected by image blurring than a single image. When an image blur of an amount that cannot be absorbed by driving the shift lens 111 occurs between continuously shot images, an angle of view shift occurs between the images, and an outline blur also occurs in the HDR image obtained by combining, The image quality is degraded. Therefore, the present embodiment is characterized in that the HDR moving image frame is generated while dynamically switching between the in-plane HDR and the inter-plane HDR according to the output of the acceleration sensor 181.

  FIG. 2 is a flowchart for explaining the operation related to the HDR system switching operation at the time of HDR moving image shooting in the imaging apparatus of the present embodiment. The processing in each step of the flowchart described below including FIG. 2 is realized by the system control unit 121 developing and executing a program stored in the flash memory 122 in the memory unit 123.

Here, it is assumed that the imaging apparatus is set to the HDR moving image shooting mode and is in a shooting standby state. First, the system control unit 121 monitors the state of the moving image button 180 (S201), and if pressing is detected, the process proceeds to S202.
In step S <b> 202, the system control unit 121 sets an imaging synchronization signal in the SSG circuit of the image processing unit 117. Further, the system control unit 121 performs imaging drive settings including initialization and moving image timing settings for the imaging element 115 and the timing generation unit 116.

Next, the system control unit 121 detects the amount of motion M per unit time of the imaging apparatus 100 based on the output of the acceleration sensor 181 (S203). The system control unit 121 determines whether or not the detected motion amount M is greater than or equal to a predetermined threshold value Mt (S204). If the motion amount M is less than the threshold value Mt, the system control unit 121 performs inter-face HDR video processing (S205). A moving image frame is generated by in-plane HDR, and the process proceeds to S207. On the other hand, when the amount of motion M detected in S203 is equal to or greater than the threshold value Mt, the system control unit 121 performs in-plane HDR moving image processing (S206), generates an HDR moving image frame by in-plane HDR, and advances the processing to S207. .
Details of the inter-plane HDR moving image processing in S205 and the in-plane HDR moving image processing in S206 will be described later.

  In S207, the system control unit 121 checks whether or not the moving image button 180 has been pressed again (whether a moving image recording stop instruction has been input). If not pressed again, the system control unit 121 returns to S203 and continues the HDR moving image recording. If it is detected that the moving image button 180 is pressed again, the system control unit 121 stops moving image recording (S208) and ends the HDR moving image processing.

  In the example of FIG. 2, for the sake of simplification, the configuration in which the method of generating the HDR video frame is immediately switched based on the result of comparing the detected motion amount M with the threshold value Mt has been described. However, frequent switching occurs. It is also conceivable that is not preferable. In this case, when generating an HDR video frame using the inter-plane HDR, it should not be switched to use the in-plane HDR until the motion amount M is equal to or greater than the threshold value Mt for a certain period of time. A time constant can be set for switching. Also, the time constant may be different between switching from the in-plane HDR to the in-plane HDR and switching from the in-plane HDR to the in-plane HDR.

FIG. 3 is a flowchart showing details of the inter-plane HDR moving image processing in S205.
In step S301, the system control unit 121 sets the shooting frame rate to 120 FPS and the recording frame rate to 60 FPS. Specifically, the system control unit 121 sets the frame rate by setting the imaging synchronization signal to the SSG circuit in the image processing unit 117 and the timing setting to the imaging element 115 and the timing generation unit 116.

  In step S302, the system control unit 121 calculates a charge accumulation time Exp_long that is one step over the appropriate exposure amount and a charge accumulation time Exp_short that is one step under. Appropriate exposure conditions can be determined from the average brightness of the image taken immediately before and a program diagram prepared in advance. From this appropriate exposure condition, one of the aperture value, shutter speed, and sensitivity is changed by one step. Conditions can be used. Here, the shutter speed (charge accumulation time) is changed from the appropriate exposure condition so as not to change the depth of field.

  In step S303, the system control unit 121 sets a common charge accumulation time Exp_long for the image sensor 115, and performs an over-image shooting and reading operation (S304). Next, the system control unit 121 sets a common charge accumulation time Exp_short for the image sensor 115 (S305), and captures and reads an under image (S306). Each of the image data read in S304 and S306 is temporarily stored in the memory unit 123. As described above, in the inter-plane HDR, the charge accumulation time common to the image sensor 115 is set differently for each frame shooting, and a plurality of frames as a combination source are acquired by shooting a plurality of frames.

  In step S <b> 307, the system control unit 121 reads the over image and the under image stored in the memory unit 123, and generates an HDR video frame using the image composition circuit of the image processing unit 117. Although any known method can be used for the synthesis process for generating the HDR image frame, for example, the over image and the under image are divided into a plurality of blocks, and the blocks are divided according to the luminance level of the pixels in the block. The synthesis ratio can be determined for each. Further, the system control unit 121 inputs the generated HDR moving image frame to the image processing unit 117, processes it with an internal signal processing circuit, raster-block conversion circuit, and compression circuit, and then stores it in the memory unit 123 as a recording image. .

In step S <b> 308, the system control unit 121 reads the recording image from the memory unit 123 and writes the recording image on the recording medium 200.
Here, one HDR video frame is generated and recorded in one-time HDR video processing, but more HDR video frames may be generated and recorded.

FIG. 4 is a flowchart showing details of the in-plane HDR moving image processing in S206.
In step S401, the system control unit 121 sets the shooting frame rate and the recording frame rate to 60 FPS. Specifically, the system control unit 121 sets the frame rate by setting the imaging synchronization signal to the SSG circuit in the image processing unit 117 and the timing setting to the imaging element 115 and the timing generation unit 116.

  The system control unit 121 also performs mode setting for enabling the image sensor 115 to set the charge accumulation time for each predetermined unit region (S402). Here, it is assumed that the unit area is two lines.

In S403, the system control unit 121 calculates the charge accumulation time Exp_long that is one step over the appropriate exposure amount and the charge accumulation time Exp_short that is one step under the same as S302.
In S404, the system control unit 121 sets the charge accumulation times Exp_long and Exp_short for each unit region (here, two lines) for the image sensor 115, and performs photographing and reading operations for one frame (S405). Thereby, an over image and an under image are photographed. The image data read in step S405 is temporarily stored in the memory unit 123 for each pixel data of the line constituting the over image and each pixel of the line constituting the under image.

  In step S <b> 406, the system control unit 121 reads the over image and the under image stored in the memory unit 123, and generates an HDR video frame using the image composition circuit of the image processing unit 117. Further, the system control unit 121 inputs the generated HDR moving image frame to the image processing unit 117, processes it with an internal signal processing circuit, raster-block conversion circuit, and compression circuit, and then stores it in the memory unit 123 as a recording image. .

In step S <b> 407, the system control unit 121 reads the recording image from the memory unit 123 and writes the recording image on the recording medium 200.
Here, one HDR video frame is generated and recorded in one in-plane HDR video process, but more HDR video frames may be generated and recorded.

  FIG. 5 is a timing chart showing an example of over-image and under-image exposure and readout control read from the image sensor 115 during the in-plane HDR moving image processing. Here, control for eight lines of the image sensor 115 is shown. In the present embodiment, exposure and reading for one frame are executed while alternately repeating overexposure (charge accumulation time Exp_long) and underexposure (charge accumulation time Exp_short) every two lines. In FIG. 5, lines 1, 2, 5, 6,..., 4n + 1, 4n + 2 are overexposed lines, and lines 3, 4, 7, 8,... 4n + 3, 4 (n + 1) are underexposed lines. (N = 0, 1, 2,...).

  When the pixel information obtained from the overexposed line is collected, an over image is obtained, and when the pixel information obtained from the underexposed line is collected, an under image is obtained. However, the vertical resolution is ½ of the image obtained during normal shooting (when shooting one frame in inter-plane HDR). Thus, although the recording frame rate is the shooting frame rate / (number of combined images) in the inter-plane HDR, the vertical resolution of the image used for combining is high, and the recording frame rate is kept the same as the shooting frame rate in the in-plane HDR. However, the vertical resolution of the image used for composition is low.

  The in-plane HDR is not easily affected by the movement of the imaging device or the subject, but the vertical resolution of the image used for composition is low. On the other hand, the inter-surface HDR is easily affected by the movement of the imaging device and the subject, but generates a high-quality HDR image for the part when there is no movement (or there is a negligible movement) between the images used for composition. it can. Based on such characteristics, during the recording of the HDR moving image, when it is determined that the movement of the image capturing apparatus 100 is not suitable for performing the inter-plane HDR, switching to the in-plane HDR moving image is performed.

  In the flowchart of FIG. 2, in-plane HDR is used when the amount of motion M of the imaging apparatus is equal to or greater than the threshold value Mt. Here, the threshold value Mt may be determined as a threshold value indicating whether or not the amount of motion allows camera shake correction when the imaging apparatus 100 has a camera shake correction function. Note that the amount of motion that can be corrected by hand shake is not limited to a fixed amount, and may be a fixed value with a margin in advance, or may be determined dynamically according to the correctable amount at the time of determination. In the case where the threshold value considering the camera shake correction function is determined, the threshold value may be determined for each direction in consideration of not only the amount of motion but also the direction of motion. The threshold value can be changed according to the imaging apparatus and photographing conditions such as presence / absence of the camera shake correction function, its performance, and the angle of view of the lens, and therefore is determined in advance by an experiment or the like. Even when the threshold value is changed according to the condition at the time of determination, a threshold value corresponding to the condition is determined in advance, or a table or a calculation formula for calculating the threshold value from the condition is prepared.

  Note that the movement of the imaging device 100 may be detected by a configuration other than the acceleration sensor 181 such as a gyro sensor. In the case where the gyro sensor is used, it is possible to detect not only vertical / horizontal angle blur but also rotational blur.

  Further, the movement amount of at least one of the imaging device and the subject is detected from images taken at different times (for example, taken continuously), and switching between the in-plane HDR and the in-plane HDR is performed according to the amount of movement. You may go. The amount of motion can be detected by calculating a difference between images taken at different times using a signal processing circuit of the image processing unit 117 or a matching method used for motion compensation encoding of a moving image. . If the detected amount of motion is equal to or greater than a threshold value that is determined to affect the combined image by the inter-plane HDR, switching can be performed so that the inter-surface HDR is used, and if it is less than the threshold value, the inter-surface HDR is used. Note that this method can also detect a part of the movement (for example, a moving object) in the field of view, and may be used in combination with the detection of the amount of movement of the imaging apparatus 100. Even if the imaging apparatus 100 is stationary, for example, when a moving body that moves at high speed is detected, the in-plane HDR is used to reduce the uncomfortable feeling of the moving body region portion of the combined HDR image. Can do.

  The method for detecting the amount of movement of the imaging device and the subject from the image does not require an acceleration sensor or a gyro sensor, and therefore can be implemented even in an imaging device that does not have such a sensor, and can also reduce costs. is there.

  As described above, according to the present embodiment, at the time of HDR video recording, the amount of motion of at least one of the imaging device and the subject is detected. If the amount of motion is greater than or equal to the threshold, the in-plane HDR is used. Switch to use the inter-plane HDR. Therefore, when the movement of the imaging device or the subject is large, an HDR moving image in which the influence of the movement is suppressed can be recorded, and when the movement is small, an HDR moving image with a resolution can be recorded. As a whole, it is possible to record high-quality HDR moving images.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. The present embodiment is characterized in that in-plane HDR and inter-plane HDR are switched (selected) according to a composition ratio when generating an HDR image.

  In the in-plane HDR, in order to acquire a plurality of images with different exposure conditions by one shooting (operation in which each pixel of the image sensor is read once), the number of images to be acquired (a single HDR image is generated). The greater the number of images used to do), the lower the resolution of each image. Then, the resolution of the HDR image after synthesis changes depending on the ratio at which the synthesis source image is synthesized. The composition ratio of the composition source image used when generating each pixel of the HDR image changes according to the luminance level of the pixel and the exposure condition at the time of HDR shooting. When using an over image and an under image taken under ± 1 step exposure conditions with respect to the proper exposure conditions as the composition source image as in the present embodiment, the composition when generating a pixel with a luminance level corresponding to the proper exposure The ratio is the same for both over images and under images, which is about 50%.

  FIGS. 6A to 6C show examples in which HDR images of triangular blocks painted in white, black, and gray are generated by the in-plane HDR method, and FIGS. ) To (c) are shown schematically in a state where parts 601 to 603 are enlarged (focusing only on the shape).

Here, it is assumed that the composition ratio of the over image and the under image when each HDR image is generated is as follows.
Building block color: Composite ratio
(A) White: 90% under, 10% over
(B) Black: 10% under, 90% over
(C) Gray: 50% under, 50% over

  6 (d) to 6 (f), the white blocks (a) and black blocks (b), which are biased in the composition ratio of the under image and the over image, decrease the resolution, whereas the composition ratio is It can be seen that the equal gray blocks (c) remain well maintained. As described above, the HDR image generated using the in-plane HDR has a reduced resolution due to the bias of the composition ratio.

Therefore, in this embodiment, when the composition ratio is biased to such an extent that it is determined that the resolution of the HDR image generated by the in-plane HDR is lowered, switching is performed so that the inter-plane HDR is used.
FIG. 7 is a flowchart for explaining the operation related to the HDR system switching operation at the time of HDR moving image shooting in the imaging apparatus of the present embodiment. In FIG. 7, the same processes as those in FIG. If the moving image button has not been pressed again in S207, the system control unit 121 advances the processing to S701, and detects a predetermined region in the HDR image generated in S205 or S206. The predetermined area may be, for example, a main subject area, specifically a face area. The system control unit 121 inputs the HDR image to the image processing unit 117, and acquires information such as the presence / absence of the face area, the size, position, and reliability of the face area by executing a known face detection process. If the main subject area is not detected, the system control unit 121 returns the process to S203.

  When the main subject area can be detected, the system control unit 121 determines whether the ratio S of the over image and under image composition ratio in the main subject area is within a range of two predetermined threshold values (S1, S2) (S1 ≦ S ≦ S2) is checked (S709). Note that the composition ratio or the ratio of composition ratios can be stored in the memory unit 123 during the image composition processing in S205 and S206 (S307, S406). The ratio of the composition ratio can be obtained as (over image composition ratio) / (under image composition ratio) or (under image composition ratio) / (over image composition ratio).

  Since the determination here is a determination as to whether or not the composition ratio (use ratio) of the composition source image is biased, other methods may be used. For example, when the total composition ratio is 100%, the determination may be made based on the difference between the maximum composition ratio and the minimum composition ratio. If there are two composition source images, one composition ratio is very large. If it is very small, it may be determined that the composition ratio is biased. Here, the composition ratio for a certain area can be calculated as a statistical value such as an average value of composition ratios for individual pixels included in the area. When the main subject area is not found, the same determination may be performed as an area where deterioration in image quality is conspicuous, such as a fixed area at the center of the image. Further, the determination may be made without being limited to the composition ratio in a specific region, such as the ratio of pixels having a bias in the composition ratio in the HDR image. However, since processing load increases, it carries out considering balance with an effect.

  If the ratio of the combination ratio is within the range of the two thresholds, the system control unit 121 returns the process to S203. On the other hand, if the ratio of the composition ratio is outside the range of the two threshold values (S <S1 or S> S2), the system control unit 121 advances the process to S205 and causes the inter-face HDR video process to be executed. This is because the composition ratio of the composition source image is biased, and it is considered that the resolution is lowered when the in-plane HDR is used. If the moving image button has been pressed again in S207, the system control unit 121 stops moving image recording (S208) and ends the process.

  As described above, in this embodiment, when it is determined that the composition ratio of the over image and the under image is biased, the HDR moving image frame is generated using the inter-surface HDR, thereby reducing the resolution. It is possible to record a high-quality HDR moving image in which the deterioration is suppressed.

(Other embodiments)
In the second embodiment, the HDR method selection based on the composition ratio of the main subject region has been described only in combination with the HDR method selection based on the motion described in the first embodiment. The selection of the HDR method may be performed independently.
Further, the detection of the main subject region and the determination of the composition ratio (S701, S702) may be performed only when in-plane HDR is selected.

  In the embodiment described above, the resolution of the recorded HDR video is not mentioned for ease of explanation and understanding. However, it goes without saying that the resolution (number of pixels) of the recorded HDR moving image frame is equal regardless of whether the HDR moving image frame is generated by the in-plane HDR or the inter-plane HDR. Therefore, as necessary, the number of pixels of the moving image frame generated by the in-plane HDR is decreased, or the number of pixels of the moving image frame generated by the in-plane HDR is increased.

  In the above-described embodiment, the selection of the HDR method based on the motion and the selection of the HDR method based on the composition ratio of the main subject region are mentioned. The above-described selection of the HDR method is necessary because each HDR method has merits and demerits. However, a reduction in in-plane HDR resolution may not be a problem with an image sensor having a large number of pixels. It is done. For example, if the image sensor 115 has an effective number of pixels exceeding 9.8 million pixels (3840 × 2560), it is possible to obtain data for two frames of HD video resolution (1920 × 1080) by shooting one frame. It is. Therefore, when the HDR method is selected, the setting information of the number of recorded or displayed pixels (resolution) is referred to. If the setting does not cause a decrease in resolution even in the in-plane HDR, the in-plane HDR method is used. You may control to select. The step of confirming the setting information of the recording or the number of display pixels (resolution) and the determination of the HDR method thereby is performed based on the acquisition of the setting information and the determination based on the setting information in accordance with the motion amount determination in S204 of FIG. May be. Further, the determination may be made based on this setting information in the imaging drive setting of S202.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (5)

An imaging device,
An image sensor capable of setting a charge accumulation time for each predetermined unit area;
A synthesizing unit for synthesizing a plurality of images obtained by photographing under different exposure conditions to generate one synthesized image;
Obtaining means for obtaining a composition ratio of the plurality of images relating to a predetermined region of the composite image;
Control means for controlling the operation of the image sensor according to the composition ratio acquired by the acquisition means,
The control means includes
(A) When it is determined that the composition ratio is biased,
By controlling the operation of the image sensor so as to obtain the plurality of images by photographing a plurality of frames using the first exposure control that sets a charge accumulation time common to the image sensor,
(B) When it is not determined that the composition ratio is biased,
Controlling the operation of the image sensor so that the plurality of images can be obtained by photographing one frame using second exposure control that sets a plurality of different charge accumulation times according to the unit region;
An imaging apparatus characterized by that. Wherein the predetermined region, the imaging apparatus according to claim 1, characterized in that the area of the main subject determined in advance. The imaging apparatus according to claim 2, wherein the predetermined area is a face area. Wherein, when performing the first exposure control, the imaging frame rate of the imaging device, wherein to increase according to the number of said plurality of images, one of claims 1 to 3 The imaging apparatus of Claim 1. An image sensor capable of setting a charge accumulation time for each predetermined unit area;
A method for controlling an imaging apparatus, comprising: a combining unit that combines a plurality of images obtained by shooting under different exposure conditions to generate a single combined image;
An obtaining step for obtaining a composition ratio of the plurality of images with respect to a predetermined region of the composite image;
The control means
(A) When it is determined that the composition ratio is biased,
By controlling the operation of the image sensor so as to obtain a plurality of images with different exposure amounts by photographing a plurality of frames using the first exposure control that sets a charge accumulation time common to the image sensor,
(B) When it is not determined that the composition ratio is biased,
Control for controlling the operation of the image sensor so that a plurality of images having different exposure amounts can be obtained by photographing one frame using second exposure control that sets a plurality of different charge accumulation times according to the unit area. Steps,
A method for controlling an imaging apparatus, comprising: