JP6278729B2 - Imaging apparatus, control method thereof, and program - Google Patents

Description

  The present invention relates to an imaging apparatus such as a digital camera or a digital video camera, and more particularly to an imaging apparatus having a function of synthesizing a plurality of continuously captured images.

  In an imaging device such as a digital camera, the frame rate of the image sensor is recorded as a recording frame rate during moving image recording or recording, or an image captured at a high frame rate is recorded at a low frame rate and a decisive moment is slowly reproduced. There is something. Also disclosed is a technique for synthesizing and recording outputs of a plurality of images that are continuously captured.

  For example, an image processing method has been proposed in which, when a plurality of images that are continuously captured in time series are combined, the presence or absence of the combination is determined in accordance with the change in the subject or the change in the imaging condition for the subject (Patent Document 1). .

  In this proposal, a moving image with a wide dynamic range (hereinafter referred to as an HDR moving image) and a display image are generated by continuously capturing and synthesizing each image while alternately changing the exposure time between long seconds and short seconds. Is possible. Further, if the composition / non-composition is switched according to the state of the subject, it is also possible to dynamically switch between the HDR moving image and the normal (non-combined) moving image during recording.

  Further, a technique has been proposed in which background extraction is performed based on the distance to the subject and the luminance, and when there is a luminance difference greater than or equal to a predetermined value compared to the subject, HDR video shooting is performed (Patent Document 2).

JP 2007-36742 A JP 2005-142953 A

  In Patent Documents 1 and 2, contrast measurement is performed when focus detection (ranging) is performed. However, there is a problem when contrast measurement is performed by acquiring a contrast measurement value from an output signal of an image sensor.

  That is, in HDR moving image shooting, since all frames are not shot with an appropriate exposure time, the contrast measurement value may not be reliable, and the reliability of the focus detection result decreases. Further, if the contrast measurement is performed only with the frame of proper exposure, the time required for focus detection becomes longer.

  Therefore, an object of the present invention is to provide a mechanism that can acquire a highly reliable focus detection result in a short time during display of an HDR image or shooting of an HDR image.

  In order to achieve the above object, an imaging apparatus according to the present invention combines an imaging unit that continuously images a subject in time series and a plurality of image data output from the imaging unit with different exposures and outputs the synthesized image data. Means, exposure control means for controlling the exposure of the imaging means, focus detection means for performing focus detection from the image data, and a first gain for applying a first gain to the image output to the image composition means And a second gain means for applying a second gain to the image output to the focus detection means, and a plurality of the first gain and the second gain that are different from each other and output to the focus detection means. Control means for controlling the second gain so that the brightness of images having different exposures is the brightness corresponding to the appropriate exposure obtained from the images.

  According to the present invention, a highly reliable focus detection result can be acquired in a short time during display of an HDR image or shooting of an HDR image.

It is a block diagram which shows the structural example of the digital camera which is an example of embodiment of the imaging device of this invention. It is a flowchart figure which shows the image composition process at the time of HDR moving image photography. It is a graph which shows the table of the synthetic | combination ratio of three types of images from which exposure differs in step S205 of FIG. It is a figure for demonstrating the relationship between the synthesis process of an appropriate image, an under image, and an over image, and AF evaluation value. It is a flowchart figure which shows an example of the exposure control at the time of HDR moving image photography. FIG. 6 is a flowchart illustrating an example of exposure control for focus detection in step S504 in FIG. 5. It is a flowchart figure which shows an example of normal exposure control in step S503 of FIG. (A) is a flowchart showing processing when an under image is output from an image sensor in exposure control for focus detection, and (b) is a case where an appropriate image is output from the image sensor in exposure control for focus detection. It is a flowchart figure which shows a process. FIG. 6C is a flowchart illustrating processing when an over image is output from the image sensor in exposure control for focus detection.

  Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram illustrating a configuration example of a digital camera which is an example of an embodiment of an imaging apparatus of the present invention.

  As shown in FIG. 1, the digital camera 100 of this embodiment includes a zoom lens 110, an image blur correction lens 111, a focus lens 112, a mechanical shutter 113, and a diaphragm 114. The image sensor 115 continuously captures a subject in time series, photoelectrically converts subject light that has passed through the lenses 110 to 112, the mechanical shutter 113, and the aperture 114 into an electrical signal and outputs the electrical signal to the image processing unit 117. The timing generator 116 generates timing pulses necessary for driving and sampling the image sensor 115.

  The image processing unit 117 includes a preprocessing circuit, an AF evaluation value calculation circuit, a luminance integration circuit, a signal processing circuit, an SSG circuit, an image synthesis circuit, a reduction circuit, a raster block conversion circuit, and a compression circuit, and serves as an imaging output. Receive digital output.

  The SSG circuit generates an imaging drive synchronization signal. Further, the SSG circuit receives an imaging drive clock from the timing generator 116, generates horizontal and vertical synchronization signals, and outputs them to the timing generator 116 and the imaging device 115.

  The preprocessing circuit distributes the input image to the luminance integration circuit and the signal processing circuit in units of rows. Further, the preprocessing circuit performs processing such as data correction between channels necessary for imaging data.

  The AF evaluation value calculation circuit performs horizontal filtering on the luminance component of the image signal input in the set plurality of evaluation areas, selects the maximum value while extracting a predetermined frequency representing the contrast, and selects the vertical direction Perform the integral operation.

  The luminance integration circuit mixes and generates luminance components from the RGB signals, divides the input image into a plurality of regions, and generates a luminance component for each region.

  The signal processing circuit performs color carrier removal, aperture correction, gamma correction processing, and the like on the output data of the image sensor 115 to generate a luminance signal. At the same time, the signal processing circuit performs color interpolation, matrix conversion, gamma processing, gain adjustment, and the like to generate a color difference signal, and forms YUV format image data in the memory unit 123. In the signal processing circuit, the luminance signals (Y) of the YUV format image data to be generated are tabulated for each level, and luminance distribution data for each image data is generated.

  The image synthesizing circuit inputs a plurality of RGB-format captured images formed in the memory unit 123 or a plurality of YUV-format image data obtained by performing signal processing on the captured images. A composite image is generated by multiplying and adding the set coefficient.

  Based on the output of the signal processing circuit, the reduction circuit performs extraction, thinning, linear interpolation processing, and the like of input pixel data, and performs pixel data reduction processing in the horizontal and vertical directions.

  In response to the reduction processing by the reduction circuit, the raster block conversion circuit converts the raster scan image data scaled by the reduction circuit into block scan image data. Such a series of image processing is realized by using the memory unit 123 as a buffer memory.

  The compression circuit compresses the YUV image data converted into the block scan image data by the buffer memory according to the moving image compression method, and outputs a moving image bit stream.

  The exposure control unit 118 controls driving of the mechanical shutter 113 and the diaphragm 114. The lens driving unit 119 moves the zoom lens 110 and the focus lens 112 along the optical axis to form a subject image on the image sensor 115. The lens driving unit 119 drives the image blur correction lens 111 corresponding to the outputs of the angular velocity sensor and the acceleration sensor, and optically corrects camera shake. The release switch 120 issues a shooting instruction corresponding to the operation of the release button.

  The system control unit 121 includes a CPU and its interface circuit, a DMAC (Direct Memory Access Controller), a bus arbiter, and the like, and controls the entire camera 100. A program executed by the CPU is stored in the flash memory 122. The memory unit 123 is composed of a DRAM or the like, and temporarily stores data in the middle of each process. In addition, the program in the flash memory 122 is expanded and used.

  The I / F unit 124 is an interface with the recording medium 200. A connector 203 of the recording medium 200 is connected to the connector 125. The recording medium attachment / detachment detection switch 126 detects attachment / detachment of the recording medium 200. The microphone 127 converts the input sound into an audio signal. The A / D converter 128 converts the sound output of the microphone 127 into a digital sound signal. The audio processing unit 129 performs predetermined audio processing on the digital audio data and outputs an audio bitstream.

  The power supply control unit 130 controls the battery detection circuit and the DC-DC converter, and supplies a necessary voltage to each unit including the recording medium 200 for a necessary period. The power supply unit 132 includes a primary battery, a secondary battery, an AC adapter, or the like, and is connected to the camera 100 via a connector 131. The moving image recording switch 180 instructs to record a moving image in accordance with the operation of the moving image recording start button. The display device 151 is configured by an LCD or the like.

  The reproduction circuit 150 converts the image data generated by the image processing unit 117 and stored in the memory unit 123 into a display image and transfers it to the display device 151. Further, the reproduction circuit 150 separates the YUV format image data into a luminance component signal Y and a modulated chrominance component C, and applies LPF to the Y signal analogized by D / A conversion. Further, the reproduction circuit 150 performs BPF on the analog C signal that has been subjected to D / A conversion, and extracts only the frequency component of the modulated color difference component.

  Based on the signal component thus generated and the subcarrier frequency, the signal is converted into a Y signal and an RGB signal and output to the display device 151. Thus, EVF (electronic viewfinder) is realized by sequentially processing the image data output from the image sensor 115 and displaying it on the display device 151.

  The subject detection unit 133 detects position information of a specific region of the subject image, for example, a human face, from the output signal of the image sensor 115. The recording medium 200 includes a recording unit 201 composed of a semiconductor memory, an interface 202 with the camera 100, a connector 203 for connecting to the camera 100, and a medium recording prohibition switch 204.

  FIG. 2 is a flowchart showing an image composition process at the time of HDR video shooting in the camera 100 of the present embodiment. Each process in FIG. 2 is executed by the CPU of the system control unit 121 after the program stored in the flash memory 122 is expanded in the memory unit 123. Note that HDR movie shooting is performed by shooting three types of exposure images of appropriate / under / over and expanding the dynamic range by combining processing.

  In FIG. 2, in step S201, the system control unit 121 proceeds to step S202 when recording of the HDR moving image is started.

  In step S202, the system control unit 121 captures and acquires an image of an exposure amount with appropriate exposure (hereinafter referred to as an appropriate image), and the process proceeds to step S203.

  In step S203, the system control unit 121 captures and acquires an image (hereinafter referred to as an under image) with an underexposure amount that is less than the appropriate exposure amount, and proceeds to step S204.

  In step S204, the system control unit 121 captures and acquires an image (hereinafter referred to as an over image) with an exposure amount that is over the appropriate exposure amount, and proceeds to step S205.

  In step S205, the system control unit 121 performs composition processing on the basis of the under image for the three types of images with different exposures acquired in step S202 to step S204, and the process proceeds to step S206. Details of the synthesis process here will be described later with reference to FIG.

  In step S206, since the image synthesized in step S205 is in the state of the raw data format output from the image sensor 115, the system control unit 121 performs the development process in the YUV format and then performs the encoding process. And proceed to step S207.

  In step S207, the system control unit 121 records the moving image on the recording medium 200. When it is determined in step S208 that recording of the moving image on the recording medium 200 is finished, the HDR moving image shooting is ended.

  FIG. 3 is a graph showing a synthesis ratio table of three types of images with different exposures in step S205 of FIG. In the present embodiment, the composition ratio is determined based on an under image in which whiteout is difficult to occur among appropriate images, under images, and over images.

Specifically, the luminance level of the under image is obtained in units of pixels or regions, and the over image composition ratio α, the appropriate image composition ratio β, and the under image composition ratio γ are obtained from the composition ratio table shown in FIG. decide. Then, based on the determined composition ratio, a composite image is generated using the following equation (1).
HDR_Image [x, y] = OverImage [x, y] × α + NormalImage [x, y] × β + UnderImage [x, y] × γ (1)

  FIG. 4 is a diagram for explaining the relationship between the synthesis process of the appropriate image, the under image, and the over image and the AF evaluation value.

  As shown in FIG. 4, in HDR video shooting, the appropriate image N1, the under image U1, and the over image O1 are combined, and the appropriate image N2, the under image U2, and the over image O2 are combined. At that time, for each image, the exposure amount is changed by changing the shutter speed (exposure time).

  In the present embodiment, the shutter speed of the appropriate image is 1/250, the shutter speed of the under image is 1/500, and the shutter speed of the over image is 1/125. The frame rate of the image sensor 115 is 90 fps, and an HDR moving image is generated by synthesizing three images. Therefore, the frame rate of the moving image file (composite image) is 30 fps.

  By the way, when the focus detection data for determining the position of the focus lens 112 is acquired, if the focus detection data after the synthesis process is used, not only the contrast is lost by the synthesis process but also the frame rate is 30 fps. Acquisition of detection data is slow.

  On the other hand, if focus detection data is to be acquired from all images, the frame rate increases to 90 fps, but the reliability of the focus detection data acquired from the under image and the over image decreases.

  Therefore, in the present embodiment, when the focus detection data (AF evaluation value) is acquired by the system control unit 121, the exposure amount is changed from that in the normal state and compensated with a digital gain, thereby acquiring highly reliable focus detection data. Enable.

  FIG. 5 is a flowchart showing an example of exposure control at the time of HDR video shooting in the camera 100. Each process in FIG. 5 is executed by the CPU of the system control unit 121 after the program stored in the flash memory 122 is expanded in the memory unit 123.

  In FIG. 5, in step S501, the system control unit 121 determines whether or not the focus is being detected when the HDR moving image is captured. If the focus is being detected, the process proceeds to step S504. The process proceeds to S502.

  In step S502, the system control unit 121 determines whether or not the subject being imaged has a certain amount of movement. If there is no certain amount of movement, the system control unit 121 proceeds to step S503, and if there is a certain amount of movement. Advances to step S504.

  In step S503, the system control unit 121 performs normal exposure control, and proceeds to step S505. Normal exposure control will be described later with reference to FIG.

  In step S504, the system control unit 121 executes exposure control for focus detection, and the process proceeds to step S505. The exposure control for focus detection will be described later with reference to FIG.

  In step S505, the system control unit 121 ends the process when shooting of the HDR moving image ends.

  FIG. 6 is a flowchart showing an example of exposure control for focus detection in step S504 of FIG.

  In FIG. 6, in step S601, the system control unit 121 determines whether the output of the image sensor 115 is an under image frame. If the output is an under image, the system control unit 121 proceeds to step S604. Proceed to

  In step S602, the system control unit 121 determines whether the output of the image sensor 115 is a proper image frame. If the output is a proper image, the system control unit 121 proceeds to step S605. If the output is not a proper image, the system control unit 121 proceeds to step S603.

  In step S603, the system control unit 121 determines that the output of the image sensor 115 is an over-image frame, and proceeds to step S606.

  In step S604, the system control unit 121 adjusts the exposure amount of the under image to one step under (−1) from the exposure amount of the appropriate image, and proceeds to step S607. Here, if the aperture value of the aperture 14 is changed, the depth of field of the image changes, so the shutter speed is changed by changing the pulse of the electronic shutter input to the image sensor 115.

  In step S607, the system control unit 121 sets the digital gain for the moving image output (for example, display image) to the same magnification, and doubles the digital gain for the focus detection image output. As a result, in step S610, the exposure setting for the image for the moving image is one step lower.

  Here, the process of setting the digital gain for the image output for moving images corresponds to an example of the first gain means of the present invention, and the process of setting the digital gain for the image output for focus detection includes This corresponds to an example of the second gain means of the present invention.

  In step S605, the system control unit 121 sets the exposure amount (± 0) of the appropriate image and proceeds to step S608.

  In step S608, the system control unit 121 sets the digital gain for the moving image output to the same magnification and the digital gain for the focus detection image output to the same magnification. As a result, in step S611, the exposure setting of the moving image is set to appropriate exposure.

  In step S606, the system control unit 121 sets the over image exposure setting to an appropriate image exposure amount (± 0), and proceeds to step S609.

  In step S609, the system control unit 121 doubles the digital gain with respect to the moving image output and the digital gain with respect to the focus detection image output. As a result, in step S612, the moving image is one step over. Finally, in step S613, the exposure setting for focus detection is appropriate exposure in all frames of the under image, the appropriate image, and the over image, and the exposure control for focus detection ends.

  FIG. 7 is a flowchart showing an example of normal exposure control in step S503 of FIG.

  In FIG. 7, in step S701, the system control unit 121 determines whether the output of the image sensor 115 is a frame of an under image. If the image is an under image, the system control unit 121 proceeds to step S704. move on.

  In step S702, the system control unit 121 determines whether the output of the image sensor 115 is a proper image frame. If the output is a proper image, the system control unit 121 proceeds to step S705. If the output is not a proper image, the system control unit 121 proceeds to step S703.

  In step S703, the system control unit 121 determines that the output of the image sensor 115 is an over-image frame, and proceeds to step S706.

  In step S <b> 704, the system control unit 121 adjusts the under image exposure setting to one level under from the exposure amount of the appropriate image. Here, since changing the aperture 14 changes the depth of field of the image, the electronic shutter pulse input to the image sensor 115 is changed to change the shutter speed. As a result, in step S707, the exposure setting of the image for moving images is one step lower.

  In step S705, the system control unit 121 sets the appropriate image exposure setting to an appropriate exposure amount (± 0). As a result, in step S708, the exposure setting of the appropriate image for the moving image becomes the appropriate exposure.

  In step S706, the system control unit 121 increases the over image exposure setting by one step. As a result, in step S709, the exposure setting of the moving image is over one step.

  When focus detection is performed by normal exposure control, the exposure settings of the focus detection images in steps S707, S708, and S709 are one step under and proper one step over, respectively. The digital gain is x1 for both the moving image and the focus detection image.

  Further, when the subject to be imaged in step S502 is not moving, it is not necessary to quickly perform focus detection. Therefore, normal exposure control is performed even at the time of focus detection, and focus detection is performed only on a frame with proper exposure, that is, three frames. It may be carried out at a rate of once.

  FIG. 8A is a flowchart illustrating processing when an under image (moving image) is output from the image sensor 115 in the focus detection exposure control.

  In FIG. 8A, the output image signal of the image sensor 115 is adjusted by an electronic shutter after A / D conversion, and the exposure amount is reduced by one step. After that, the digital gain for the image data output to the circuit for generating the moving image is set to x1, and the exposure setting for the moving image is set one step under. Also, the digital gain for the image data output to the AF evaluation value acquisition (focus detection) circuit is set to x2, and the exposure setting of the focus detection image is set to appropriate exposure.

  FIG. 8B is a flowchart showing processing when a proper image (moving image) is output from the image sensor 115 in the focus detection exposure control.

  In FIG. 8B, the output image signal of the image sensor 115 is adjusted with an electronic shutter after A / D conversion to obtain an appropriate exposure amount. Thereafter, the digital gain for the image data output to the moving image generating circuit and the image data output to the AF evaluation value acquisition (focus detection) circuit is set to x1, and the exposure setting is set to appropriate exposure.

  FIG. 8C is a flowchart showing processing when an over image (moving image) is output from the image sensor 115 in the exposure control for focus detection.

  In the exposure control for focus detection in FIG. 8C, the signal of the image sensor 115 is adjusted by an electronic shutter after A / D conversion to obtain an appropriate exposure amount. Thereafter, the digital gain for the image data to be output to the circuit for generating the moving image is set to x2, and the exposure setting of the moving image is set one step over. Also, the digital gain for the image data output to the AF evaluation value acquisition (focus detection) circuit is set to x1, and the exposure setting of the focus detection image is set to appropriate exposure.

  Note that the numerical values in parentheses in FIG. 8C are values in normal exposure control. When the autofocus operation is not performed, the output image signal (over image) of the image sensor 115 is adjusted by the electronic shutter after A / D conversion, and the exposure amount is increased by one step.

  Thereafter, the digital gain for the image data to be output to the circuit for generating the moving image is set to x1, and the exposure setting of the moving image is set one step over. In other words, when adjusting the exposure amount with the electronic shutter after A / D conversion, different exposure control is performed for normal and focus detection only for frames in which an over image is generated.

  As described above, in the present embodiment, a highly reliable focus detection result can be acquired in a short time during display of an HDR moving image or shooting of an HDR moving image.

  In addition, this invention is not limited to what was illustrated to the said embodiment, In the range which does not deviate from the summary of this invention, it can change suitably.

  For example, in the above embodiment, the case where data of an arbitrary pixel of the image sensor 115 is used as focus detection data is illustrated, but the present invention is not limited to this.

  That is, when a specific pixel for focus detection is arranged on the image sensor 115, normal exposure control and digital gain are applied to the output of the normal pixel, and the output of the focus detection pixel is applied. May perform exposure control and digital gain for focus detection.

  In the above-described embodiment, moving image shooting is exemplified. However, the present invention can also be applied to focus detection in a shooting standby state in which a release button is pressed before still image shooting.

  Moreover, although the digital camera was illustrated as an imaging device in the said embodiment, it is not limited to this.

  For example, the present invention can be applied to a device capable of imaging such as a mobile phone terminal having an image sensor, a portable image viewer, a television with a camera, a digital photo frame, a music player, a game machine, and an electronic book reader.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed. It can also be realized by executing software (program) acquired via a network or various storage media on a personal computer (CPU, processor).

DESCRIPTION OF SYMBOLS 100 Digital camera 115 Image pick-up element 117 Image processing part 118 Exposure control part 120 Release switch 121 System control part 122 Flash memory 123 Memory part 133 Subject detection part 150 Playback circuit 151 Display apparatus 180 Movie recording switch

Claims (7)

Imaging means for continuously imaging a subject in time series;
Image combining means for combining and outputting a plurality of image data with different exposures output from the imaging means;
Exposure control means for controlling exposure of the imaging means;
Focus detection means for performing focus detection from the image data;
First gain means for applying a first gain to an image output to the image composition means;
Second gain means for applying a second gain to the image output to the focus detection means;
The first gain and the second gain are different from each other, and the brightness of an image with different exposures output to the focus detection unit is the brightness corresponding to the appropriate exposure obtained from the image. An image pickup apparatus comprising: control means for controlling a gain.   The control means determines whether or not focus detection is performed by the focus detection means, and when it is determined that focus detection is performed by the focus detection means, the control is performed by the exposure control means. The imaging apparatus according to claim 1, wherein an exposure setting to be changed is changed.   3. The control unit according to claim 1, wherein the control unit changes an exposure setting controlled by the exposure control unit only when an output image of the imaging unit is overexposed than the appropriate exposure. 4. Imaging device. Subject detection means,
When the subject detected by the subject detection unit does not move beyond a certain level, the control unit does not change the exposure setting controlled by the exposure control unit and the second gain without changing the second gain. 4. The imaging apparatus according to claim 1, wherein the focus detection unit is operated only when an output image is the appropriate exposure. 5. The imaging apparatus according to claim 1, wherein the focus detection unit performs focus detection by extracting contrast from the image data. An imaging step of continuously imaging a subject in time series;
An image synthesis step of synthesizing and outputting a plurality of image data with different exposures output in the imaging step;
An exposure control step for controlling exposure in the imaging step;
A focus detection step of performing focus detection from the image data;
A first gain setting step of applying a first gain to an image used in the image synthesis step;
A second gain setting step of applying a second gain to the image used in the focus detection step;
The first gain and the second gain are different from each other, and the second gain is set so that the brightness of an image having different exposures used in the focus detection step corresponds to an appropriate exposure obtained from the image. And a control step for controlling the imaging device. An imaging step of continuously imaging a subject in time series;
An image synthesis step of synthesizing and outputting a plurality of image data with different exposures output in the imaging step;
An exposure control step for controlling exposure in the imaging step;
A focus detection step of performing focus detection from the image data;
A first gain setting step of applying a first gain to an image used in the image synthesis step;
A second gain setting step of applying a second gain to the image used in the focus detection step;
The first gain and the second gain are different from each other, and the second gain is set so that the brightness of an image having different exposures used in the focus detection step corresponds to an appropriate exposure obtained from the image. And a control step for controlling the program.

Images