JP2010219940A - Imaging device and imaging method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a wide dynamic range image without object fluctuation caused by image synthesis when the object is in active state, and also a wide dynamic range image with high resolution when the object is in inactive state. <P>SOLUTION: An imaging device is equipped with: an imaging portion 23 having a plurality of pixel groups; an imaging control portion 108 which performs first imaging of image with an exposure time different for every pixel group, and a second imaging of image a plurality of times continuously in a different imaging condition for every time with a plurality of pixel ; a motion vector detection portion 110 which detects the motion vector of the object; an object determination portion 112 which determines whether the object is in inactive state or in active state, based on the motion vector of the object; and an image synthesis portion 114 which synthesizes a plurality of images obtained by every pixel group by the first imaging when determined to be in active state, and synthesizes a plurality of high resolution images obtained by the second imaging when determined to be in inactive state. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an imaging apparatus and an imaging method having a function of expanding a dynamic range of an image.

  An imaging apparatus having a function of expanding the dynamic range of an image is known.

  Patent Document 1 discloses a configuration that expands the dynamic range by continuously shooting and combining two images having different exposure times.

  In Patent Document 2, an image sensor having a first pixel group and a second pixel group is used, and the exposure time of the first pixel group is different from the exposure time of the second pixel group. A configuration is disclosed in which the dynamic range is expanded by combining the first captured image obtained by the pixel group and the second captured image obtained by the second pixel group.

JP 2000-138868 A JP 2007-235656 A

  However, there is a problem that subject blurring increases in addition to the expansion of the dynamic range and a problem that the number of pixels of the captured image decreases with respect to the number of pixels arranged in the image sensor.

  In the configuration of Patent Document 1, when the subject is in a moving state, the subject blur on the image is additionally increased by combining two images with different imaging timings.

  In the configuration of Patent Document 2, the number of pixels in the captured image is halved with respect to the number of pixels arranged in the image sensor.

  The present invention has been made in view of such circumstances. When the subject is in a moving state, a wide dynamic range image free from subject blur caused by image composition can be obtained, and the subject is in a stationary state. An object of the present invention is to provide an imaging apparatus and an imaging method capable of obtaining a high-resolution wide dynamic range image.

  To achieve the above object, the present invention provides an imaging device having a plurality of pixel groups, a first imaging for imaging with different exposure times for each of the pixel groups of the imaging device, and the plurality of pixel groups. Imaging control means for performing second imaging for imaging a plurality of times continuously under different imaging conditions as a single high-resolution pixel group, and motion for detecting a motion vector of a subject from an image obtained by the imaging element When it is determined that the subject is in a moving state, a subject detection means for determining whether the subject is in a stationary state or a moving state based on at least a motion vector of the subject, and a vector detecting means When a first wide dynamic range image is generated by synthesizing a plurality of pixel group images obtained by the first imaging, and the subject is determined to be stationary Image synthesizing means for generating a second wide dynamic range image having a resolution higher than that of the first wide dynamic range image by synthesizing a plurality of high resolution images obtained by the second imaging. And an imaging apparatus characterized by comprising:

  According to this configuration, when it is determined that the subject is in a moving state, a plurality of images for each pixel group obtained by one imaging with different exposure times for each pixel group are combined. When the first wide dynamic range image is generated and it is determined that the subject is in a stationary state, a plurality of high-resolution images obtained by continuously imaging a plurality of times under different imaging conditions each time. Is combined to generate a second wide dynamic range image having a higher resolution than the first wide dynamic range image. Therefore, when the subject is in a moving state, subject blurring due to image synthesis is generated. When there is no wide dynamic range image and the subject is stationary, it is possible to obtain a high resolution wide dynamic range image.

  In addition, the present invention captures an image sensor having a plurality of pixel groups and an exposure time that is different for each pixel group of the image sensor, and the exposure times are interchanged between the pixel groups, so that a plurality of times An imaging control means for imaging, a motion vector detection means for detecting a motion vector of a subject from an image obtained by the imaging device, and whether the subject is stationary or moving based at least on the motion vector of the subject A subject determination means for determining whether or not the subject is in a moving state; and when the subject is determined to be in a moving state, a first wide dynamic image is obtained by synthesizing a plurality of images by pixel group obtained by one imaging. When a range image is generated and it is determined that the subject is in a stationary state, a plurality of pixels group-by-pixel images obtained by a plurality of times of imaging are synthesized for the same pixel group. To provide an imaging apparatus characterized by comprising an image synthesizing means for generating a second wide dynamic range image is a high resolution than the first wide dynamic range image by combining among the pixel groups.

  According to this configuration, when it is determined that the subject is in a moving state, a plurality of images for each pixel group obtained by one imaging with different exposure times for each pixel group are combined. When the first wide dynamic range image is generated and it is determined that the subject is in a stationary state, the second wide dynamic range image having a higher resolution than the first wide dynamic range image is generated. When the subject is in a moving state, it is possible to obtain a wide dynamic range image without subject blurring due to image composition, and when the subject is in a stationary state, a high resolution wide dynamic range image can be obtained. It becomes.

  In one embodiment of the present invention, a sensor for detecting the movement of the imaging apparatus is provided, and the image synthesizing unit performs image synthesis using a detection result of the sensor.

  According to this configuration, it is possible to record an accurate wide dynamic range image without adding camera shake.

  In the aspect of the invention, the motion vector detection unit detects a motion vector of the subject based on an image obtained by the imaging element and a detection result of the sensor.

  According to this configuration, it is possible to reliably determine whether or not the subject is stationary even when there is a camera shake.

  In one embodiment of the present invention, the motion vector detection unit detects a motion vector of the imaging device from an image obtained by the imaging device, and the image synthesis unit calculates a motion vector of the imaging device. Image synthesis.

  According to this configuration, it is possible to record an accurate wide dynamic range image without adding camera shake even without a sensor for detecting the movement of the imaging apparatus.

  In one aspect of the present invention, the motion vector detection means detects the motion vector using a plurality of images captured under the same imaging conditions.

  According to this configuration, since a motion vector is detected using a plurality of images captured under the same imaging conditions, the motion vector detection accuracy is improved.

  In one embodiment of the present invention, the imaging element has a honeycomb arrangement in which the second pixel group is arranged between lattices of the first pixel group.

  In one embodiment of the present invention, the imaging element is a square lattice array in which a large number of pixels are arrayed in a square lattice at regular intervals in two directions.

  In addition, the present invention uses a first imaging that uses an imaging device having a plurality of pixel groups and takes an exposure time different for each of the pixel groups of the imaging device, and a single high resolution pixel for the plurality of pixel groups. An imaging step for performing a second imaging for imaging a plurality of times continuously under different imaging conditions as a group, and a motion vector detection step for detecting a motion vector of a subject from the image obtained in the imaging step; A subject determination step for determining whether the subject is stationary or moving based on at least a motion vector of the subject; and when the subject is determined to be moving, the first When a first wide dynamic range image is generated by combining a plurality of pixel group images obtained by imaging, and the subject is determined to be stationary, An image combining step of generating a second wide dynamic range image having a resolution higher than that of the first wide dynamic range image by combining a plurality of high resolution images obtained by the second imaging; An imaging method is provided.

  Further, the present invention uses an imaging device having a plurality of pixel groups, images with different exposure times for each of the pixel groups of the imaging device, and continuously switches the exposure times between the pixel groups a plurality of times. An imaging step for imaging, a motion vector detection step for detecting a motion vector of the subject from the image obtained in the imaging step, and whether the subject is in a stationary state or a moving state based on at least the motion vector of the subject A subject determination step for determining whether or not the subject is in a moving state, and by combining a plurality of images for each pixel group obtained by one imaging, the first wide range is obtained. When a dynamic range image is generated and it is determined that the subject is in a stationary state, a plurality of pixels group-by-pixel images obtained by a plurality of times of imaging are obtained for the same pixel group. And an image synthesis step of generating a second wide dynamic range image having a higher resolution than the first wide dynamic range image by combining and combining between pixel groups. To do.

  According to the present invention, when the subject is in a moving state, it is possible to obtain a wide dynamic range image free from subject blur caused by image synthesis, and when the subject is in a stationary state, a high resolution wide dynamic range image is obtained. Can be obtained.

Front view of a digital camera to which the present invention is applied Rear view of a digital camera to which the present invention is applied 1 is an overall configuration diagram of an example of a digital camera according to a first embodiment. Plan view schematically showing a configuration example of an image sensor The principal part functional block diagram of CPU in 1st Embodiment (A) is an input / output characteristic chart of overexposure, (B) is an input / output characteristic chart of underexposure, and (C) is an input / output characteristic chart used for explaining a wide dynamic range image. The flowchart which shows the flow of the example of an imaging process in 1st Embodiment. (A) is a first example of imaging control, and (B) is a second example of imaging control. Overall configuration diagram of an example of a digital camera according to the second embodiment Main part functional block diagram of CPU in 2nd Embodiment The flowchart which shows the flow of the example of an imaging process in 2nd Embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a front view of a digital camera 100 to which the present invention is applied.

  A lens barrel 81 is disposed in front of the camera body 80 (the main body of the digital camera 100). The lens barrel 81 incorporates a photographic lens 24. The lens barrel 81 is configured by a retracting mechanism, and can be extended from a state where it is retracted to the camera body 80. The camera body 80 is provided with a lens cover 82 that protects the photographing lens 24 by covering the front surface of the photographing lens 24 and blocking the photographing lens 24 from the outside when not photographing. The lens cover 82 is configured by a mechanism that can be freely opened and closed. The lens cover 82 covers the front surface of the photographing lens 24 in the open state, and exposes the front surface of the photographing lens 24 to the outside in the closed state. The retracting mechanism of the lens barrel 81 and the opening / closing mechanism of the lens cover are well-known mechanisms and will not be described.

  Further, a flash 60, an AF auxiliary light lamp 66, a self-timer lamp 68, and the like are disposed on the front surface of the camera body 80. The flash 60 irradiates the object scene with light for photographing. For example, it is composed of a xenon tube. You may comprise with LED. The AF auxiliary light lamp 66 emits light for AF described later, and the self-timer lamp 68 is a lamp for informing the self-timer photographing.

  On the upper surface of the camera body 80, a mode dial 83 (mode changeover switch) having a release switch 84 disposed at the center and a power switch 85 are disposed. The release switch 84 is configured by a two-stage stroke type hardware switch, and can be so-called “half-pressed” and “full-pressed”. A shooting preparation instruction is input by half-pressing, and a shooting instruction is input by full-pressing. When the release switch 84 is pressed halfway, shooting preparation processing, that is, AE (Automatic Exposure), AF (Auto Focus), AWB (Automatic White Balance), etc. should be performed before shooting. Process. When the release switch 84 is fully pressed, shooting processing, that is, main imaging of the subject and recording of the captured image are performed. The mode dial 83 is a switch for switching the mode of the digital camera 100.

  FIG. 2 is a rear view of the digital camera 100 shown in FIG. In addition, the same code | symbol is attached | subjected to the component shown in FIG. 1, and description is abbreviate | omitted.

  On the back of the digital camera 100, a monitor 50, a cross key 86, a menu / OK key 87, and a zoom switch 88 are arranged.

  The monitor 50 can be configured using a known display device, but generally, a thin display device such as an LCD (liquid crystal display device) or an organic EL (electroluminescence) display device is used for a portable device. .

  The cross key 86 can be used to input instructions for various settings by operating up, down, left, and right. For example, instructions for brightness adjustment, self-timer setting, macro photography, flash emission, and the like are input.

  The menu / OK key 87 is used to call a menu function that displays a menu screen and accepts input of setting information. The menu / OK key 87 is used to confirm the selection contents displayed on the menu screen and to input a process execution instruction.

  The zoom switch 88 is used for a zoom operation of the photographing lens 24. When the “T” (tele) side indicating telephoto is pressed, the lens barrel 81 is extended to the outside of the camera body 80 and the focal length is set to the telephoto side while continuing to press. When the “W” (wide) side indicating the wide angle is pressed, the lens barrel 81 is drawn into the camera body 80 while the button is being pressed, and the focal length is set to the wide angle side.

  FIG. 3 is a block diagram illustrating an internal configuration example of the digital camera 100 according to the first embodiment. The elements shown in FIGS. 1 and 2 are denoted by the same reference numerals, and description thereof is omitted.

  In FIG. 3, a digital camera 100 is a camera that electronically records an image, and includes a CPU (central processing unit) 10, an operation unit 12, a ROM 14, an SDRAM 16, a VRAM 18, an EEPROM 20, a bus 22, a photographing lens 24, and an image sensor 30. , Timing generator 32, analog signal processing unit 34, A / D converter 36, image input controller 38, image signal processing unit 40, compression / decompression processing unit 42, media interface 44, memory card 46, monitor driver 48, monitor 50 AE detection unit 52, AF detection unit 54, camera shake correction unit 56, flash driver 58, flash 60, power supply control unit 62, battery 64, and the like.

  The CPU 10 functions as a control unit that controls the overall operation of the digital camera 100, and controls each unit of the digital camera 100 according to a program based on an input from the operation unit 12. Further, the CPU 10 functions as an arithmetic processing unit for calculating various control values necessary for controlling the digital camera 100, and executes various arithmetic processes according to a program. Each unit of the digital camera 100 is connected to the CPU 10 via the bus 22.

  The CPU 10 of this embodiment has functions such as AE control, AF control, dynamic range determination, imaging control, motion vector detection, subject determination, image composition, and recording control. These various functions will be described in detail later.

  The operation unit 12 includes hardware switches such as a mode dial 83, a release switch 84, a power switch 85, a cross key 86, a menu / OK key 87, and a zoom switch 88 shown in FIGS. Signals corresponding to the operation of these hardware switches are output to the CPU 10. The digital camera 100 has a playback mode for playing back recorded images in addition to a shooting mode for performing actual imaging of an object and recording of an image. Switching between the shooting mode and the playback mode is performed by a mode dial 83 or the like. Done. Note that a mode switching switch other than the mode dial 83 may be provided in order to switch various detailed modes. The mode can be switched by a menu.

  The ROM 14 stores programs executed by the CPU 10 and various data necessary for control. The SDRAM 16 is used as a work area for the CPU 10 and also as a temporary storage area for image data. The VRAM 18 is used as a temporary storage area dedicated to display image data. The EEPROM 20 stores various setting information unique to the user.

  The photographing lens 24 includes a zoom lens 24z that performs zooming, a focus lens 24f that performs focusing, and an iris 24i that performs light amount adjustment. The taking lens 24 of the present example is a retractable type, and when the digital camera 100 is turned on by the power switch 85, the taking lens 24 is extended from the camera body.

  The zoom lens 24z is driven by a zoom motor 26z and moves on the optical axis. Thereby, the subject image formed on the light receiving surface of the image sensor 30 is optically scaled. The CPU 10 controls the movement of the zoom lens 24z by controlling the driving of the zoom motor 26z via the zoom motor driver 28z, and performs optical zoom switching.

  The focus lens 24f is driven by the focus motor 26f and moves back and forth on the optical axis. Thereby, focusing is performed. The CPU 10 controls the movement of the focus lens 24f by controlling the drive of the focus motor 26f via the focus motor driver 28f, and performs focusing.

  The iris 24i is constituted by, for example, an iris diaphragm, and is driven by an iris motor 26i to change its opening amount (aperture value). The CPU 10 controls the opening amount of the iris 24 i by controlling the driving of the iris motor 26 i via the iris motor driver 28 i, and controls the amount of subject light incident on the light receiving surface of the image sensor 30.

  The image sensor 30 is, for example, a CCD (Charge Coupled Device) image sensor. Other types of image sensors (for example, CMOS (Complementary Metal Oxide Semiconductor) image sensors) may be used instead of the CCD.

  For example, a large number of light receiving elements (hereinafter also referred to as “pixels”) to which color filters of three primary colors of R (red), G (green), and B (blue) are attached are arranged on the light receiving surface of the image sensor 30. Yes. Each pixel is configured by, for example, a photodiode. It is also possible to use color filters in combinations other than the three primary colors (for example, complementary colors).

  The subject light transmitted through the photographic lens 24 forms an image on the light receiving surface of the image sensor 30. The image sensor 30 is driven by a timing generator (TG) 32 to operate. That is, the signal charges accumulated in each pixel are read out by the drive pulse supplied from the timing generator 32 and output as RGB image signals. The CPU 10 controls the exposure timing, exposure time (charge accumulation time, shutter speed), and image signal readout of the image sensor 30 by controlling the driving of the timing generator 32.

  The analog signal processing unit 34 performs correlated double sampling processing on the analog image signal output from the image sensor 30, and amplifies and outputs it.

  The A / D converter 36 converts the analog image signal output from the analog signal processing unit 34 into a digital image signal and outputs the digital image signal.

  The image input controller 38 has a built-in line buffer with a predetermined capacity. Under the control of the CPU 10, the image input controller 38 captures an image signal for one frame output from the A / D converter 36 and stores it in the SDRAM 16.

  The image signal processing unit 40 takes in an image signal stored in the SDRAM 16 under the control of the CPU 10 and performs a required signal processing to obtain an image signal (Y) and a color signal (Cr, Cb). Y / C signal) is generated and written to the VRAM 18.

  The compression / decompression processing unit 42 takes in an image signal (Y / C signal) under the control of the CPU 10 and performs predetermined compression processing to generate compressed image data (for example, JPEG). Further, under the control of the CPU 10, the compressed image data is captured and subjected to a predetermined expansion process to generate an uncompressed image signal (Y / C signal).

  The media interface 44 reads / writes data to / from the memory card 46 under the control of the CPU 10. For example, in response to a recording instruction from the CPU 10, image data obtained by photographing is recorded on the memory card 46. Also, corresponding image data is read from the memory card 46 in response to a read instruction from the CPU 10. The memory card 46 is detachably loaded in a card slot provided in the photographing apparatus main body.

  The monitor 50 is used as a reproduction display unit for a recorded image in the reproduction mode, and an image generated by the image sensor 30 and captured in the SDRAM 16 by the image input controller 38 in the photographing mode as a continuous image in time series. indicate.

  The monitor driver 48 controls display on the monitor 50 under the control of the CPU 10. In this example, an image signal (Y / C signal) is taken from the VRAM 18, converted into a display signal format, and output to the monitor 50.

  The AE detection unit 52 takes in R, G, and B image signals stored in the SDRAM 16 under the control of the CPU 10, and calculates an AE evaluation value (photometric value) indicating the brightness of the captured image necessary for AE control. And output to the CPU 10.

  The AF detection unit 54 takes in R, G, and B image signals stored in the SDRAM 16 under the control of the CPU 10, and calculates an AF evaluation value (contrast value) indicating the contrast of the captured image necessary for AF control. , Output to the CPU 10. The AF detection unit 54 of this example includes a high-pass filter that passes only a high-frequency component of the G signal, an absolute value processing unit, a focus area extraction unit that extracts a signal within a predetermined focus area set by the operation unit 12, and An integration unit that integrates the absolute value data in the focus area is included, and the absolute value data in the focus area integrated by the integration unit is output to the CPU 10 as an AF evaluation value.

  The camera shake correction unit 56 performs camera shake correction in accordance with an instruction from the CPU 10.

  The flash driver 58 drives the flash 60 under the control of the CPU 10. That is, the flash 60 is caused to emit light at the light emission timing and the light emission amount instructed from the CPU 10.

  The power control unit 62 controls the supply of power from the battery 64 loaded in the apparatus body to each unit of the digital camera 100 in accordance with an instruction from the CPU 10.

  Note that the imaging unit 23 is configured by devices 24 to 36 in FIG. 3 (including the imaging lens 24, the imaging device 30, the analog signal processing unit 34, and the A / D converter 36).

  FIG. 4 is a plan view schematically showing a configuration example of the image sensor 30.

  A large number of pixels (light receiving elements) to which red, green, and blue color filters are attached are arranged on the light receiving surface of the image sensor 30. In FIG. 4, a group of pixels with uppercase R, G, and B is hereinafter referred to as “A-plane pixel group”, and a group of pixels with lowercase r, g, and b is referred to as “B-plane pixel”. "Group". Both the A side pixel group and the B side pixel group include pixels to which filters of three primary colors of red, green, and blue are respectively attached. As a result, it is possible to generate a full-color image using only the A-plane pixel group or the B-plane pixel group.

  The imaging element 30 of this example has a pixel arrangement called a honeycomb arrangement, and A-plane pixel groups (R, G, B pixel groups) are arranged in a grid in the row direction and the column direction. The B-plane pixel group (r, g, b pixel group) is arranged in a grid pattern with a shift of half the pixel pitch (1/2 pixel pitch) in the row direction and the column direction with respect to the group. That is, the honeycomb arrangement is such that the B-plane pixel group is arranged between the lattices of the A-plane pixel group.

  Also, two pixels of the same color of red (R, r), green (G, g), and blue (B, b) are adjacent to each other in an oblique direction. By treating these two adjacent pixels as one pixel and performing signal processing, the dynamic range can be expanded.

  In this example, the A color pixel and B color pixel (for example, R pixel and r pixel) of the same color have the same shape and the same saturation charge amount. In addition, the A plane pixel and the B plane pixel are provided with independent charge transfer paths (80A, 80B) and readout electrodes (82A, 82B), respectively. The charges accumulated in the A plane pixel and the B plane pixel The accumulated charges can be read out and transferred independently of each other.

  The A-plane pixel group and the B-plane pixel group of the image sensor 30 can be imaged by one exposure with different exposure times for each pixel group. For example, exposure can be started at the same time for the A-plane pixel group and the B-plane pixel group, and the exposure of the A-plane pixel group can be terminated after the exposure of the B-plane pixel group is completed. Thereby, an A-side image obtained by long-time exposure of the A-side pixel group and a B-side image obtained by short-time exposure of the B-side pixel group are obtained. For such driving of the image sensor 30, for example, a method described in Japanese Patent Application Laid-Open No. 2007-235656 may be used, and detailed description thereof is omitted here. However, the present invention is not particularly limited when exposure is simultaneously started in the A-plane pixel group and the B-plane pixel group. For example, the exposure period can be overlapped and the exposure time can be made different also in an aspect in which the exposure start time is made different and an aspect in which both the exposure start time and the exposure end time are made different.

  In addition, by using the A-plane pixel group and the B-plane pixel group of the image pickup device 30 as one high-resolution pixel group, and imaging with the same exposure time in the A-plane pixel group and the B-plane pixel group, the image by pixel group It is also possible to obtain a high-resolution image having a higher resolution than (A-plane image, B-plane image).

  Note that the shape of the pixel is not limited to the rectangular shape shown in FIG. For example, hexagonal or octagonal pixels may be used.

  In the present invention, the pixel arrangement is not particularly limited to the honeycomb arrangement. For example, a square lattice arrangement in which a large number of pixels are arranged in a square lattice shape at regular intervals in the row direction and the column direction, and every other A plane pixel and B plane pixel are arranged in each of the row direction and the column direction. May be.

  FIG. 5 is a main part functional block diagram of the CPU 10 in the first embodiment.

  The AE control unit 102 performs automatic exposure (AE) control based on the AE evaluation value calculated by the AE detection unit 52.

  The AF control unit 104 performs automatic focusing (AF) control based on the AF evaluation value calculated by the AF detection unit 54.

  The dynamic range determination unit 106 determines whether or not the dynamic range of the captured image output from the imaging unit 23 is within an appropriate range. Specifically, the dynamic range is compared with a threshold value, and if the dynamic range is less than the threshold value, it is determined that the dynamic range is within an appropriate range and expansion is not necessary. That is, it is estimated that saturation does not occur when the main imaging is performed under the standard imaging conditions. If the dynamic range is equal to or greater than the threshold, it is determined that the dynamic range is outside the appropriate range and expansion is necessary. In this case, there is a possibility of saturation when actual imaging is performed under the standard imaging conditions. Here, the main imaging is imaging performed to obtain an image to be recorded after a shooting instruction is input by the release switch 84.

  The imaging control unit 108 controls the imaging element 30 using the timing generator 32. A captured image obtained by imaging is stored in the SDRAM 16.

  In this example, pixel group-by-pixel imaging (first imaging) in which the A-plane pixel group and the B-plane pixel group of the image sensor 30 are imaged by one exposure (single imaging) with different exposure times for each pixel group; The same exposure time is set as one high-resolution pixel group for the A-plane pixel group and the B-plane pixel group, and images are taken by multiple exposures (multiple imaging) at different exposure times each time. High-resolution continuous shooting (second imaging) is continuously performed. In this case, at least three exposures (three times imaging) are performed. Any of the order of pixel group imaging and high-resolution continuous shooting may be first.

  The motion vector detection unit 110 detects a motion vector of the subject from a plurality of images obtained by continuously imaging the subject. In addition, the motion vector detection unit 110 of the present embodiment detects the motion vector of the camera body 80 from a plurality of images together with the motion vector of the subject.

  The subject determination unit 112 determines whether the subject being continuously shot is in a stationary state or a moving state based on the motion vector of the subject. In the following, a stationary subject may be referred to as a “stationary object” and a moving subject may be referred to as a “moving object”.

  When the subject determination unit 112 determines that the subject being continuously shot is in a moving state, the image composition unit 114 performs one exposure (one time) with different exposure times for the A-side pixel group and the B-side pixel group. The A-side image and B-side image obtained by imaging) are synthesized. Further, the image composition unit 114 treats the A-plane pixel group and the B-plane pixel group as one high-resolution pixel group when the subject determination unit 112 determines that the subject being continuously shot is stationary. A plurality of high-resolution images obtained by performing a plurality of exposures (imaging a plurality of times) continuously at different exposure times each time are synthesized.

  That is, the image composition unit 114 has an image (long-time exposure image) captured with the overexposure input / output characteristic 201 shown in FIG. 6A and an underexposure input / output characteristic 202 shown in FIG. The captured image (short-time exposure image) is added. In overexposure, the dynamic range is large in the low luminance region where the incident light amount is small, and is saturated in the high luminance region where the incident light amount is large. Underexposure does not saturate even in high brightness areas. Note that when the long-time exposure image and the short-time exposure image are simply added, the same image as that captured with the input / output characteristics indicated by reference numeral 203 in FIG. 6C is obtained, the input / output characteristics are not smooth, and the saturation point is obtained. Therefore, the same image as that captured with the input / output characteristics indicated by reference numeral 204 in FIG. 6C is generated by weighted addition using the γ table.

  The image generated by the image composition of the image composition unit 114 is hereinafter referred to as “composite image” or “wide dynamic range image”.

  Further, the image composition unit 114 of the present embodiment synthesizes a plurality of continuous shot images using the motion vector of the camera body 80 detected by the motion vector detection unit 110.

  The recording control unit 120 records the wide dynamic range image on the memory card 46 via the media interface 44.

  The recording control unit 120 of this example records the composite image generated by the image composition unit 114 on the memory card 46.

  The system control unit 122 controls each part in the CPU 10 and each part outside the CPU 10.

  FIG. 7 is a flowchart illustrating a flow of an imaging process example in the first embodiment. This process is executed by the CPU 10 of FIG. 3 according to the program.

  When the release switch 84 is half-pressed, the AE control unit 102 performs AE processing in step S1. In the AE process, the brightness (brightness) of the captured image is obtained based on the AE evaluation value calculated by the AE detector 52, and the reference exposure time and aperture value are determined. For example, it is determined using a program diagram.

  In step S2, the AF control unit 104 performs AF processing. In the AF process, the focus motor driver 28f gradually moves the focus lens 24f from the close-up lens position focused at the close distance to the infinitely-focused lens position focused at infinity while the AF detection unit 54 adjusts the contrast of the captured image. An AF evaluation value is calculated, and focus detection processing is performed in which the focus lens 24f determines the lens position at which the contrast reaches a peak as the focus lens position (the position of the focus lens 24f that focuses on the main subject). Then, the focus lens 24f is moved to the determined focus lens position.

  In this example, the AE process and the AF process are performed before the release switch 84 is fully pressed, but at least one of the AE process and the AF process may be performed after the release switch 84 is fully pressed.

  In step S3, the dynamic range determination unit 106 compares the dynamic range of the captured image output from the imaging unit 23 with a threshold (for example, 6 EV) to determine whether the dynamic range is within an appropriate range. . If the dynamic range is greater than or equal to the threshold, the process proceeds to step S4, and if less than the threshold, the process proceeds to step S9.

  In this example, based on the AE evaluation value (photometric value) calculated in step S1, the light / dark difference (luminance difference) in the captured image before the main imaging is calculated as a dynamic range. For example, the captured image is divided into a plurality of regions, the integrated value of luminance is calculated for each divided region by the AE detection unit 52, the maximum value and the minimum value of the integrated values in all the regions are obtained, and the appropriate exposure value ( The EV value is converted into an appropriate EV value, and the difference between the maximum value and the minimum value is calculated as a dynamic range.

  Although the case where the dynamic range determination is performed based on the captured image after the release switch 84 is half-pressed has been described as an example, the dynamic range determination may be performed based on a through image captured before the release switch 84 is half-pressed.

  When the release switch 84 is fully pressed, in step S4, the imaging control unit 108 controls the imaging unit 23, and as shown in FIG. 8A, first, the exposure time t1 of the A-plane pixel group is set to B. Exposure is performed once longer than the exposure time t2 of the surface pixel group, and the accumulated charge is read for each pixel group (first imaging). Next, the A-plane pixel group and the B-plane pixel group are used as one high-resolution pixel group, exposed twice with different exposure times (t2 <t1), and the accumulated charge is read from all the pixel groups each time (first) Second and third imaging). The first image pickup (image pickup for each pixel group) obtains two images for each pixel group of normal resolution Rn having different dynamic ranges, and the second and third image pickup (high-resolution continuous shot) for the dynamic range. Two high resolution images with different high resolution Rh (= Rn × 2) are obtained.

  In step S 5, the motion vector detection unit 110 detects the motion vector of the subject and the motion vector of the camera body 80.

  Although motion vectors can be detected between images with different imaging conditions (exposure time, aperture value, etc.), in order to improve the detection accuracy of motion vectors, multiple images obtained by imaging the subject under the same imaging conditions It is preferable to detect between these images. For example, a motion vector is detected from the A plane image obtained by the first imaging and the A plane image included in the high-resolution image obtained by the third imaging. A B-side image may be used.

  The motion vector may be detected using a known technique. For example, when the main subject is identified as a person, the face portion is detected in each continuous shot image, the motion vector of the face portion is detected between a plurality of continuous shot images, and this is used as the motion vector of the subject. There are aspects. Here, the periphery of the face portion (or the peripheral portion of the captured image) may be estimated as the background, the background motion vector may be detected, and this may be estimated as the motion vector of the camera body 80 (ie, the amount of camera shake).

  Even when the main subject is arbitrary without being specified by a person or the like, the motion vector of the subject and the camera body 80 can be detected. For example, for each divided region of the captured image, a motion vector is detected between a plurality of continuous shot images, a correlation between the motion vectors is obtained between the divided regions, and the motion of each divided region is determined based on the correlation. It is possible to extract the motion vector of the subject and the motion vector of the camera body 80 by determining which of the motion of the subject and the motion of the camera body 80 is added or determined.

  Of course, the motion vector of the subject and the motion vector of the camera body 80 may be detected using a method other than these.

  In step S6, the subject determination unit 112 determines whether the subject during continuous shooting is a stationary body (that is, a stationary state) or a moving body (that is, a moving state) based on the motion vector of the subject.

  If it is determined that the object is a moving object, in step S7, the image composition unit 114 synthesizes the A-side image and the B-side image to generate a composite image with an expanded dynamic range.

  For example, an A-plane image captured with the overexposure input / output characteristic 201 shown in FIG. 6A, that is, an image classified by pixel group that is not saturated in the low luminance region but saturated in the high luminance portion, and FIG. The B-side image captured with the underexposure input / output characteristic 202 shown in FIG. 5, that is, the pixel group-specific image that does not saturate even in the high luminance portion, and these pixel group-specific images are added. That is, by adding the pixel values of the A-side image and the B-side image, a wide dynamic range image having a wide dynamic range from the low luminance region to the high luminance region is generated. Note that when the A-side image and the B-side image are simply added, the same image as that obtained when the image is captured with the input / output characteristics indicated by reference numeral 203 in FIG. 6C is generated. By using this, the same image as that obtained when the image is captured with the input / output characteristics indicated by reference numeral 204 in FIG. 6C is generated. That is, weighted addition using the γ table is performed. Thereby, a wide dynamic range image with good input / output characteristics can be obtained.

  If it is determined that the object is a stationary body, in step S8, the image composition unit 114 synthesizes two high-resolution images obtained by the second and third imaging, thereby wide dynamic range image. Is generated. That is, an over-exposed high-resolution image captured with the first input / output characteristic 201 shown in FIG. 6A and an under-exposed high image captured with the second input / output characteristic 202 shown in FIG. 6B. Add the resolution image. Note that when two high-resolution images are simply added, the same composite image as when captured with the input / output characteristics indicated by reference numeral 203 in FIG. 6C is generated. Using the table, the same composite image as that obtained when the image is captured with the input / output characteristics indicated by reference numeral 204 in FIG. 6C is generated. Thereby, a wide dynamic range image with good input / output characteristics can be obtained.

  Even in the case of a stationary body, when the camera body 80 moves, so-called camera shake increases additively when a plurality of continuous shot images are combined. Therefore, when compositing a high-resolution image, the motion vector of the camera body 80 is used to synthesize the image without adding a camera shake component. Specifically, another high-resolution image is relatively shifted by image processing with respect to one high-resolution image and then combined. Here, the relative shift of the high-resolution image corresponds to the inverse vector of the motion vector of the camera body 80.

  The wide dynamic range image obtained by the image composition is subjected to predetermined image processing by the image signal processing unit 40 in step S10, compressed by the compression / decompression processing unit 42 in step S11, and recorded in step S12. The data is recorded on the memory card 46 by the unit 120.

  If it is determined in step S3 that the dynamic range is less than the threshold value, once the release switch 84 is fully pressed, in step S9, once with the reference exposure time and aperture value calculated in step S1. One high-resolution recording image is acquired by imaging. The image processing in step S10, the compression processing in step S11, and the recording processing in step S12 are the same as in the case of the above-described continuous shooting.

  In addition, although the case where it imaged 3 times (3 times exposure) was demonstrated to the example, this invention is not limited to such a case. Image synthesis may be performed by imaging twice (exposure twice) as follows.

  In the case of two-time imaging, in step S4 in FIG. 7, the imaging control unit 108 continuously images a plurality of times with different exposure times for each pixel group in the A-plane pixel group and the B-plane pixel group of the image sensor 30. . Here, each time an image is taken, the accumulated charge is read from each pixel group, and the exposure times are interchanged between the pixel groups. For example, as shown in FIG. 8B, in the first imaging, the second exposure time t2 that is shorter than the first exposure time t1 at the same time as the exposure with the A-plane pixel group at the first exposure time t1. Then, exposure is performed with the B-side pixel group, and the accumulated charge is read from all the pixel groups. In the second imaging, the exposure time is changed, and exposure is performed by the A-plane pixel group at the second exposure time t2, and at the same time, exposure is performed by the B-plane pixel group at the first exposure time t1, and accumulated charges are collected from all the pixel groups. Read.

  In the case of twice imaging, when the subject determining unit 112 determines that the subject is in a moving state in step S6 in FIG. 7, the image synthesizing unit 114 performs one imaging in step S7 in FIG. A first wide dynamic range image is generated by combining the A-side image and the B-side image obtained in (for example, the first imaging). When it is determined that the subject is in a stationary state, in step S8 in FIG. 7, the image composition unit 114 synthesizes a plurality of images by pixel group obtained by a plurality of times of imaging for each identical pixel group. At the same time, a high-resolution wide dynamic range image is generated by combining the pixel groups. That is, the A-side image with a wide dynamic range is obtained by adding the A-side image with the exposure time t1 obtained by the first imaging and the A-side image with the exposure time t2 obtained by the second imaging. And the B-side image of the exposure time t2 obtained by the first imaging and the B-side image of the exposure time t1 obtained by the second imaging are added to add a B of a wide dynamic range. A plane image is generated, and a wide dynamic range image with a high resolution is generated by combining the A plane image and the B plane image having a wider dynamic range.

  FIG. 9 is an overall configuration diagram illustrating an example of the digital camera 100 according to the second embodiment. The elements shown in FIG. 3 are denoted by the same reference numerals, and the description of the contents already described is omitted.

  In FIG. 9, the angular velocity sensor 70 detects the motion vector (that is, the amount of camera shake) of the imaging unit 23 by detecting the motion of the camera body 80.

  FIG. 10 is a main part functional block diagram of the CPU 10 in the second embodiment. The elements shown in FIG. 3 are denoted by the same reference numerals, and the description of the contents already described is omitted.

  The motion vector detection unit 110 according to the present embodiment detects a motion vector of a subject based on a plurality of captured images obtained by the image sensor 30 and the motion of the camera body 80 detected by the angular velocity sensor 70. Further, the image composition unit 114 of the present embodiment synthesizes a plurality of continuous shot images using the detection result of the angular velocity sensor 70.

  FIG. 11 is a flowchart illustrating a flow of an imaging process example in the second embodiment. This processing is executed by the CPU 10 of FIG. 9 according to the program.

  Steps S21 to S24 are the same as S1 to S4 in the first embodiment shown in FIG.

  In step S25, in this embodiment, the motion vector detection unit 110 detects the motion vector of the subject based on the plurality of captured images and the detection result of the angular velocity sensor 70. In this embodiment, the angular velocity sensor 70 detects the movement of the camera body 80 (that is, the amount of camera shake). Therefore, the result of subtracting the motion vector of the camera body 80 detected by the angular velocity sensor 70 from the motion vector between a plurality of captured images corresponds to the motion vector of the subject.

  In step S26, the subject determination unit 112 determines whether the subject is stationary or moving based on the motion vector of the subject.

  In step S28, in this embodiment, a plurality of continuous shot images are synthesized using the detection result of the angular velocity sensor 70.

  In the first embodiment described above, image synthesis is performed using the motion vector of the camera body 80 detected by the motion vector detection unit 110 so as not to add a camera shake component, but in this embodiment, the angular velocity sensor 70 is used. By using the detected motion vector of the camera body 80, it is possible to reliably eliminate the addition of camera shake. Specifically, the other captured image is relatively shifted by image processing with respect to one captured image and then combined. Here, the relative shift vector of the captured image corresponds to the inverse vector of the motion vector of the camera body 80 detected by the angular velocity sensor 70.

  Steps S27 and S29 to S32 are the same as S7 and S9 to S12 in the first embodiment shown in FIG.

  In the first embodiment and the second embodiment described above, the number of continuous shooting is not particularly limited. It goes without saying that by increasing the number of images, the accuracy of motion vector detection and the accuracy of a wide dynamic range image may be improved.

  Moreover, although the case where image composition is performed by the calculation of the CPU 10 has been described as an example, it is not particularly limited to such a case. It goes without saying that image synthesis and gain increase may be performed by a hardware circuit.

  Moreover, although the case where the exposure time is switched as an example of switching the imaging condition of the high resolution image has been described, the present invention is not particularly limited to such a case. For example, a plurality of continuous shot images having different dynamic ranges may be acquired by switching the aperture value. Further, a plurality of continuous shot images having different dynamic ranges may be acquired by using a light source with a variable light emission amount as the flash 60 and switching the illuminance on the subject. Further, a filter may be provided in the photographing lens 24 to switch the light transmittance of the photographing lens 24. Further, so-called sensitivity may be switched by switching the gain in the analog signal processing unit 34.

  In the first embodiment, a case where the motion of the camera body 80 is electronically detected by image analysis will be described as an example. In the second embodiment, a motion of the camera body 80 is detected by the angular velocity sensor 70. However, it is a matter of course that the motion of the camera body 80 may be detected with high accuracy by using image analysis and a sensor together.

  In the second embodiment, the case where an angular velocity sensor (gyro sensor) is used as the sensor for detecting the movement of the camera body 80 has been described as an example. However, the sensor is not particularly limited to the angular velocity sensor. Any sensor that can detect the amount of movement of the camera body 80 may be used.

  Although detailed description of the camera shake correction unit 56 is omitted, camera shake correction can be performed using a known technique. Camera shake correction may be performed by either hardware or software. For example, camera shake correction can be performed by using known hardware that moves the image sensor 30 and the photographing lens 24 in accordance with the amount of camera shake. Camera shake correction may be performed on the image signal using a hardware circuit. Although the description of subject blur correction has been omitted, it goes without saying that subject blur correction may be performed using a known technique. At the time of these blur corrections, at least one of the motion vector detected by the motion vector detection unit 110 and the detection result of the angular velocity sensor 70 can be used.

  In this specification, a digital camera has been described as an example. However, the present invention is not particularly limited to a digital camera, and can be applied to any imaging device having an imaging function, such as a mobile phone with a camera.

  The present invention is not limited to the examples described in the present specification and the examples illustrated in the drawings, and various design changes and improvements may be made without departing from the scope of the present invention. is there.

  DESCRIPTION OF SYMBOLS 10 ... CPU, 12 ... Operation part, 16 ... SDRAM, 23 ... Imaging part, 24 ... Shooting lens, 30 ... Imaging element, 32 ... Timing generator, 34 ... Analog signal processing part, 36 ... A / D converter, 38 ... Image input controller 44 ... Media interface 46 ... Memory card 52 ... AE detector 54 ... AF detector 70 ... Angular velocity sensor 84 ... Release switch 100 ... Digital camera 106 ... Dynamic range determiner 108 ... Imaging control unit, 110 ... motion vector detection unit, 112 ... subject determination unit, 114 ... image composition unit, 118 ... image correction unit, 120 ... recording control unit, 122 ... system control unit

Claims (10)

An image sensor having a plurality of pixel groups;
First imaging for imaging with different exposure times for each pixel group of the imaging device, and imaging the plurality of pixel groups as a single high-resolution pixel group for a plurality of times under different imaging conditions each time. Imaging control means for performing second imaging;
Motion vector detection means for detecting a motion vector of a subject from an image obtained by the image sensor;
Subject determination means for determining whether the subject is stationary or moving based on at least a motion vector of the subject;
When it is determined that the subject is in a moving state, a first wide dynamic range image is generated by synthesizing a plurality of pixel group-by-pixel images obtained by the first imaging, and the subject Is determined to be in a stationary state, a second resolution higher than that of the first wide dynamic range image is synthesized by combining a plurality of high resolution images obtained by the second imaging. Image synthesizing means for generating a wide dynamic range image of
An imaging apparatus comprising: An image sensor having a plurality of pixel groups;
An imaging control unit that captures images with different exposure times for each of the pixel groups of the image sensor, and sequentially captures a plurality of times by exchanging exposure times between the pixel groups;
Motion vector detection means for detecting a motion vector of a subject from an image obtained by the image sensor;
Subject determination means for determining whether the subject is stationary or moving based on at least a motion vector of the subject;
When it is determined that the subject is in a moving state, a first wide dynamic range image is generated by synthesizing a plurality of images for each pixel group obtained by one imaging, and the subject is stationary. When it is determined that the state is in a state, the first wide dynamic range is obtained by combining a plurality of images by pixel group obtained by imaging a plurality of times for the same pixel group and combining them between the pixel groups. Image synthesizing means for generating a second wide dynamic range image having a higher resolution than the image;
An imaging apparatus comprising: A sensor for detecting the movement of the imaging device;
The imaging apparatus according to claim 1, wherein the image composition unit performs image composition using a detection result of the sensor.   The imaging apparatus according to claim 3, wherein the motion vector detection unit detects a motion vector of the subject based on an image obtained by the imaging element and a detection result of the sensor. The motion vector detecting means detects a motion vector of the imaging apparatus from an image obtained by the imaging element;
The imaging apparatus according to claim 1, wherein the image synthesis unit performs image synthesis using a motion vector of the imaging apparatus.   The imaging apparatus according to claim 1, wherein the motion vector detection unit detects the motion vector using a plurality of images captured under the same imaging condition.   The image pickup device according to claim 1, wherein the image pickup element has a honeycomb arrangement in which a second pixel group is arranged between lattices of the first pixel group.   The imaging device according to any one of claims 1 to 6, wherein the imaging device has a square lattice arrangement in which a large number of pixels are arranged in a square lattice pattern at regular intervals in two directions. A first imaging that uses an imaging device having a plurality of pixel groups and images with different exposure times for each of the pixel groups of the imaging device, and the plurality of pixel groups as one high-resolution pixel group each time An imaging step of performing a second imaging for imaging a plurality of times continuously under different imaging conditions;
A motion vector detection step of detecting a motion vector of a subject from the image obtained in the imaging step;
A subject determination step for determining whether the subject is stationary or moving based on at least a motion vector of the subject;
When it is determined that the subject is in a moving state, a first wide dynamic range image is generated by synthesizing a plurality of pixel group-by-pixel images obtained by the first imaging, and the subject Is determined to be in a stationary state, a second resolution higher than that of the first wide dynamic range image is synthesized by combining a plurality of high resolution images obtained by the second imaging. An image synthesis step for generating a wide dynamic range image of
An imaging method comprising: An imaging step that uses an imaging device having a plurality of pixel groups, images with different exposure times for each of the pixel groups of the imaging device, and alternately switches the exposure times between the pixel groups and continuously images a plurality of times. ,
A motion vector detection step of detecting a motion vector of a subject from the image obtained in the imaging step;
A subject determination step for determining whether the subject is stationary or moving based on at least a motion vector of the subject;
When it is determined that the subject is in a moving state, a first wide dynamic range image is generated by synthesizing a plurality of images for each pixel group obtained by one imaging, and the subject is stationary. When it is determined that the state is in a state, the first wide dynamic range is obtained by combining a plurality of images by pixel group obtained by imaging a plurality of times for the same pixel group and combining them between the pixel groups. An image synthesis step for generating a second wide dynamic range image having a higher resolution than the image;
An imaging method comprising:

Images