JP2007104200A - Imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus capable of properly correcting the exposure and the white balance independently in spite of a rapid change in a screen image. <P>SOLUTION: The imaging apparatus for photographing a moving picture by sequentially recording image data of each frame obtained by applying photoelectric conversion to an object image formed on an imaging element through a photographing lens includes: a region setting means for setting an optional reference region to an imaging range captured by the imaging element on the basis of characteristics of the image data of a prescribed frame; a tracking means that sets a reference region to each frame by tracking the object image or its part captured by the reference region set to the prescribed frame from each of succeeding frames; and a control value calculation means that calculates a control value associated with at least either of an exposure time and the white balance on the basis of pixel values of each pixel included in the reference region. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image pickup apparatus that performs moving image shooting while automatically correcting exposure and white balance based on image data obtained by photoelectrically converting a subject image formed on an image pickup element by a shooting lens. .

In an image pickup apparatus that performs moving image shooting, image data obtained by an image pickup device such as a CCD is read out in a full screen mode in which pixel values of all pixels of the image pickup device are read and a line selected for each of a plurality of lines is selectively used. In general, a draft mode for reading data is prepared.
In a conventional imaging apparatus, appropriate control values for exposure and white balance adjustment are determined based on the draft image read in the draft mode described above, and these control values are read in the full screen mode. Applied to captured image data.

Since the draft image is an image extracted uniformly from the entire image sensor, the exposure and white balance are adjusted based on the information on the entire field of view captured by the image sensor by the above-described processing.
On the other hand, a technique has also been proposed in which a control value related to white balance or the like is independently determined for each region cut out from an image of one frame, and the determined control value is applied to each region (see Patent Document 1).

In addition, in the conventional exposure and white balance adjustment technology, in order to avoid the control value fluctuating from frame to frame depending on camera shake or illumination fluctuation, in particular, in determining the control value used for white balance adjustment, In many cases, strong hysteresis is provided.
Japanese Patent Laid-Open No. 2004-187124

  By the way, in moving image shooting, the position of the main subject may change greatly in the screen, or the area of the area occupied by the elephant of the subject in the screen may vary greatly. In such a case, if the white balance is determined in consideration of the color information of the entire screen according to the conventional technique, the skin color of the face of the person who is the subject is determined when the face of the person occupies most of the screen. In some cases, the color of the illumination light that illuminates the background is removed.

  In the conventional technique, the draft image and the main image have the same accumulation time. Therefore, if the image sensor becomes saturated, such as when the camera pans following a subject that has moved in the sun from the shade, naturally the draft image used to calculate the exposure adjustment control value will also be overexposed. End up. In such a case, since it is necessary to search for an appropriate exposure time while gradually reducing the accumulation time, it takes time to determine an appropriate exposure.

In addition, control with a large hysteresis as in the past can prevent the white balance from fluctuating from frame to frame due to the effects of minute fluctuations such as camera shake and lighting fluctuations. It is difficult to quickly respond to changes in image characteristics when the subject to be changed changes greatly.
An object of the present invention is to provide an imaging apparatus capable of appropriately correcting exposure and white balance regardless of a sudden change in the screen.

A first imaging apparatus according to the present invention includes an area setting unit, a tracking unit, and a control value calculation unit.
The principle of the first imaging apparatus according to the present invention is as follows.
In an imaging apparatus that shoots moving images by sequentially recording image data of each frame obtained by photoelectrically converting a subject image formed on an imaging element by a photographing lens, an area setting unit includes an image of a predetermined frame An arbitrary reference area is set in the imaging range captured by the imaging device based on the data characteristics. The tracking means tracks the subject image or a part thereof captured in the reference area set for a predetermined frame for each subsequent frame, and sets a reference area for each frame. The control value calculation means calculates a control value related to at least one of the exposure time and the white balance in each corresponding frame based on the pixel value of each pixel included in the reference region.

The operation of the first imaging apparatus configured as described above is as follows.
For example, the region setting means sets a region indicating the representative color of the main subject captured in the screen (for example, a skin color region when the subject is a person) as a reference region, and the tracking unit sets this region. The reference area is made to follow the movement of the subject in the screen, and the pixel value of each pixel included in the reference area is used for the control value calculation process. In this way, by making the reference region follow the movement of the subject in the screen, the exposure value and the image data representing the same portion of the subject (for example, the cheek portion of a person) in each frame are determined. A control value related to white balance is calculated. By applying the control values obtained in this way to exposure correction and white balance correction, stable image processing can be realized.

A second image pickup apparatus according to the present invention is configured by including a reading means in the control value calculation means provided in the first image pickup apparatus described above.
The principle of the second imaging apparatus according to the present invention is as follows.
In the control value calculation means provided in the first imaging device described above, the reading means has the accumulation time determined independently of the recording of the image data for moving image shooting with respect to the pixel value of each pixel included in the reference area. The data is read out and used for calculation processing for obtaining a control value.

The operation of the second imaging apparatus configured as described above is as follows.
The accumulation time when the pixel value of each pixel belonging to the reference area is read by the reading means is set independently of normal moving image data. For example, if the readout means reads out the pixel value of each pixel in the reference area with a shorter accumulation time than normal moving image data, the pixel value read out by this readout means even when the illuminance of the subject increases rapidly Is likely not saturated, and it is possible to quickly search for an appropriate exposure time.

A third imaging apparatus according to the present invention includes a tracking unit included in the first imaging apparatus described above, and a holding unit, a collecting unit, and an estimating unit.
The principle of the third imaging apparatus according to the present invention is as follows.
In the tracking means provided in the first imaging device described above, the holding means holds information indicating the image of the subject captured in the set reference area or a part of the characteristics thereof. The collecting means collects the photographing conditions including the angular velocity, the focal length, and the photographing distance obtained by the angular velocity sensor provided in the photographing lens for a new frame. The estimation unit estimates a range in which a reference region may exist in a new frame based on the imaging conditions collected by the collection unit. The extraction unit extracts a reference region in a new frame from the range indicated by the estimation result by the estimation unit based on the feature information held in the holding unit.

The operation of the third imaging apparatus configured in this way is as follows.
Based on the amount of camera shake, the focal length, and the change in the shooting distance indicated by the shooting conditions collected by the collecting unit, the approximate position that the reference area occupies in the new frame is estimated by the estimating unit. From the range indicated by the estimation result, for example, a region having a feature (for example, color) that matches the feature information held in the holding unit is extracted by the extracting unit, and used for processing by the control value calculating unit.

A fourth imaging device according to the present invention is configured by including, in the tracking unit provided in the above-described third imaging device, an identity determination unit and a setting instruction unit.
The principle of the fourth imaging apparatus according to the present invention is as follows.
In the tracking means provided in the third imaging apparatus described above, the identity determination means is based on the imaging conditions collected by the collection means and information indicating operations related to imaging including power-on and recording start of the imaging apparatus. Thus, the identity of the subject feature in the previous frame and the subject feature in the new frame is determined. The setting instructing unit instructs the region setting unit to set a new reference region when the identity regarding the subject feature is denied by the identity determining unit.

The operation of the fourth imaging device configured as described above is as follows.
For example, when the angular velocity sensor indicates that the field of view of the imaging device has been panned greatly, or when the focal length or shooting distance has changed significantly, a completely different subject is captured on the screen by the photographer's intention. Probability is high. In such a case, the identity determination unit obtains a determination result that the feature of the subject in the previous frame and the feature of the subject in the new frame are not the same, and accordingly, the setting instruction unit The area setting unit is instructed to set a new reference area.

  As described above, according to the imaging device according to the present invention, the exposure based on the image data representing the same portion of the subject (for example, the cheek portion of a person) according to the movement of the subject in the screen. A control value related to the value and white balance can be calculated and applied to exposure correction and white balance correction. Thus, for example, even when the subject approaches the imaging device and a sudden screen change occurs where the subject image occupies most of the screen, appropriate exposure correction and white balance correction are performed. It is possible to improve the stability of the image processing.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
(First embodiment)
FIG. 1 shows a first embodiment of an imaging apparatus according to the present invention.
In the imaging apparatus shown in FIG. 1, an operation instruction by the user including power-on and shooting button operation is received by the operation reception unit 201, and the input operation instruction is interpreted by the operation reception unit 201. The image processing unit 210, the recording processing unit 203, and the image quality adjustment processing unit 220 are used to create control instructions.

  In the imaging processing unit 210 shown in FIG. 1, an image of a subject formed on the imaging device 212 by the optical system 211 is converted into an electric signal by a photoelectric conversion action by the imaging device 212, and further, the signal processing unit 213. Thus, it is converted into image data representing the image of the subject and held in the memory 214. In addition, the automatic focus control unit 215 shown in FIG. 1 measures the image data held in the memory 214 or the distance to the subject by the distance measurement unit (not shown) in response to an instruction from the operation reception unit 201. Based on the result, the optical system 211 is operated to keep the subject in focus. On the other hand, the detection result obtained by the angular velocity sensor 216 shown in FIG. 1 is used to drive a correction optical system (not shown) provided in the optical system 211 and is processed by the image quality adjustment processing unit 220. Provided.

Also, the recording processing unit 203 illustrated in FIG. 1 starts a recording operation in response to a recording start instruction input via the operation receiving unit 201, and converts the image data of each frame obtained by the imaging processing unit 210 into image quality. The data is received via the adjustment processing unit 220 and is recorded on the recording medium 202 after appropriate conversion processing (for example, conversion processing to M-JPEG data).
Next, a detailed configuration of the image quality adjustment processing unit 220 will be described along with its operation.

FIG. 2 is a flowchart showing the image quality adjustment operation.
When the recording start button is operated by the user, the image quality adjustment operation by the image quality adjustment processing unit 220 is started in response to an instruction from the operation receiving unit 201 shown in FIG.
First, the adjustment control unit 224 of the image quality adjustment processing unit 220 sends appropriate instructions to the read processing unit 221 and the control value calculation unit 226 to perform initial setting of the image quality control operation (step 301). At this time, the adjustment control unit 224 instructs the reading processing unit 221 to read in the draft mode, and passes the draft image obtained in response thereto to the control value calculation unit 226. Based on the draft image, the control value calculation unit 226 executes a control value calculation process similar to the conventional one, determines an exposure control value and a white balance control value, and uses these image control values as processing of the image quality adjustment unit 227. To serve.

  At this time, based on the focus control information obtained from the automatic focus control unit 215 by the reference region determination unit 222 shown in FIG. 1, for example, one of the areas of the screen where the subject in focus is captured. Are extracted as a reference area (for example, an area of 10 pixels × 10 pixels). Then, the address information indicating this area is passed to the adjustment control unit 224, and based on the draft image obtained for the area indicated by this address information, the adjustment control unit 224 causes the color and brightness of the image in the reference area described above. The reference data is obtained and stored in the reference data holding unit 225 together with the address indicating the position of the reference area (step 302 in FIG. 2).

  Next, based on the exposure control value included in the instruction from the image quality adjustment unit 227 shown in FIG. 1, the reading processing unit 221 causes the image sensor 212 of the imaging processing unit 210 to store the accumulation time corresponding to the above-described exposure control value. The image data corresponding to the output signal obtained by setting is read in the full screen mode corresponding to all pixels for one frame (step 303 in FIG. 2), and passed to the image quality adjustment unit 227. The image quality adjustment unit 227 applies the white balance control value obtained by the control value calculation unit 226 to the image data in the full screen mode described above, and the image processing unit 203 records the image data whose image quality has been adjusted. It is recorded on the medium 202 (step 304).

After the recording of one frame is completed in this way, if an operation indicating the end of recording is not accepted by the operation accepting unit 201 (No determination in step 305), it is based on the subsequent search of the reference area and the image of the reference area. Perform image quality adjustment processing.
The area tracking unit 223 illustrated in FIG. 1 receives the address of the reference area held in the reference data holding unit 225 described above via the adjustment control unit 224, and receives the address of the reference area in the previous frame and the automatic focus control unit 215. Based on the detection result by the angular velocity sensor 216, the position of the region that occupies the portion of the subject image captured in the reference region in the previous frame in the new frame is estimated, and the corresponding region is searched (step of FIG. 2). 306).

  Based on the estimation result regarding the new reference region obtained as described above, the readout processing unit 221 instructs the signal processing unit 213 of the imaging processing unit 210 to have a predetermined range including the estimated reference region. For the area of the image sensor 212 corresponding to (for example, an area of 30 pixels × 30 pixels), signal processing is performed on an output signal obtained with an accumulation time determined independently of the accumulation time of the draft mode for area search Is instructed. In response to this, image data of a predetermined region including the estimated reference region is obtained by the signal processing unit 213 described above, and this image data is read by the read processing unit 221 via the memory 214 and adjusted. It is used for the process of the control part 224 (step 307 of FIG. 2). It should be noted that the size of the area read by the reading processing unit 221 for searching for the reference area is determined in advance, for example, about 1/256 of all the pixels according to the total number of pixels of the image sensor 212. be able to.

  The adjustment control unit 224 shown in FIG. 1 collates information indicating the color and brightness obtained from the image data of the predetermined area including the reference area obtained as described above with the reference data, and within the predetermined range. If the two images match, it is determined that a part of the subject image captured in the matched region is the same as the target portion of the subject image captured in the reference region in the previous frame, and a new frame It is determined that the search of the reference region for the above has succeeded (positive determination in step 308). At this time, for example, the adjustment control unit 224 divides the image data of a predetermined area including the reference area into blocks each having about 10 pixels × 10 pixels, and obtains chromaticity coordinate values and luminance values for each of these blocks. Based on whether any of the difference between the chromaticity coordinate value and the luminance value and the chromaticity coordinate value and the luminance value included in the reference data held in the reference data holding unit 225 is equal to or less than a predetermined threshold value. Thus, it can be determined whether or not the search is successful.

  When the difference between all the chromaticity coordinate values and luminances of the respective blocks obtained as described above and the reference data is larger than the threshold value (in the case of negative determination in step 308), the adjustment control unit 224 It is determined whether or not the region tracking process relating to the frame has been tried a predetermined number of K times (for example, K = 3) (step 311), and if the determination is negative, the region tracking unit 223 is instructed to retry the region tracking process. Then, the processing of steps 306 and 307 described above is performed.

  In this manner, when the reference region search is successful before the region tracking process is tried for a predetermined number K times for each frame (affirmative determination in step 308), the adjustment control unit 224 has the chromaticity closest to the reference data. The block from which the coordinate value and the luminance value are obtained is set as a new reference area, and the contents of the reference data holding unit 225 are updated using the address, the chromaticity coordinate value, and the luminance value (step 309), and a new value is obtained. The obtained reference data is passed to the control value calculation unit 226.

In response to this, a new control value is calculated by the control value calculation unit 226 (step 310), and is subjected to image quality adjustment processing in a new frame read out in the full screen mode, and an image in which image quality is appropriately adjusted. Data is recorded on the recording medium 202 by the recording processing unit 203 (steps 303 and 304).
As described above, in the imaging apparatus according to the present invention, the target portion of the image of the subject captured in the reference area in the previous frame within the range estimated from the detection results obtained by the automatic focus control unit 215 and the angular velocity sensor 216. As long as there is, the appropriate exposure correction value and white balance control value can be obtained while reflecting the change in color and luminance related to the corresponding portion, and can be applied to the entire image data of a new frame.

  As a result, for example, as shown in FIGS. 3A and 3B, a person who becomes a subject suddenly approaches the imaging apparatus, and the face of the subject occupies most of the screen. However, it is possible to reliably calculate and apply an exposure correction value and a white balance control value suitable for the color and brightness of the person's face. This is because in such a case, the portion corresponding to the reference area that has been tracked in the previous frame (in the example of FIG. 3, the cheek portion of the subject person) is also reliably captured in the new frame. This is because, as described above, it is possible to reliably extract the portion of the image that faithfully represents the color and luminance of the subject that the user is paying attention to.

  Further, by obtaining the image control value based on the image data included in the tracked reference area, it is possible to obtain an image control value that reflects the color and brightness of the same part of the subject that the user is paying attention to. Therefore, regardless of camera shake or flickering of lighting, the stability of the image control value is kept extremely high, and the followability when there is a large change in the color or size of the subject image captured by the imaging device. It can be realized.

  On the other hand, when the subject captured by the imaging device has completely changed, as in the case where the camera pans greatly, the reference region cannot be found by the above-described reference region search process (Yes in step 311). Judgment). In such a case, the adjustment control unit 224 illustrated in FIG. 1 determines that the subject captured in the image data of the new frame is not the same as the subject captured in each of the previous frames. The image quality adjustment process based on the draft image similar to that at the start is performed.

  At this time, the draft image read by the read processing unit 221 is transferred to the control value calculation unit 226 via the adjustment control unit 224, and the image control value is calculated based on the draft image (steps 312 and 313). ). If the image control value obtained in this way is appropriate (affirmative determination in step 314), the process proceeds to step 302, where a new reference area is determined and reference data is held, and a new one is stored. The image control value obtained in step (b) is applied to the image data read out in the full screen mode, and used for recording processing on the recording medium 202 (steps 303 and 304).

As described above, in the imaging apparatus according to the present invention, in response to the failure of the reference area search, the image quality adjustment processing based on the draft image obtained in the draft mode is automatically performed in the same manner as at the start of recording, thereby enabling a large scene. It is possible to respond quickly to changes.
Of course, it is also possible to automatically perform image quality adjustment processing based on the draft image when the operation accepting unit 201 accepts operations such as power-on, recording start, stop, and restart. Further, for example, when the angular velocity sensor 216 notifies that the angular velocity exceeding a predetermined threshold is detected, the area tracking unit 223 causes the scene to be detected by the user's conscious camera operation (that is, camera pan). It is also possible to determine that there has been a significant change, stop the tracking process, and immediately start the image quality adjustment process based on the draft image.

  In the procedure of step 307 shown in FIG. 2, the signal processing unit 213 is completely independent of the accumulation time in the full screen mode or the draft mode according to the instruction from the reading processing unit 221. Signal processing relating to the output signal of the image sensor 212 obtained with the storage time for area search is performed, and the obtained image data is provided to the processing of the image quality adjustment processing unit 220 via the memory 214. Therefore, for example, by obtaining image data corresponding to the output signal of the image sensor 212 related to the above-described reference region in an accumulation time shorter than that of the full screen mode or the draft mode, it is highly possible that the image data is not saturated. Can be used for the calculation process of the exposure control value.

As a result, for example, as shown in FIG. 3B, even when the subject's brightness suddenly increases because the subject's person has suddenly approached the imaging device, it can be obtained corresponding to the reference region. By obtaining the exposure control value based on the obtained image data, it is possible to adjust the exposure time in anticipation of an increase in luminance of the subject.
In a shooting environment where sufficient brightness can be expected, such as outdoors in the daytime, the accumulation time required to acquire image data in the full screen mode recorded on the recording medium 202 is short. Prior to the acquisition of the image data, it is sufficiently possible to perform limited exposure for area search and acquisition of partial image data. Furthermore, if the storage time for area search is set to an appropriate value, it is possible to search the reference area by repeating the processes of steps 306 to 308 and step 311 of FIG. 2 a plurality of times.

  On the other hand, in an environment where sufficient brightness cannot be expected, for example, when a reference area search in a certain frame fails, the image control value of the previous frame is temporarily applied to record the frame, and a predetermined number When the search fails continuously over the frames, it is possible to proceed to step 312 shown in FIG. 2 and reset the image control value.

  According to the image pickup apparatus according to the present invention, in the control related to the image quality adjustment value at the time of moving image shooting, it is possible to achieve both stability that is not influenced by fluctuations related to the image of the subject and followability to changes in the subject. The present invention is extremely useful in the field of moving image photographing devices that are intended for various users.

1 is a diagram illustrating a first embodiment of an imaging apparatus according to the present invention. It is a flowchart showing image quality adjustment operation. It is a figure explaining a reference area tracking process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 201 ... Operation reception part, 202 ... Recording medium, 203 ... Recording processing part, 210 ... Imaging processing part, 211 ... Optical system, 212 ... Imaging device, 213 ... Signal processing part, 214 ... Memory, 215 ... Automatic focus control part, 216 ... Angular velocity sensor, 220 ... Image quality adjustment processing unit, 221 ... Reading processing unit, 222 ... Reference region determination unit, 223 ... Region tracking unit, 224 ... Adjustment control unit, 225 ... Reference data holding unit, 226 ... Control value calculation unit 227 ... Adjustment processing unit

Claims (4)

In an imaging apparatus that performs moving image shooting by sequentially recording image data of each frame obtained by photoelectrically converting a subject image formed on an imaging element by a shooting lens,
An area setting means for setting an arbitrary reference area in an imaging range captured by the imaging element based on characteristics of the image data of a predetermined frame;
Tracking means for tracking the subject image captured in the reference area set for the predetermined frame or a part thereof for each subsequent frame and setting the reference area for each frame;
An image pickup apparatus comprising: a control value calculation unit that calculates a control value related to at least one of an exposure time and a white balance in a corresponding frame based on a pixel value of each pixel included in the reference region. . The imaging device according to claim 1,
The control value calculation means reads out the pixel value of each pixel included in the reference area at an accumulation time determined independently of recording of image data for moving image shooting, and provides the calculation value to obtain the control value An image pickup apparatus comprising a reading unit. The imaging device according to claim 1,
The tracking means is
Holding means for holding information indicating an image of a subject captured in the set reference area or a characteristic of a part thereof;
Collecting means for collecting the imaging conditions including the angular velocity and the focal length obtained by the angular velocity sensor provided in the photographing lens, and the photographing distance for a new frame;
Estimating means for estimating a range in which a reference area may exist in a new frame based on the imaging conditions collected by the collecting means;
An image pickup apparatus comprising: extraction means for extracting a reference region in the new frame from a range indicated by an estimation result by the estimation means based on feature information held in the holding means. The imaging device according to claim 3.
The tracking means includes
Based on the shooting conditions collected by the collecting means and information indicating operations related to shooting including power-on and recording start of the imaging device, the characteristics of the subject in the previous frame and the characteristics of the subject in the new frame Identity determination means for determining identity;
An imaging apparatus comprising: setting instruction means for instructing the area setting means to set a new reference area when the identity regarding the feature of the subject is denied by the identity determination means.

Images