JP7254555B2 - IMAGING DEVICE AND METHOD OF CONTROLLING IMAGING DEVICE - Google Patents

Description

The present invention relates to an imaging device and an imaging device control method.

Panning, which is one of camera shooting methods, is a method of shooting a moving subject while following it with the camera. According to panning, it is possible to take a photograph with a sense of dynamism by letting the subject stand still while the background is flowing. However, in panning, it is difficult to match the moving speed of the subject with the speed at which the camera is shaken (hereinafter also referred to as panning speed). Therefore, when a user who is unfamiliar with panning takes a picture, there is a high possibility that the subject will not stand still and a blur residue will occur.

In recent years, an imaging apparatus having a panning assist function has been proposed so that even a beginner can easily perform this panning. For example, in Japanese Unexamined Patent Application Publication No. 2002-100003, blurring of an object during exposure is corrected based on the relative angular velocity of the object with respect to the imaging device before exposure and the angular velocity of the imaging device obtained from an angular velocity sensor before exposure. The relative angular velocity of the subject with respect to the imaging device is calculated from the amount of movement of the subject on the image plane detected from the image and the angular velocity sensor. Further, Japanese Patent Laid-Open No. 2002-200000 discloses calculating a subject portion in which the motion vector does not change based on the motion vector detection and the panning speed of the imaging screen during panning.

On the other hand, during panning, the moving subject is tracked by the camera, so there is a tendency for the luminance fluctuation in the background portion to increase. For this reason, during panning, it is preferable to perform photometric calculations that emphasize the brightness of the subject compared to normal still image shooting.

JP 2016-171541 A

However, for example, when shooting a low-contrast subject, or due to changes in the brightness of the imaging screen, it may not be possible to acquire the motion vector of the subject included in the imaging screen. In this case, the subject area cannot be calculated, making it difficult to calculate a photometric value suitable for panning.

SUMMARY OF THE INVENTION The present invention has been made in view of the above situation, and provides an image pickup apparatus capable of calculating a photometric value suitable for a subject even when the subject area cannot be detected during panning.

An image capturing apparatus, which is an example of the present invention, includes detection means for detecting movement of the image capturing apparatus, area detection means for detecting a subject area included in a captured image, movement of the image capturing apparatus, and an area prediction means for estimating the position of the subject area from movement of the subject area between images; and a photometry means for performing photometry calculation of the subject, wherein the photometry means detects the subject area from the images. calculating a first photometric value based on the subject area detected by the area detecting means when the subject area can be detected, and calculating the subject area predicted by the area predicting means when the subject area cannot be detected from the image; A second photometric value is calculated, and a third photometric value is calculated by weighted addition of the previously calculated first photometric value and the second photometric value.

According to the imaging apparatus that is an example of the present invention, even if the subject area cannot be detected during panning, a photometric value suitable for the subject can be calculated.

It is a block diagram which shows the structural example of an imaging device. It is a figure which shows the apparatus structural example of an imaging device. 7 is a flowchart illustrating an example of photometry calculation during operation of the panning assist function; FIG. 10 is a diagram for explaining calculation of an object angular velocity; FIG. 11 is a flowchart for explaining a method of predicting a subject area in step S309; FIG. FIG. 4 is a diagram showing an example of angular velocity and angular acceleration of a subject; FIG. 11 is a flowchart illustrating another example of photometry calculation during operation of the panning assist function; FIG. FIG. 12 is a flowchart for explaining an example of weighted average processing with past photometric values in step S<b>717 ; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<First embodiment>
FIG. 1 is a block diagram showing a configuration example of an imaging device 100 according to the first embodiment.
The imaging apparatus 100 of this embodiment has a panning assist function, which is a shooting assisting function suitable for panning. Note that panning in the following description is premised on panning in which the imaging device is moved in the horizontal direction, but the present invention can be similarly applied to tilting in which the imaging device is moved in the vertical direction.

Here, the imaging device 100 may be any electronic device having a camera function, such as a camera such as a digital camera or a digital video camera, a camera-equipped mobile phone, or a camera-equipped computer. Note that the configuration of the imaging apparatus shown in the embodiment is merely an example, and not all combinations of features described in the embodiment are essential to the present invention.

The imaging apparatus 100 includes an optical system 101 , an imaging element 102 , a focus detection circuit 103 , a CPU (Central Processing Unit) 104 , a primary storage device 105 , an angular velocity sensor 106 and an image processing section 107 . The imaging apparatus 100 also includes an interface for the recording medium 108 , a secondary storage device 109 , a display unit 110 , an operation unit 111 , a motion vector detection unit 112 and an object angular acceleration calculation unit 113 .

An optical system 101 includes a lens, a shutter, and an aperture, and forms an image of light from a subject on an imaging device 102 . The lens, shutter, and diaphragm of the optical system 101 are driven under the control of the CPU 104 . Also, the lenses of the optical system 101 include a focus lens, a zoom lens, a shift lens, and the like. The shift lens described above is an example of correction means for optically correcting image blur.

The imaging element 102 is a device that converts light that forms an image through the optical system 101 into an image signal. Examples of the imaging element 102 include a CCD (charge-coupled device) type or a CMOS (complementary metal oxide semiconductor) type image sensor.
The focus detection circuit 103 executes focus detection processing (Auto Focus) and outputs focal length information to the CPU 104 .

The CPU 104 is a control device that controls each unit that configures the imaging device 100 in accordance with input signals and pre-stored programs. Various functions of the imaging apparatus 100 are realized under the control of the CPU 104 .

The primary storage device 105 is, for example, a volatile storage device such as a RAM (Random Access Memory). The primary storage device 105 stores temporary data and is used for work by the CPU 104 . Information stored in the primary storage device 105 is used in the image processing unit 107 or recorded in the recording medium 108 .

The angular velocity sensor 106 detects angular velocity representing the amount of movement of the imaging device 100 , converts the detected angular velocity into an electrical signal, and transmits the electrical signal to the CPU 104 . An example of the angular velocity sensor 106 is a gyro sensor. For example, as will be described later, when the imaging device is a lens-interchangeable camera, the angular velocity sensor 106 may be a component on the lens side, or may be a component on the main body side of the imaging device.

The image processing unit 107 is a processor for image processing such as image processing called development processing and color tone adjustment according to the shooting mode. At least part of the functions of the image processing unit 107 may be implemented by the CPU 104 in software.
The image processing unit 107 also has a plurality of image processing patterns applied to the captured image. The image processing patterns are associated with different shooting modes, respectively, and the image processing unit 107 changes the image processing patterns according to the set shooting mode.

The recording medium 108 is a non-volatile storage medium that is detachable from the imaging device 100 and from which data can be read and written. An example of the recording medium 108 is a semiconductor memory card. The recording medium 108 records, for example, image data obtained by shooting, which is stored in the primary storage device 105 . Data recorded on the recording medium 108 can be read by a personal computer or the like having an interface with the recording medium 108 .

The secondary storage device 109 is a non-volatile storage device such as EEPROM (Electrically Erasable Programmable Read Only Memory). The secondary storage device 109 stores a program (firmware) for controlling the imaging device 100 and various setting information. Information in the secondary storage device 109 is used by the CPU 104 .

The display unit 110 displays a viewfinder image at the time of shooting, a shot image, a GUI image for interactive operation, and the like.
An operation unit 111 is a group of input devices that receive user (photographer) operations and transmit input information to the CPU 104 . The operation unit 111 may include, for example, a release switch having SW1 and SW2, a button, a lever, and a touch panel, as well as an input device using voice, line of sight, and the like. Also, the shooting mode is set by operating the operation unit 111 .

A motion vector detection unit 112 detects a motion vector using an image obtained by shooting.
The subject angular acceleration calculator 113 reads out the history of the subject angular velocity accumulated in the primary storage device 105, and calculates the amount of change in the subject angular acceleration that is continuous in the time axis direction. Here, the subject angular velocity is the angular velocity of the subject with respect to the imaging device 100 .

FIG. 2 is a diagram showing a specific device configuration example of the imaging device. In FIG. 2, as an example of an imaging device, a configuration of an interchangeable lens type camera in which a lens unit 400 can be exchanged with respect to a camera body 200 will be described.

First, the configuration of the camera body 200 shown in FIG. 2 will be described.
A camera body 200 includes a camera microcomputer 201 , a memory 202 , an imaging element 203 , a shutter 204 , a half mirror 205 , a focusing screen 206 , a display element 207 , a photometric sensor 208 and a pentaprism 209 . The camera body 200 also includes a focus detection circuit 210 , an AF mirror 211 , an APU 212 and a memory 213 .

A camera microcomputer 201 is a processor that controls each element of the camera body 200 . In the example of FIG. 2, the camera microcomputer 201 has the function of calculating the subject angular velocity during exposure.
A memory 202 is a storage device such as a RAM or ROM connected to the camera microcomputer 201 .

An image pickup device 203 is an element corresponding to the image pickup device 102 in FIG. 1, and is an image sensor that photoelectrically converts light from an object during shooting. The imaging device 203 has an infrared cut filter, a low-pass filter, and the like on its imaging surface.
The shutter 204 shields the imaging surface of the imaging element 203 from light during non-shooting, and opens to guide light to the imaging surface of the imaging element 203 during imaging.

The half mirror 205 is driven to be in a mirror-down state when the imaging device 203 is not performing imaging, and to be in a mirror-up state when imaging by the imaging device 203 to be retracted from the optical path. FIG. 2 shows the arrangement of the half mirror 205 in the mirror-down state.
The half mirror 205 reflects part of the light incident from the lens unit 400 in the mirror-down state, guides it toward the optical viewfinder, and transmits the remaining part of the light. The light reflected by the half mirror 205 forms an image of the subject on the focusing screen 206 .

A display element 207 displays a position of focus detection (AF) to the user looking through the optical viewfinder. The display element 207 is composed of, for example, a PN liquid crystal or the like, and displays a subject image on the focusing screen 206 by superimposing a point for focus detection (AF range-finding point).

A photometry sensor 208 is a sensor incorporating an image sensor, and performs photometry of a subject based on light reflected by the half mirror 205 in the mirror-down state. A pentaprism 209 guides the subject image on the focusing screen 206 to the photometric sensor 208 and optical viewfinder. As shown in FIG. 2, the photometric sensor 208 sees the subject image formed on the focusing screen 206 via the pentaprism 209 from an oblique position.

The AF mirror 211 guides part of the light incident from the lens unit 400 and transmitted through the half mirror 205 in the mirror-down state to the AF sensor in the focus detection circuit 210 . A focus detection circuit 210 corresponds to the focus detection circuit 103 in FIG. The focus detection circuit 210 performs focus detection (distance measurement) based on light incident through the AF mirror 211 . Note that the AF mirror 211 is retracted from the optical path together with the half mirror 205 in the mirror-up state.

The APU 212 is a processor that performs signal processing using the output of the photometric sensor 208 . The APU 212 has, for example, the functions of the image processing unit 107 and the motion vector detection unit 112 shown in FIG. A memory 213 is a storage device such as a RAM or ROM connected to the APU 212 . In the example of FIG. 2, the configuration in which the APU 212 is provided as a dedicated processor for the photometric sensor 208 has been described, but the camera microcomputer 201 may execute the processing performed by the APU 212 .

Next, the configuration of the lens unit 400 shown in FIG. 2 will be described.
The lens unit 400 has an LPU 401 , an angular velocity sensor 402 and a lens 403 .

The LPU 401 is a processor for controlling each section within the lens unit 400 . For example, the LPU 401 transmits information about the distance to the subject, angular velocity information (to be described later), and the like to the camera microcomputer 201 via electrical contacts (not shown).

The angular velocity sensor 402 detects angular velocity representing the amount of movement of the imaging device, converts the angular velocity information into an electrical signal, and transmits the electrical signal to the LPU 401 . The angular velocity sensor 402 is composed of, for example, a gyro sensor. This angular velocity sensor 402 is an element corresponding to the angular velocity sensor 106 in FIG. 1, and is an example of detection means.

Here, the LPU 401 calculates the driving amount of the shift lens included in the lens 403 based on the subject angular velocity obtained by communication with the camera microcomputer 201 . Then, the LPU 401 drives the shift lens based on this drive amount, thereby correcting image blurring of the subject (subject blurring). As a result, when the panning assist function is in operation, even if the panning action of the photographer deviates from the movement of the subject, it is easy to realize shooting with the background flowing while suppressing subject blur.

FIG. 3 is a flow chart for explaining an example of photometry calculation during operation of the panning assist function.
When the imaging apparatus is activated with the panning assist function turned on, the processing of the flowchart shown in FIG. 3 is started. Note that in the following flowchart, the photometric calculation may be performed using the output of the photometric sensor 208 in the mirror down state, or the photometric calculation may be performed using the output of the image sensor 203 in the mirror up state.

In step S301, the camera microcomputer 201 determines whether a half-press operation of the release switch by the photographer has been received (whether SW1 has been turned ON). If SW1 is ON, the process proceeds to step S302, and photometry calculation for panning is started. On the other hand, if SW1 is not turned ON, the camera microcomputer 201 waits until SW1 is turned ON.

In step S<b>302 , the camera microcomputer 201 acquires information on the panning angular velocity of the imaging device from the angular velocity detected by the angular velocity sensor 402 .

In step S303, the camera microcomputer 201 detects a subject area (subject area) from an image acquired for photometry calculation. As described above, an image acquired for photometric calculation is acquired by the image sensor 203 when shooting with the mirror up, and is acquired by the photometric sensor 208 when shooting with the mirror down.

For example, in the mirror-down state, the motion vector detector in the APU 212 detects motion vectors from the image obtained by the photometric sensor 208 . Also, in the mirror-up state, a motion vector detection unit in the camera microcomputer 201 detects a motion vector from the image obtained by the imaging device 203 . Then, the camera microcomputer 201 detects the subject area using the motion vector detected by the motion vector detection unit and the panning angular velocity information (S302). That is, the camera microcomputer 201 is an example of area detection means.
Various methods such as template matching have been proposed for determining the area of the subject. Therefore, in step S303, these well-known methods may be appropriately used to determine the subject area.

In step S304, the camera microcomputer 201 determines whether the subject area has been detected. If the subject area is detected, the process proceeds to step S305. On the other hand, if the subject area cannot be detected, the process proceeds to step S309.

When the subject area is detected in step S304, the camera microcomputer 201 acquires the subject angular velocity on the imaging plane in step S305.
Then, in step S306, the camera microcomputer 201 calculates the subject angular velocity.

Here, the processing of steps S305 and S306 is performed as follows.
FIG. 4 is a diagram illustrating calculation of the subject angular velocity in this embodiment.
For example, consider the case where the object moves from point A to point B in t seconds. In this case, the image of the subject formed on the sensor of the imaging device moves from point C to point D in accordance with the movement of the subject. When the distance between points C and D (the amount of movement on the imaging plane) is d [pixel], the focal length is f [mm], and the pixel pitch of the sensor is p [μm/pixel], the object on the imaging plane is The angular velocity ω [rad/sec] can be expressed by Equation 1 below.

When the photographer pans the imaging device, the angular velocity ω of the subject on the imaging plane is obtained by subtracting the panning angular velocity ω _p from the angular velocity of the subject itself (subject angular velocity) ω _S . Therefore, the angular velocity ω of the subject on the imaging plane can be expressed by Equation 2 below.
ω = ω _S - ω _p Formula 2

Therefore, the camera microcomputer 201 can calculate the object angular velocity ω _S using the following equation 3 using the panning angular velocity ω _p of the imaging device detected by the angular velocity sensor 402 and the object angular velocity ω on the imaging plane.
ω _S = ω + ω _p Equation 3

Returning to the description of FIG. 3, in step S307, the camera microcomputer 201 converts the information of the subject area obtained in step S303 (for example, information about the shape, size, and position of the subject area) to the point at which the information is generated. Save in the subject area history in a state linked with the information.
As an example, when the camera microcomputer 201 obtains the subject area from the image captured by the image sensor 203, it stores the subject area history in the memory 202, and when the subject area is obtained from the image obtained by the photometric sensor 208, , the object area history is stored in the memory 213 . Note that the storage destination of the subject region history is only an example, and can be changed as appropriate.

In step S<b>308 , the camera microcomputer 201 saves the subject angular velocity information calculated in step S<b>306 in the subject angular velocity history in a state of being associated with the information indicating the generation point of the information. After the process of step S308 ends, the process proceeds to step S310. Note that the storage destination of the subject angular velocity history is the same as the storage destination of the subject region history described above, so description thereof will be omitted.

Step S309 is processing when the subject area cannot be detected in step S304. The camera microcomputer 201 cannot detect the subject area when the subject is not detected from the motion vector and panning angular velocity information. In this case, the camera microcomputer 201 predicts the amount of movement of the subject area from past detection results. Details of the processing in step S309 will be described later with reference to FIG. After the process of step S309 ends, the process proceeds to step S310.

In step S310, the camera microcomputer 201 uses the image acquired by the image sensor 203 or the photometric sensor 208 to calculate a photometric value based on the subject area. The subject area for which the photometric value is calculated in step S310 is either the subject area detected in step S303 or the subject area predicted in the process of step S309. Since the method of calculating the photometric value in the subject area is known per se, detailed description thereof will be omitted.

In step S311, the camera microcomputer 201 determines whether the full-press operation of the release switch by the photographer has been accepted (whether SW2 has been turned ON). When SW2 is ON, the camera microcomputer 201 starts shooting operation using the photometric value calculated in step S309.

On the other hand, if SW2 is not ON, the process returns to step S301. If the release switch is half-pressed again before the release switch is fully-pressed, the camera microcomputer 201 executes the processes from step S302 onward again to calculate the photometric value based on the subject area. This completes the description of FIG.

Next, the subject area prediction method performed in step S309 will be described with reference to FIG.
In the following example, a case will be described in which the object angular velocity is calculated with an object moving away in uniform linear motion as the object. Note that if the subject angular velocity can be predicted from the history of the subject angular velocity, then the subject area can be predicted by applying this method, and the movement of the subject is not limited to the above examples.

FIG. 6 is a diagram showing an example of angular velocity and angular acceleration of a subject.
Even if the subject is in uniform linear motion, the subject will accelerate when calculated as the angular velocity seen from the imaging device. Therefore, the subject angular velocity ω _last acquired at the time of the image (frame) in which the subject area was last detected and the subject angular velocity ω _t at the present time have different values.
Therefore, the camera microcomputer 201 in the process of FIG. 5 uses the subject angular velocity history saved in step S305 to predict the amount of movement of the subject area on the imaging plane from the last frame in which the subject area was detected.

In step S<b>501 , the camera microcomputer 201 reads subject angular velocity history information stored in the memory 202 or memory 213 .

In step S502, the camera microcomputer 201 calculates subject angular acceleration, which is the amount of change in the subject angular velocity. Assuming that the subject angular velocity at a certain point in time is _ωn , the subject angular acceleration at a certain point in time _ωan can be calculated by the following equation 4.
ω _an =ω _n -ω _n-1 Formula 4

In step S503, the camera microcomputer 201 predicts the current subject angular acceleration based on the subject angular acceleration obtained in step S502.
As a method of predicting the subject angular acceleration, any method such as a method of linearly approximating from an arbitrary history, a method of registering a predetermined generalized approximation formula in advance in the memory 202 or memory 213 and reading it out, etc., is adopted. can do. In this embodiment, a method of approximately predicting the object angular acceleration by the least squares method at N points is adopted. Assuming that the latest angular acceleration of the object is ω _a and the time is t, the prediction approximation formula is given by Equation 5 below.

From the above approximation formula, the angle θ _s [rad] that has changed from the time t of the last frame in which the subject was detected to the current time t _a can be calculated using Equation 6 below.

In step S504, the camera microcomputer 201 uses the angle and panning amount calculated in step S503 to calculate the amount of movement of the subject on the imaging plane.
Let θ _S be the angle (angular variation amount) by which the subject has changed so far, and let θ _p be the angular variation amount due to panning from the last frame in which the subject was detected to the current frame. At this time, the amount of vector change θ [rad] on the imaging plane can be obtained using Equation 7 below.
θ = θ _s - θ _p Formula 7

Also, if the amount of vector change on the imaging plane is θ [rad], the focal length is f [mm], and the pixel pitch of the sensor is p [μm/pixel], then the amount of movement d [pixel] on the imaging plane is as follows: can be expressed by the following equation 8.

In step S<b>505 , the camera microcomputer 201 reads subject area history information stored in the memory 202 or memory 213 .
In step S506, the camera microcomputer 201 obtains the position of the subject area moved by the amount of movement on the imaging plane calculated in step S504 with reference to the position of the subject area of the frame in which the subject was last detected. Accordingly, the camera microcomputer 201 predicts the current position of the subject area from the amount of movement on the imaging plane. With this, the processing in FIG. 5 is completed.

As described above, when the subject area cannot be detected, the camera microcomputer 201 calculates the amount of movement of the subject on the imaging plane from the amount of angular variation of the subject and the amount of angular variation due to panning (S504). Then, the camera microcomputer 201 predicts the current position of the subject area using the position of the subject area in the last frame in which the subject was detected and the amount of movement on the imaging plane (S506). As a result, in this embodiment, even if detection of the subject area fails in panning, it is possible to calculate a photometric value suitable for the subject.

Here, although one type of subject angular velocity calculation method is applied in the above example, plural types of subject angular velocity calculation methods may be switched as appropriate depending on the situation. Also, the approximation formula applied in the subject angular acceleration prediction method is not limited to the least squares method, and various approximation formulas can be applied.

<Second embodiment>
In the first embodiment, the subject area is predicted on the assumption that the subject to be photographed is moving at an assumed subject angular velocity. However, the movement of the subject rarely coincides perfectly with the assumption. If the subject region is predicted as in the first embodiment when the subject moves unexpectedly, the correct position of the subject region cannot be predicted, and a photometric value suitable for panning may not be calculated. For the above reasons, it is preferable to change the photometric value according to the accuracy (reliability) of prediction of the subject area.

FIG. 7 is a flowchart for explaining another example of photometry calculation during operation of the panning assist function as the second embodiment. Note that the configuration of the imaging apparatus of the second embodiment is the same as that of the first embodiment, so redundant description will be omitted.
In the second embodiment, instead of the control shown in FIG. 3, the following control shown in FIG. 7 is performed. In the control of FIG. 7, when the subject area has not been detected, the process branches depending on whether the subject area has been detected a predetermined number of times within a predetermined period. Further, in the control of FIG. 7, when predicting the subject area, the weighting of the photometric values based on the subject area is changed based on the reliability of the prediction of the subject area.

The processing from steps S701 to S703 in FIG. 7 respectively correspond to the processing from steps S301 to S303 in FIG. Therefore, redundant description of these processes will be omitted in the description of FIG.
In step S704, the camera microcomputer 201 determines whether the subject area has been detected in the latest frame. If the subject area is detected, the process proceeds to step S705. On the other hand, if the subject area cannot be detected, the process proceeds to step S711.

The processing from steps S705 to S708 in FIG. 7 respectively correspond to the processing from steps S305 to S308 in FIG. Therefore, redundant description of these processes will be omitted in the description of FIG. After a series of processes from steps S705 to S708 are completed, the process proceeds to step S709.

In step S709, the camera microcomputer 201 sets the weighting of the photometric values of the subject area. Note that if the weighted average of the photometric values is not performed in step S710, which will be described later, the processing in step S709 can be omitted.

For example, when the camera microcomputer 201 performs weighted averaging of the photometric values of the past frame and the current frame, the camera microcomputer 201 weights the photometric values of the subject region based on the average luminance difference between the frames and the time difference from the current frame. may be applied.

For example, the greater the average luminance difference between frames, the lower the weight of photometric values based on past frames is set. Also, for example, the greater the time difference from the current frame, the lower the weight of the photometric value based on the past frame is set.

In step S710, the camera microcomputer 201 uses the image acquired by the image sensor 203 or the photometric sensor 208 to calculate the photometric value of the subject area detected in step S703. Since the method of calculating the photometric value in the subject area is known per se, detailed description thereof will be omitted.
In step S710, the camera microcomputer 201 obtains a weighted average of the photometric values calculated in the past and the currently calculated photometric values based on the above-described weight setting (S709). may be calculated. For example, if the shooting mode is set to the continuous shooting mode, which emphasizes the stability of photometry, the weighted average addition of the photometry value calculated in the past and the photometry value calculated at the present time It is preferable to
When the process of step S710 ends, the process proceeds to step S719.

On the other hand, if the subject area cannot be detected in step S704, the process of step S711 is performed as described above. In step S711, the camera microcomputer 201 determines whether the subject area has been detected a predetermined number of times or more within a predetermined period.
If the subject area has been detected more than the predetermined number of times within the predetermined period, the process proceeds to step S712. On the other hand, if the number of times the subject area is detected within the predetermined period is less than the predetermined number of times, the process proceeds to step S718.

In step S711 of the present embodiment, for example, five frames past the current frame are set as a predetermined period, and it is determined whether or not the subject area has been detected twice or more during the five frames.
The predetermined period and the predetermined number of times in step S711 are parameters for excluding the case where the subject angular velocity cannot be calculated and the area prediction cannot be performed, and can be appropriately changed according to, for example, the subject area prediction method. Note that the predetermined period may be counted in units of frames, or may be counted in elapsed time.

In step S712, the camera microcomputer 201 predicts the amount of movement of the subject area from past detection results. Since the process of step S712 corresponds to the process of step S309 of FIG. 3 and the process of FIG. 5, redundant description will be omitted.

In step S713, the camera microcomputer 201 calculates the reliability of prediction of the subject area. For example, the camera microcomputer 201 uses the approximated curve f(x) calculated in step S503, the measured values (x _i , y _i ) used for prediction in the subject angular velocity history, and the average value y _ave of y _i . A coefficient is calculated and set as the reliability T of prediction of the object region. T is obtained by the following formula 9, and 0≤T≤1.

Note that, in this embodiment, the reliability of prediction of the subject region is calculated using the coefficient of determination obtained from Equation 9 above. However, the method of calculating the reliability may also be changed as appropriate, such as using a coefficient of determination adjusted for degrees of freedom or changing the reliability depending on the parameter of the data.

In step S714, the camera microcomputer 201 associates information linking the subject area detected in step S703 with the prediction reliability calculated in step S713 with information indicating the generation time point of the information. Save to the prediction reliability history with . The storage destination of the prediction reliability history is the same as, for example, the storage destination of the subject area history and the subject angular velocity history, and therefore the description thereof is omitted.

In step S715, the camera microcomputer 201 sets the weighting of the photometric values of the subject area. When weighted averaging the photometric values of the past frames and the current frame, the camera microcomputer 201 weights the photometric values of the subject region based on the average luminance difference between frames and the time difference from the current frame. you can go
For example, the greater the average luminance difference between frames, the lower the weight of the photometric value of the subject region based on the past frame may be set. That is, in S715, if the average luminance difference between frames is large, the weight of the photometric value based on the predicted subject area may be relatively increased.
Also, for example, the greater the time difference from the current frame, the lower the weight of the photometric value of the subject region based on the past frame may be set. That is, in S715, the weight of the photometric value based on the past frame may be decreased, and the weight of the photometric value based on the predicted subject area may be relatively increased.

In step S716, the camera microcomputer 201 uses the image acquired by the image sensor 203 or the photometric sensor 208 to calculate the photometric value of the subject area predicted in step S712. Since the method of calculating the photometric value is the same as that in step S710 described above, redundant description will be omitted.

In step S717, the camera microcomputer 201 performs weighted average addition of the photometric value obtained in step S716 and the photometric value calculated in the past (past photometric value) to calculate the final photometric value. For example, if the shooting mode is set to the continuous shooting mode, which emphasizes the stability of photometry, the weighted average of the photometry value calculated in the past and the photometry value calculated at the present time is performed. is preferred.
When the process of step S717 ends, the process moves to step S719.

FIG. 8 is a flowchart for explaining an example of weighted average processing with past photometric values in step S717.

In step S801, the camera microcomputer 201 reads out the photometric value history that stores information on past photometric values. Note that the photometric value history is stored in the memory 202 or memory 213 in the same manner as the subject region history, subject angular velocity history, and prediction reliability history.
In step S<b>802 , the camera microcomputer 201 reads the prediction reliability history from the memory 202 or memory 213 .

In step S803, the camera microcomputer 201 performs a weighted average of the past photometric value and the photometric value calculated in step S716. When the process of step S803 ends, the process returns to the process of FIG.

Here, a specific example of calculation when weighted averaging of photometric values is performed based on the reliability of prediction of the subject area in step S803 will be described. As described in step S715, the weighting of the photometric value acquired in step S716 and the past photometric value is further adjusted based on factors such as the average luminance difference between frames and the time difference from the current frame. may

Let i=0 be the last frame in which the object was detected, i=n be the latest frame, Bv _i be the i-th photometric value, and T _i be the reliability of i-th object prediction. At this time, the photometric value _{Bv_val} , which is the result of weighted averaging the photometric values of the latest frame from the photometric values of the frame in which the subject region was last detected, can be obtained by the following equation (10).

According to the processing in step S803, when the reliability of the subject area prediction is high, the weight of the photometric value of the predicted subject area is increased. On the other hand, when the reliability of prediction of the subject area is low, the weight of the photometric value of the last frame in which the subject area is detected is increased.
In other words, when there is a high possibility that the prediction of the subject area is incorrect, the weight of the photometric value of the last frame in which the subject area is detected is increased. You can restrain yourself from leaving.

Returning to FIG. 7, if it is determined in step S711 that the subject area has not been detected more than the predetermined number of times within the predetermined period, the process of step S718 is performed as described above.
In step S718, the camera microcomputer 201 calculates a photometric value for the current frame by evaluative photometry according to the photometry mode. Since the method of calculating the photometric value by evaluative photometry is well known, detailed description thereof will be omitted.
When the process of step S718 ends, the process moves to step S719.

Here, when the photometric value is calculated by evaluative metering according to the photometric mode without predicting the subject as in step S718, weighting with the past photometric value is performed in order to give priority to the stability with the previous photometric value. You can add by averaging. For example, the camera microcomputer 201 weights the photometric values based on the average brightness difference between the frames and the time difference from the current frame when weighted averaging the photometric values of the past frame and the current frame. may Since these processes have already been described in step S709, redundant description will be omitted.

In step S719, the camera microcomputer 201 stores the information of the photometric value calculated in any one of steps S710, S717, and S718 in the memory 202 or 213 in a state in which it is associated with the information indicating the generation time of the information. saved in the photometric value history. As a result, the photometric value calculated this time can be referred to as the past photometric value in the next and subsequent photometric calculations.

In step S720, the camera microcomputer 201 determines whether the full-press operation of the release switch by the photographer has been accepted (whether SW2 has been turned ON). When SW2 is ON, the camera microcomputer 201 starts shooting operation using the photometric value calculated in any one of steps S710, S717, and S718. On the other hand, if SW2 is not ON, the process returns to step S702. This completes the description of FIG.

In the second embodiment, the weighted average of the photometric value of the frame in which the subject region was last detected and the photometric value of the frame in which the subject region was predicted is calculated (S717). At this time, the photometric value of the frame in which the subject area is predicted is adjusted based on the reliability of the subject area prediction (S713). Therefore, even if prediction of the position of the subject fails, such as when the subject moves unexpectedly, it is possible to prevent the photometric value from deviating from the proper exposure.

<Conclusion>
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof.

201 camera microcomputer 203 image sensor 208 photometric sensor 402 angular velocity sensor

Claims (10)

detection means for detecting movement of the imaging device;
an area detection means for detecting a subject area included in the captured image;
an area prediction means for predicting the position of the subject area from the movement of the imaging device and the movement of the subject area between previously captured images;
and a photometric means for performing photometric calculation of a subject,
The photometric means
calculating a first photometric value based on the subject area detected by the area detecting means when the subject area can be detected from the image;
When the subject area cannot be detected from the image, a second photometric value is calculated based on the subject area predicted by the area prediction means , and the previously calculated first photometric value and the second photometric value are calculated. and calculating a third photometric value by weighted addition of the values . The area prediction means calculates a reliability of prediction of the subject area,
When the subject region cannot be detected from the image, the photometry means performs weighted addition of the previously calculated first photometry value and the second photometry value according to the reliability. 2. The imaging apparatus according to claim 1 , wherein a third photometric value is calculated. 3. The photometric means calculates the third photometric value by weighting the second photometric value more than the first photometric value when the reliability is high. 3. The imaging device according to 2 . When the time difference between the calculation of the first photometric value and the calculation of the second photometric value is large, the photometric means measures the second photometric value higher than the first photometric value. 4. The imaging apparatus according to any one of claims 1 to 3, wherein the third photometric value is calculated with a greater weight. The photometry means, when the luminance difference between the image used to calculate the first photometric value and the image used to calculate the second photometric value is large, 5. The imaging apparatus according to claim 1 , wherein the weight of the second photometric value is increased to calculate the third photometric value. The photometry means calculates the second photometry value even if the subject region cannot be detected from the image when the time difference between the time when the first photometry value is calculated and the current time exceeds a predetermined period. 6. The image pickup apparatus according to claim 1 , wherein the fourth photometry value is calculated by evaluative photometry without first calculating the fourth photometry value. 7. The method according to claim 6 , wherein said photometry means calculates said fourth photometry value by said evaluative photometry when the number of times said subject area is detected within said predetermined period is less than a predetermined number of times. Imaging device. 6. The photometry unit calculates the fourth photometry value by weighted addition of the photometry value calculated in the past and the photometry value calculated by the evaluative photometry. 8. The imaging device according to 7 . 9. The method according to any one of claims 1 to 8 , wherein, when the photographing mode is the continuous shooting mode, the photometry means performs weighted addition with previously calculated photometry values. Imaging device. a detection step of detecting motion of the imaging device;
an area detection step of detecting a subject area included in the captured image;
a region prediction step of predicting the position of the subject region from the motion of the imaging device and the motion of the subject region between previously captured images;
a photometry step of performing photometry calculation of the subject,
In the photometry step,
when the subject area can be detected from the image, calculating a first photometric value based on the subject area detected by the area detection step;
When the subject area cannot be detected from the image, a second photometric value is calculated based on the subject area predicted by the area prediction step , and the previously calculated first photometric value and the second photometric value are calculated. A method of controlling an imaging device , wherein a third photometric value is calculated by weighted addition of a value and a value .

Images