JP2012199803A - Imaging apparatus, control method of the same, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce correction remaining or over-correction when correcting distortion due to a rolling shutter problem.SOLUTION: When capturing images by using an imaging device driven by a rolling shutter system, vibrations of an imaging apparatus during an exposure period of the imaging device are detected, and a time characteristic of the detected vibrations is acquired. The time characteristic of the vibrations is separated into a plurality of bands for frequencies of the vibrations, a frequency characteristic of a strain amount of an object image generated in the image due to the vibrations is calculated for each band, and a band in which the frequency characteristic of the strain amount becomes maximum is determined as the band including a center frequency of the strain amount. Then, the time characteristic of the vibrations is compensated so as to eliminate a phase shift generated when the time characteristic of the vibrations is acquired at least for one frequency of the band including the center frequency. Further, a correction amount calculated from the compensated time characteristic of the vibrations is used to correct the strain of the object image generated due to the vibrations for each line of the imaging device.

Description

  The present invention relates to a technique for correcting image distortion caused by camera shake included in a captured image.

  In recent years, an imaging apparatus such as a digital video camera has come to use a CMOS sensor whose power consumption is lower than that of a CCD as an imaging element. The difference between the CCD and the CMOS sensor is not only the power consumption but also the exposure method at the time of photographing.

  In the CCD, when one captured image is captured, the exposure timing and the exposure period of all the pixels of the image sensor are the same, whereas in the CMOS sensor, the shutter opening / closing timing differs for each line constituting the image sensor. Differences occur in the exposure period. A driving method such as a CMOS sensor is called a rolling shutter method.

  When shooting is performed with such a rolling shutter type imaging device, the subject image moves for each line of the imaging device due to, for example, movement of the subject during exposure or camera shake of the photographer holding the imaging device during exposure. Therefore, distortion may occur in the captured image. The phenomenon in which the distortion occurs is called “rolling shutter problem” or “focal plane phenomenon”. In particular, distortion due to the rolling shutter problem can be more noticeable in moving image shooting that does not use a mechanical shutter than in still image shooting that uses a mechanical shutter.

  Among the distortions due to the rolling shutter problem as described above, a technique for correcting distortion caused by camera shake of a photographer is disclosed in Patent Document 1. In Patent Document 1, a time characteristic of camera shake is acquired by integrating a camera shake speed detected by an angular velocity sensor provided in the image pickup apparatus, and distortion is corrected according to a change in position of the image pickup apparatus at the time of shooting for each line of the image sensor. It is disclosed to correct.

JP 2006-186481 A

  However, while the sampling frequency of the angular velocity sensor is several kHz to several tens of kHz, the frequency of camera shake is about 0 Hz to 15 Hz, and the order of both frequencies is significantly different. In order to acquire the time characteristic of camera shake from the output signal of such an angular velocity sensor, when signal processing is performed using the FIR (finite impulse response) filter disclosed in Patent Document 1, a large number of taps are required. This will increase the circuit scale.

  In order to avoid an increase in circuit scale, it is conceivable to use an IIR (infinite impulse response) filter instead of an FIR filter. However, since an IIR filter is not a linear phase, a phase lag or phase lead that differs for each frequency band is used. Arise. That is, since the phase shift due to the phase characteristic of the IIR filter occurs in the time characteristic of the camera shake obtained by using the IIR filter in this way, a correction amount for correcting distortion due to the rolling shutter problem depending on the frequency of the camera shake. May not be calculated correctly. In other words, when the distortion due to the rolling shutter problem is corrected using the correction amount calculated from the time characteristic of camera shake obtained using the IIR filter, a phenomenon such as residual correction or overcorrection occurs in the image.

  The present invention has been made in view of the above-described problems, and an object thereof is to reduce residual correction or overcorrection when correcting distortion due to a rolling shutter problem.

In order to achieve the above object, one aspect of an imaging apparatus of the present invention has the following configuration.
An image pickup apparatus including an image pickup element driven by a rolling shutter method, and when an image is picked up using the image pickup element, detection means for detecting vibration of the image pickup apparatus during an exposure period of the image pickup element, and detection by the detection means The acquisition means for acquiring the time characteristics of the vibration, and the time characteristics of the vibration are separated into a plurality of bands with respect to the vibration frequency, and the frequency characteristics of the distortion amount of the subject image generated in the image due to the vibration are obtained for each band. Calculating means for calculating the distortion amount, determining means for determining a band having the maximum frequency characteristic of the distortion amount as a band including the center frequency of the distortion amount, among the frequency characteristics of the distortion amounts of the plurality of bands, and at least the center Compensation means for compensating the time characteristics of vibration so as to eliminate a phase shift caused by the obtaining means for one frequency in a band including the frequency, and compensation means Using the correction amount calculated from a more compensated vibration, characterized in that it comprises a correction means for correcting the distortion of a subject image caused due to the vibration for each line of the image sensor.

  With such a configuration, according to the present invention, it is possible to reduce the remaining correction or overcorrection when correcting distortion due to the rolling shutter problem.

1 is a block diagram showing a functional configuration of a digital video camera according to an embodiment of the present invention. The block diagram which showed the function structure of the distortion correction amount calculation part 106 which concerns on embodiment of this invention. The block diagram which showed the structure of the blur amount calculation part 201 which concerns on embodiment of this invention. The block diagram which showed the structure of the filter process part 203 which concerns on embodiment of this invention. The figure which illustrated the frequency distribution of the general vibration in the presumed imaging | photography situation, and the center frequency of the amount of distortion based on embodiment of this invention The figure which illustrated the weighting coefficient for every imaging | photography situation based on embodiment of this invention The figure which illustrated correspondence of the parameter of auxiliary information concerning the embodiment of the present invention, and the presumed photography situation The figure for demonstrating the method of determining the weighting coefficient which multiplies the filter response of each band based on embodiment of this invention. The figure which showed the weighted filter response obtained by multiplying the filter response of each band based on embodiment of this invention by the weighting coefficient. The figure for demonstrating the determination method of the center frequency which concerns on embodiment of this invention The block diagram which showed the structure of the phase compensation part 207 which concerns on embodiment of this invention. The figure for demonstrating the phase compensation which concerns on embodiment of this invention The figure for demonstrating the calculation method of the corrected amount of the captured image which concerns on embodiment of this invention. The figure which illustrated the reliability of the photography situation estimated about the parameter of auxiliary information concerning the embodiment of the present invention The figure for demonstrating the adjustment method of the weighting coefficient in consideration of the reliability of the imaging | photography condition which concerns on embodiment of this invention.

  Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the drawings. Note that in the embodiment described below, an example in which the present invention is applied to a digital video camera that can correct distortion caused by camera shake among distortions due to a rolling shutter problem as an example of an imaging apparatus will be described. To do. However, the present invention can be applied to any device capable of correcting distortion caused by camera shake among distortions due to the rolling shutter problem. Further, in this specification, among the “distortions caused by the rolling shutter problem”, distortions caused by camera shake are described as “rolling distortion” and will be described below.

(Functional configuration of digital video camera 100)
FIG. 1 is a block diagram showing a functional configuration of a digital video camera 100 according to an embodiment of the present invention.

  The control unit 101 is a CPU, for example, and controls the operation of each block included in the digital video camera 100. Specifically, for example, the control unit 101 reads out a processing program such as a photographing process stored in the ROM 102, develops it in a volatile memory (not shown), and executes it, thereby executing each block included in the digital video camera 100. Control the behavior. The ROM 102 is a rewritable nonvolatile memory, and stores parameters and the like necessary for the operation of each block in addition to the processing program described above.

  In the present embodiment, description will be made assuming that each process is realized in each block of the digital video camera 100 as hardware. However, the present invention is not limited to this, and the process of each block is the same as that of each block. It may be realized by a program that performs the process.

  The imaging unit 103 is a block including a rolling shutter type imaging element such as a CMOS sensor. The imaging unit 103 applies A / D conversion processing to an analog image signal obtained by photoelectrically converting an optical image formed on an imaging element by an optical system (not shown), and outputs a digital image signal (image data). Is output. The image data output from the imaging unit 103 is temporarily stored in the image memory 104, for example.

  The image processing unit 107 is a block that applies image processing such as gain adjustment and white balance adjustment to image data captured by the imaging unit 103 and stored in the image memory 104. The image processing unit 107 corrects rolling distortion generated in the image data stored in the image memory 104 for each line using the distortion correction amount calculated by the distortion correction amount calculating unit 106 described later.

  When moving image shooting is performed using the digital video camera 100, image data that has been subjected to various image processing or rolling distortion correction processing in the image processing unit 107 is encoded in the image processing unit 107 according to a predetermined encoding method. The encoded moving image data is transmitted to the recording medium 108 and recorded.

  The shake detection unit 105 is, for example, an angular velocity sensor, and is a sensor that detects an angular velocity caused by a shake (hand shake) applied to the digital video camera 100. The blur detection unit 105 outputs the detected three-axis angular velocities of the digital video camera 100 to, for example, the distortion correction amount calculation unit 106.

  The distortion correction amount calculation unit 106 is a block that calculates a correction amount for correcting rolling distortion generated in an image in the image data due to camera shake, and outputs information on the calculated correction amount to the image processing unit 107.

(Internal configuration of distortion correction amount calculation unit 106)
Here, the internal configuration of the distortion correction amount calculation unit 106 and the processing performed in the distortion correction amount calculation unit 106 will be described in detail below with reference to the drawings.

  FIG. 2 is a diagram illustrating an internal configuration of the distortion correction amount calculation unit 106 according to the present embodiment. The distortion correction amount calculation unit 106 includes an auxiliary information indicating the state of the digital video camera 100 at the time of capturing the image in addition to the information on the change in angular velocity generated during the capturing of the image of one frame detected by the blur detection unit 105. Information is input by the control unit 101. The auxiliary information is, for example, information such as a maximum acceleration (gyro data) of a gyro sensor (not shown), focal length, main subject information, shutter speed, or scene mode. In the present embodiment, the auxiliary information is an auxiliary information analysis unit. 209 is input.

  The blur amount calculation unit 201 temporally characterizes the temporal characteristic of the blur amount of the image in units of pixels on the image sensor due to the angular velocity change, based on the information of the angular velocity change that occurred during the imaging of the input image of one frame. .

  The shake amount calculation unit 201 has a configuration as shown in FIG. 3. First, the input angular velocity change information is signal-processed by the IIR filter 301 in accordance with the camera shake frequency. Further, the angular velocity change information output by the shake amount calculation unit 201 is output as hand movement angular displacement information θ by integration processing in the integrator 302. The camera shake angular displacement information θ is output with respect to the angle in the pitch direction and the yaw direction with respect to the imaging surface of the imaging device that may cause, for example, rolling distortion.

The blur amount conversion unit 303 converts the angular displacement information θ input from the integrator 302 into an image blur amount in units of pixels on the image sensor caused by the angular variation. Specifically, the blur amount conversion unit 303 converts the angular displacement information θ into the blur amount of the image on the image sensor using the following equation.
f _L · tan θ / p
Here, f _L is a focal length, and p is a pixel pitch on the image sensor.

  The time characteristic of the blur amount calculated in this way is output to the phase compensation unit 207 and the downsampling unit 202.

  In the shake amount calculation unit 201, a phase shift corresponding to the camera shake frequency by the IIR filter 301 and a phase delay of π / 2 due to the integration calculation of the integrator 302 occur. Among these, the phase delay due to the latter integral calculation is a fixed phase shift regardless of the frequency of camera shake, so that the influence can be eliminated by processing to advance the phase by π / 2. In the present embodiment, in order to reduce the phase shift according to the frequency of camera shake, the center frequency of camera shake is specified from the frequency distribution of camera shake that occurs during shooting of one frame. Then, in the block processing described later, the phase compensation parameter is calculated so that the phase shift is only π / 2 by integration calculation at least for the center frequency, and is output to the phase compensation unit 207.

  The down-sampling unit 202 reduces the sampling frequency of the input camera shake time characteristic (blur amount). For example, when reducing the sampling frequency of the time characteristic of camera shake from 12 kHz to 60 Hz, the down-sampling unit 202 may acquire one data per 200 data, and change the sampling frequency by so-called thinning.

  The filter processing unit 203 simply analyzes the frequency characteristics by applying a plurality of band limiting filters to the down-sampled camera shake time characteristics. The filter processing unit 203 is configured as shown in FIG. 4, and first applies three types of bandpass filters to the input time-stamped camera shake time characteristics to separate the camera shake time characteristics for each frequency band. To do.

  The time characteristics of the hand shake after down-sampling that has passed through each band limiting filter are integrated for the exposure period of one frame in the first response calculation unit 404, the second response calculation unit 405, and the third response calculation unit 406. That is, the output (filter response) from the response calculation unit corresponds to the result of integrating the frequency response of the shake amount for the camera shake time characteristic for each band, and the frequency characteristics for the three bands of camera shake time characteristics. Can be considered. In FIG. 4, the filter response output from the first response calculation unit 404 is shown as A, the filter response output from the second response calculation unit 405 is shown as B, and the filter response output from the third response calculation unit 406 is shown as C. ing.

  Note that if an IIR filter is used as the band limiting filter, a phase shift is further generated for each frequency of the time characteristic of the camera shake. ing. In other words, by setting the sampling frequency of the camera shake time characteristic sufficiently lower than the limit band, it is possible to limit the band while suppressing an increase in circuit scale even if an FIR filter is used. Note that the band of each band limiting filter may be determined by the filter processing unit 203 in consideration of the frequency component of the camera shake time characteristic.

  An object of the present invention is to reduce the remaining correction or overcorrection when correcting distortion due to the rolling shutter problem using an IIR filter, and the center frequency of vibration is dynamically determined to solve the problem. To do. The center frequency of vibration can be specified more accurately by subdividing the band in which the frequency of camera shake is distributed and providing a large number of band limiting filters without overlapping bands. However, since providing a large number of band limiting filters leads to an increase in circuit scale, in this embodiment, by limiting the number of band limiting filters, the center frequency of vibration is dynamically determined while suppressing the circuit scale. Is described below.

  When the number of band limiting filters that do not have overlapping bands as described above is limited, when the center frequency is near the threshold of the band between the two band limiting filters, for example, when shooting a movie, the center frequency is between frames. The band determined to be different may be different. In such a case, phase compensation is performed in accordance with different bands between frames, and there is a possibility that the viewer may feel uncomfortable in the finally obtained moving image.

Therefore, in the present embodiment, the FIR low-pass filter 401, the first FIR band-pass filter 402, and the second FIR band-pass filter 403, which are band limiting filters, are mainly assumed to have frequency components of 0 to 12 Hz with respect to the time characteristic of camera shake. Set as follows.
FIR low-pass filter 401: 6 Hz or less First FIR band-pass filter 402: 3-9 Hz
Second FIR bandpass filter 403: 6-12 Hz

  That is, in order to avoid a situation where different phase compensation is performed between frames, each band limiting filter is set to have overlapping bands. When the band limiting filters have overlapping bands, the vibrations of the overlapping bands do not appear in the filter response difference for each band. For this reason, in this embodiment, the weighting unit 204, which will be described later, weights differently for each band using the estimation result of the shooting state at the time of shooting so that the vibration of the overlapping band appears as a difference in filter response for each band. To. In addition, by performing weighting in consideration of the parameters of the shooting situation, it is possible to avoid the center frequency of vibration being classified into different bands between frames.

  The weighting unit 204 is a multiplier, for example. The weighting unit 204 multiplies the filter response of each band input from the filter processing unit 203 by a weight according to the estimated shooting situation, and uses the obtained weighted filter response as the center frequency. The data is output to the determination unit 205.

  Here, the weighting coefficient by which the weighting unit 204 multiplies the filter response of each band will be described. The weighting coefficient by which the filter response of each band is multiplied is determined by the auxiliary information analysis unit 209 and the weight determination unit 210 using the auxiliary information input to the distortion correction amount calculation unit 106.

  The auxiliary information analysis unit 209 estimates from the input auxiliary information whether the shooting state of the digital video camera 100 is either a two-handed shooting, a one-handed shooting, a walk shooting shooting, or an unknown shooting situation. In general, in the shooting situation of two-handed shooting, one-handed shooting, and walk-and-shoot shooting, the frequency distribution of camera shake that may occur in the digital video camera 100 differs as shown in FIG. . Specifically, in walk-and-shoot shooting, which is shooting while moving, the amount of blur due to camera shake becomes relatively large. On the other hand, two-handed shooting in which the photographer holds the digital video camera 100 with both hands without moving. Then, it is considered that the amount of shake due to camera shake is relatively small.

  For gyro data, for example, by detecting one low frequency of about 2 to 3 Hz, the amount of blur at the frequency is classified into “large”, “standard”, and “small” using a threshold value. . The auxiliary information analysis unit 209 estimates “walking shooting” when the amount of blur is large, and “two-handed shooting” when the amount of blur is small.

  Regarding the focal length, as the zoom position is closer to the telephoto, camera shake is more likely to appear as image distortion. Therefore, it is estimated that the photographer is shooting with the digital video camera 100 firmly fixed. That is, the auxiliary information analysis unit 209 estimates “two-handed shooting” when the focal length is long, and “walking shooting” when the focal length is short.

  As for the main subject information, it is considered that the main subject in the captured image when the zoom position is wide is small or the distance to the main subject is long, that is, the zoom position is telephoto. That is, the auxiliary information analysis unit 209 performs “two-handed shooting” when the size of the main subject is small or the distance to the main subject is long, and when the size of the main subject is large or the distance to the main subject is short Is estimated to be "walking shooting".

  Regarding the shutter speed, when the exposure time is long, the influence of the accumulated blur becomes large, and it is considered that the distortion of the captured image due to the low-frequency blur becomes inconspicuous. That is, when the shutter speed is slow, the auxiliary information analysis unit 209 estimates “two-handed shooting” in which the amount of blur of the low frequency component is small, and moves the phase compensation frequency to the high frequency side.

  As for the scene mode, when the portrait mode is set, the photographer is assumed to shoot with the digital video camera 100 firmly fixed with both hands, so the auxiliary information analysis unit 209 estimates “two-handed shooting”. . When the sports mode is set, the photographer is assumed to shoot the digital video camera 100 with one hand so that the moving subject can be easily followed. Therefore, the auxiliary information analysis unit 209 estimates “one-handed shooting”. To do. Furthermore, when the landscape mode is set, it is assumed that the photographer takes a picture while strolling around the mountain, so the auxiliary information analysis unit 209 estimates “walking photography”. For scene modes in which the shooting situation cannot be limited, the auxiliary information analysis unit 209 estimates the shooting situation as “unknown”.

  That is, the auxiliary information analysis unit 209 classifies each parameter as shown in FIG. 7 for each of the input auxiliary information using a predetermined state and threshold value, and sets the shooting status for each parameter of the auxiliary information. presume. By doing in this way, the weighting coefficient can be determined by the weight determination unit 210 so that a large weight is given to a frequency band in which the center frequency is considered to exist from the frequency distribution of the general blur amount.

  The weight determination unit 210 determines a weighting coefficient to be multiplied by the filter response of each band, which is the output of the filter processing unit 203, from the imaging state for each parameter of the auxiliary information estimated by the auxiliary information analysis unit 209. Considering the frequency distribution of the distortion amount in the captured image, the center frequency tends to be as shown in FIG. For this reason, a large weight is applied to a band in which the center frequency of the time characteristic of the actual camera shake exists based on the tendency, so that the center frequency is in an overlapping band of the band limiting filters. However, an appropriate band can be a band having a center frequency.

  The weight determination unit 210 first selects a weighting coefficient for each band for each parameter of the auxiliary information according to the shooting situation estimated for each parameter. For example, when the estimated shooting state is a walk-and-shoot shooting, the center frequency is considered to be in the low frequency region as shown in FIG. 5B, and therefore, the band is heavily weighted as shown in FIG. The coefficient is set so that the larger the band, the smaller the weighting.

  Then, the weight determination unit 210 combines the weighting coefficients of each band selected for each parameter of the auxiliary information, and the weighting unit 204 determines the final weighting coefficient of each band to be multiplied by the filter response of each band. .

  For example, consider a case where the final weighting coefficient of each band is determined using two parameters of auxiliary information. When the weighting coefficient of each band selected for one parameter is FIG. 8 (a) and the weighting coefficient of each band selected for the other parameter is FIG. 8 (b), The weighting coefficient is averaged for each band, as shown in FIG.

  The weighting unit 204 outputs a weighted filter response by multiplying the corresponding filter response by the weighting coefficient of each final band determined by the weight determining unit 210. For example, when the filter response of each band input to the weighting unit 204 is as shown in FIG. 9A, when the weighting coefficient as shown in FIG. The attached filter response is as shown in FIG.

  The center frequency determination unit 205 determines the center frequency of camera shake that occurs during imaging of one frame from the input weighted filter response of each band.

Among the time characteristics of camera shake, vibration of a single frequency f uses an amplitude A (f) that is the maximum amount of shake at the frequency,
A (f) · sin (ωt) (ω = 2πf)
It can be expressed as. That is, the vibration speed for each frequency is obtained by differentiating the time characteristic of camera shake by A (f) ω · cos (ωt)
It can be expressed as.

For example, when the difference in exposure start time between successive lines of the image sensor is Δt, the image displacement in the period Δt from the start of imaging of the previous line to the start of imaging of the next line As for the quantity, the speed at the time t _{0 at} the center of Δt as shown in FIG.
A (f) ω · cos (ωt ₀ ) · Δt
It can be expressed as. As described above, the filter response (FIG. 10B) of each band obtained from the filter processing unit 203 corresponds to the result of integrating the frequency response of the shake amount for the time characteristic of camera shake for each band. For this reason, when the blur amount term A (f) is separated from the _equation and approximated with ωt ₀ being constant, the remaining terms are ω · cos (ωt ₀ ) · Δt.
⇒ 2π · cos (ωt ₀ ) · Δt × f = c × f (c = const.)
And can be expressed by a linear function with respect to the frequency as shown in FIG.

  That is, the center frequency determination unit 205 multiplies the input weighted filter response of each band as shown in FIG. 10B by a displacement amount corresponding to the average frequency of each band from FIG. 10C. Thereby, the filter response of each band in consideration of the distortion amount in the captured image as shown in FIG. 10D can be obtained.

In the present embodiment, _(in the case of frequency (3～9Hz corresponding to the center of the frequency band 6 Hz)) filter response in consideration of the distortion quantity of the average frequency f _a of a band with a maximum in the captured image is determined as the center frequency . Then, a phase compensation parameter is set in the phase compensation controller 206 described later so that the phase delay due to the IIR filter 301 becomes π / 2 at the center frequency. By doing so, it is possible to reduce the distortion at a frequency (center frequency) at which the distortion amount is the largest, that is, when a phase shift occurs, the distortion remaining due to image correction or overcorrection is conspicuous.

  In the present embodiment, the band in which the filter response considering the distortion amount in the captured image is maximized is determined as the band including the center frequency, and the frequency corresponding to the center of the band is determined as the center frequency. However, the implementation of the present invention is not limited to this. That is, the frequency determined as the center frequency may be one of the bands including the center frequency or a frequency band, and may be dynamically selected according to the tendency of the filter in the other band. For example, in the case of FIG. 10D, one frequency in the intermediate frequency region may be determined as the center frequency according to the ratio of the filter response in the low frequency region and the high frequency region.

  The phase compensation control unit 206 calculates a phase compensation parameter such that the phase delay in the vicinity of the center frequency determined by the center frequency determination unit 205 is π / 2 in the phase compensation processing of the camera shake time characteristic in the phase compensation unit 207. And output to the phase compensation unit 207. The phase compensation parameter is determined by a known method so that the phase of the blur amount after phase compensation becomes −π / 2, which is the target phase, in the frequency band near the center frequency.

  The phase compensation unit 207 is a phase compensation filter. For example, the phase characteristics of the phase compensation filter are as shown in FIG. The phase compensator 207 compensates for the phase characteristics caused by the IIR filter 301 at the center frequency determined by the center frequency determination unit 205 with respect to the time characteristics of the input camera shake, and the phase delay at the frequency is caused by the integrator 302. Phase compensation is performed so that the generated π / 2 is obtained.

  The internal configuration of the phase compensation unit 207 is, for example, as shown in FIG. The adder 1101 adds the input camera shake time characteristic (blur amount) and the output of an amplifier 1103 described later, and outputs the blur amount after phase compensation to the captured image distortion amount calculation unit 208. Further, the shift register 1102 delays the input blur amount after phase compensation and outputs the delayed amount to the amplifier 1103. The amplifier 1103 multiplies the output from the shift register 1102 by the filter coefficient K, which is the phase compensation parameter input from the phase compensation control unit 206, and outputs the result to the adder 1101.

  With such a configuration, using the phase compensation parameter appropriately set by the phase compensation control unit 206, the phase characteristic of the blur amount that is the output of the phase compensation unit 207 is obtained at the center frequency as shown in FIG. −π / 2.

  The captured image distortion amount calculation unit 208 calculates the distortion amount of the captured image for each line of the image sensor, in which the phase shift in the blur amount calculation unit 201 is compensated, using the input time characteristic of hand shake after phase compensation. To do. The amount of distortion of the captured image between the lines of the image sensor is calculated from the difference in the amount of blur at the exposure start time as shown in FIG. 13, for example, when the amount of distortion generated between the Nth and N + 1th lines is taken as an example. be able to.

  Using the correction amount determined so as to eliminate the distortion amount calculated in this way, the image processing unit 107 can correct the distortion of the captured image due to the rolling shutter problem for each line of the image sensor.

(Summary)
As described above, the imaging apparatus according to the present embodiment can reduce the remaining correction or overcorrection when correcting distortion due to the rolling shutter problem. Specifically, when an imaging device captures an image using an imaging device driven by a rolling shutter method, the imaging device detects vibration of the imaging device during the exposure period of the imaging device, and obtains time characteristics of the detected vibration. To do. The time characteristic of vibration is separated into a plurality of bands with respect to the vibration frequency, the frequency characteristic of the distortion amount of the subject image generated in the image due to vibration is calculated for each band, and the frequency characteristic of the distortion amount is maximum. Is determined as a band including the center frequency of the distortion amount. Then, for at least one frequency in a band including the center frequency, the time characteristic of the vibration is compensated so as to eliminate the phase shift generated when the time characteristic of the vibration is acquired. Further, using the correction amount calculated from the time characteristics of the compensated vibration, the distortion of the subject image caused by the vibration is corrected for each line of the image sensor.

  By doing so, it is possible to compensate for the phase shift due to the use of the IIR filter for the frequency in the vicinity of the center frequency where the distortion amount in the captured image is the largest and the distortion after correction or distortion due to overcorrection is conspicuous. For this reason, when the distortion due to the rolling shutter problem is corrected, the distortion can be made inconspicuous in the entire captured image. Furthermore, since the center frequency is dynamically determined in accordance with the time characteristics of camera shake during imaging of one frame image, distortion due to the rolling shutter problem can be corrected appropriately.

(Modification)
In the embodiment described above, the weight determination unit 210 has been described as taking the average of the weighting coefficients of each band selected for each parameter of the auxiliary information and determining the final weighting coefficient to be multiplied by the filter response of each band. . In the present modification, a method of providing reliability for the imaging situation estimated for each parameter of the auxiliary information and determining a weighting coefficient for each band based on the reliability of each imaging situation will be described.

  For example, let us consider a case where the estimation reliability as shown in FIG. 14 is set for the shooting situation estimated for each parameter of the auxiliary information. The weight determination unit 210 first selects a weighting coefficient for each predetermined band for the shooting situation estimated for each parameter of the auxiliary information by the auxiliary information analysis unit 209 as in the above-described embodiment.

  Then, the weight determining unit 210 adjusts the weighting coefficient of each selected band according to the reliability of the parameter of the auxiliary information used for the selection. For example, let us consider a case in which the weighting coefficient of each band in FIG. 15A is selected from the shooting situation estimated for one parameter of the auxiliary information. The weight determination unit 210 selects the weighting coefficient of each selected band so that the value obtained by multiplying the difference between the two bands selected in descending order of the weighting coefficient by the reliability is the difference. The weighting coefficient of each band is adjusted based on the coefficient of the high low frequency region. Specifically, assuming that the reliability of the shooting situation is 0.8, the difference 0.3 of the coefficient between the low frequency region and the intermediate frequency region and the difference 0.2 of the coefficient between the intermediate frequency region and the high frequency region are respectively 0.24 and 0.16. That is, as shown in FIG. 15B, the coefficients of each band are 1.0, 0.76, and 0.6.

  Then, the weight determination unit 210 multiplies the filter response of each band by averaging the weighting coefficients of each band adjusted based on the reliability of the shooting situation for each parameter of the auxiliary information, and finally weights each band. Determine the coefficient.

  In this way, when estimating the shooting state according to the state of the digital video camera 100 at the time of shooting, which is auxiliary information, each band corresponding to the state at the time of shooting more flexibly by using the estimation reliability. Thus, the center frequency of the time characteristic of camera shake can be determined.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

In order to achieve the above object, one aspect of an imaging apparatus of the present invention has the following configuration.
An image pickup apparatus including an image pickup element driven by a rolling shutter method, and when an image is picked up using the image pickup element, detection means for detecting vibration of the image pickup apparatus during an exposure period of the image pickup element, and detection by the detection means The acquisition means for acquiring the time characteristics of the vibration, and the time characteristics of the vibration are separated into a plurality of bands with respect to the vibration frequency, and the frequency characteristics of the distortion amount of the subject image generated in the image due to the vibration are obtained for each band. Calculating means for calculating the distortion amount, determining means for determining a band having the maximum frequency characteristic of the distortion amount as a band including the center frequency of the distortion amount, among the frequency characteristics of the distortion amounts of the plurality of bands, and at least the center for one frequency band including a frequency such that the phase shift caused by the acquisition of the time characteristic Ru good to acquisition means is eliminated, to compensate for the time characteristics of the vibration Compensation means, and correction means for correcting distortion of the subject image caused by vibration for each line of the image sensor using a correction amount calculated from the vibration compensated by the compensation means. And

Claims (10)

An image pickup apparatus including an image pickup element driven by a rolling shutter method,
Detecting means for detecting vibrations of the imaging device during an exposure period of the imaging element when an image is captured using the imaging element;
Obtaining means for obtaining a time characteristic of the vibration detected by the detecting means;
A calculation unit that separates the time characteristics of the vibration into a plurality of bands with respect to the frequency of the vibrations, and calculates a frequency characteristic of a distortion amount of a subject image generated in the image due to the vibrations for each band;
Determining means for determining, among the frequency characteristics of the distortion amounts of the plurality of bands, a band in which the frequency characteristics of the distortion amounts are maximum as a band including a center frequency of the distortion amounts;
Compensation means for compensating the time characteristic of the vibration so as to eliminate the phase shift caused by the acquisition means for at least one frequency in a band including the center frequency;
Correction means for correcting distortion of the subject image caused by the vibration for each line of the image sensor using a correction amount calculated from the vibration compensated by the compensation means;
An imaging apparatus comprising:   The calculating means calculates the frequency characteristics of the distortion amounts of the plurality of bands by integrating the time characteristics of the vibrations separated into the plurality of bands with respect to the exposure period in each band. Item 2. The imaging device according to Item 1.   3. The time characteristic of the vibration is compensated so that the compensation means eliminates a phase shift caused by the IIR filter at least in the center frequency of the band including the center frequency. 4. The imaging device described in 1.   When each of the plurality of bands includes at least a frequency band that overlaps with another band, the calculation unit is determined based on a parameter of the imaging device at the time of imaging the image The imaging apparatus according to claim 1, wherein a weighting coefficient for each of a plurality of bands is multiplied by a frequency characteristic of a distortion amount of the plurality of bands. The parameter of the imaging device is at least one of vibration at the time of imaging, focal length, main subject information, shutter speed, and scene mode,
The calculation means estimates a shooting situation of the imaging device at the time of shooting the image based on the parameters at the time of shooting, and uses the weighting coefficient predetermined for each of the estimated shooting situations. The imaging apparatus according to claim 4, wherein a weighting coefficient for each band is determined.   The calculating means determines a weighting coefficient for each of the plurality of bands based on a weighting coefficient predetermined for each of the estimated shooting situations and a reliability of estimation of the shooting situation. The imaging device according to claim 5.   The calculation means calculates a distortion amount of the subject image generated in the image due to the vibration during a period from the start of exposure of the previous line to the start of exposure of the next line between consecutive lines of the image sensor. The imaging apparatus according to claim 1, wherein the frequency characteristic is calculated for each band.   The imaging apparatus according to claim 1, wherein the calculation unit includes a downsampling unit that downsamples a sampling frequency of the time characteristic of the vibration. A method for controlling an imaging apparatus including an imaging element driven by a rolling shutter system,
A detection step of detecting vibrations of the imaging device during an exposure period of the imaging device when the detection means captures an image using the imaging device;
An acquisition step in which an acquisition unit acquires a time characteristic of the vibration detected in the detection step;
A calculating step in which the calculating means separates the time characteristic of the vibration into a plurality of bands with respect to the frequency of the vibration, and calculates a frequency characteristic of the distortion amount of the subject image generated in the image due to the vibration for each band. When,
A determining step for determining, among the frequency characteristics of the distortion amounts of the plurality of bands, a band in which the frequency characteristic of the distortion amount is maximum as a band including the center frequency of the distortion amount; and
A compensating step for compensating a time characteristic of the vibration so that a phase shift generated in the obtaining step is eliminated for at least one frequency in a band including the center frequency;
A correcting step for correcting distortion of a subject image caused by the vibration for each line of the image sensor using a correction amount calculated from the vibration compensated in the compensating step;
An image pickup apparatus control method comprising:   The program for functioning a computer as each means of the imaging device of any one of Claims 1 thru | or 8.

Images