JP2022021684A - Image shake correction device and method for controlling the same, and imaging apparatus - Google Patents

Abstract

To reduce a variation of an image shake correction effect to stabilize image quality.SOLUTION: An imaging apparatus 1000 comprises an image pick-up device 6, an image shake correction unit 14, and a vibration detection unit 15. A camera system control unit 5 included in a camera main body unit 1 estimates an offset component of output from the vibration detection unit 15 that detects vibration of the camera main body unit 1. In offset estimation processing, a parameter value of a Kalman filter is changed by using table data indicating the relation of the Allan variance σ(τ) to a measurement time interval τ of the output from the vibration detection unit 15. The camera system control unit 5 performs control of the image shake correction by moving the image pick-up device 6 in a plane perpendicular to a photographic optical axis 4 based on the output from the vibration detection unit 15 and the estimated offset component.SELECTED DRAWING: Figure 3

Description

The present invention relates to a technique for image stabilization processing of an image in an image pickup device or the like.

In the image pickup device having an image blur correction function, an inertial sensor such as an angular velocity sensor is used to detect the amount of camera shake, and control is performed to drive a part or all of the image pickup optical system based on the shake detection information of the image pickup device. The output signal of the inertial sensor contains DC components such as variations in the reference voltage due to individual differences, and drift occurs due to temperature changes.

When using an angular velocity sensor, the DC component and drift are factors that cause an angle error when the angle is obtained by time integration. Therefore, there is a method of suppressing the deterioration of the image blur correction accuracy due to the angular error by removing the DC component and the drift from the output signal of the angular velocity sensor by the filter processing. Patent Document 1 discloses a method of removing a DC component and a drift included in an output signal of an inertial sensor by using a Kalman filter.

Japanese Unexamined Patent Publication No. 2018-105938

In the conventional technique, if the parameter value of the Kalman filter is not appropriate, the influence on the image stabilization performance and the like may be large. For example, when the parameter value is determined by the experience of the setter and trial and error so as to obtain the result intended by the setter, the quality of the Kalman filter depends on the ability of the setter.

In addition, due to the large drift, there is a possibility that the output fluctuation of the inertial sensor (hereinafter referred to as "fluctuation") may occur due to the passage of time for each individual. In this case, it is not realistic for the setter to set the parameter value for each individual sensor for the gyro sensor in which the appropriate parameter value of the Kalman filter differs for each individual. Therefore, even when a gyro sensor having a large fluctuation is used, the parameter value of the Kalman filter is not set for each sensor but is set uniformly. As a result, if the image stabilization effect varies from individual to individual imager, there is a possibility that the image quality will differ even with the same model.
An object of the present invention is to suppress variations in the image stabilization effect and stabilize image quality.

The image blur correction device of one embodiment of the present invention is an image blur correction device including image blur correction means for correcting image blur of an image, and is a shake detection signal output by the detection means and a plurality of captured images. An estimation means that acquires a motion vector between images and estimates the offset component of the detection signal by filter processing, and an image blur correction means that suppresses the offset component estimated by the estimation means with respect to the detection signal. The estimation means includes, and the estimation means, the parameter value of the filtering process is changed by using statistical data showing the relationship of the angle error with respect to the measurement time interval of the output of the detection means. ..

According to the present invention, it is possible to suppress variations in the image stabilization effect and stabilize the image quality.

It is a figure which shows the configuration example of the image pickup apparatus of an embodiment. It is a block diagram which shows the functional structure of the control part provided in the image pickup apparatus. It is a figure explaining the relationship of the angle error with respect to the measurement time interval of the runout detection output. It is a flowchart explaining the process in 1st Embodiment. It is the figure which showed the time change of the angular velocity. It is a block diagram which shows the functional structure of the control part in 2nd Embodiment. It is a figure explaining the calculation method of the table which represents the allan variance. It is a flowchart explaining the process in 2nd Embodiment. It is a control block diagram of the image pickup apparatus of 3rd Embodiment. It is a block diagram for demonstrating operation of an offset correction part. It is a figure which shows the output waveform example of the angular velocity sensor at the time of static. It is a figure which shows the time change of an angle signal. It is explanatory drawing of the method of estimating the angle fluctuation by the output error of the angular velocity sensor. It is a flowchart explaining the process in 3rd Embodiment. It is a flowchart explaining the process in the modification 1 of the 3rd Embodiment. It is a flowchart explaining the process in the modification 2 of the 3rd Embodiment. It is a flowchart explaining the process in the modification 3 of the 3rd Embodiment. It is a control block diagram of the image pickup apparatus of 4th Embodiment. It is a control block diagram of the image pickup apparatus of 5th Embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[First Embodiment]
FIG. 1A is a central sectional view of the image pickup apparatus 1000 of the present embodiment, and FIG. 1B is a block diagram showing an electrical configuration of the image pickup apparatus 1000. The direction perpendicular to the paper surface of FIG. 1A is defined as the x direction, and the two directions orthogonal to each other in the paper surface are defined as the y direction and the z direction, respectively. FIG. 1 shows an example of an imaging system including a camera main body (hereinafter, simply referred to as a main body) 1 and an interchangeable lens 2 that can be attached to the main body 1. The main body 1 and the interchangeable lens 2 can be connected to each other via an electric contact 11. For example, the main body 1 has a so-called mirrorless camera configuration. The present invention is not limited to the mirrorless camera, and the present invention can be applied to an interchangeable lens type single-lens reflex camera and various electronic devices having an image pickup function.

The main body 1 includes an image sensor 6, an image stabilization unit 14, a shake detection unit 15, and a shutter mechanism 16. The interchangeable lens 2 includes an optical member constituting an image pickup optical system and a runout detection unit 18. With the interchangeable lens 2 mounted on the main body 1, the image pickup apparatus 1000 can perform image blur correction of the captured image based on the shake detection signals acquired by the shake detection units 15 and 18.

The interchangeable lens 2 includes an optical system 3, and the lens drive unit 13 performs focus adjustment, aperture drive, image shake correction, and the like. As a result, the light from the subject passes through the optical system 3, and a good image is formed by the image pickup device 6 of the main body 1. The lens drive control is performed by the lens system control unit (hereinafter referred to as the lens control unit) 12. The lens control unit 12 includes a CPU (Central Processing Unit) and controls the interchangeable lens 2.

The shake detection unit 18 detects the shake of the interchangeable lens 2 in the first direction which is the direction of the photographing optical axis 4, the second direction orthogonal to the first direction, and orthogonal to each other, and the third direction. And outputs a plurality of detection signals to the lens control unit 12. For example, the first direction is a direction parallel to the z direction, the second direction is a direction parallel to the x direction, and the third direction is a direction parallel to the y direction. The lens control unit 12 controls the correction lens (shift lens, tilt lens, etc.) included in the optical system 3 via the lens drive unit 13.

The main body 1 includes an operation detection unit 10 that detects an instruction operation by the photographer using a release button or the like. For example, according to an instruction operation, the image sensor 6 performs photoelectric conversion with respect to the light from the subject. The image processing unit 7 acquires the output of the image sensor 6, performs image processing, and stores the image data in the memory unit 8. At this time, the image processing unit 7 can perform a vector operation for detecting the amount of movement of the subject image. Vector operation is an operation to obtain the moving direction and distance of feature points from images at different times. From the result of vector calculation using a known technique, the amount of movement of the subject image and the amount of camera shake can be obtained. The specific calculation method is omitted.

The camera system control unit (hereinafter referred to as a camera control unit) 5 includes a CPU and controls the control of the image pickup apparatus 1000. The camera control unit 5 controls the image pickup system in cooperation with the lens control unit 12 while communicating with the lens control unit 12 via the electric contact 11 in a state where the interchangeable lens 2 is attached to the main body unit 1. The live view display before shooting is performed by the display unit 9 under the control of the camera control unit 5. The display unit 9 is composed of a liquid crystal display unit 9a provided on the back surface of the main body unit 1 and an electronic viewfinder 9b having a display device in the finder. The memory unit 8 has a storage medium, and image data acquired by photographing is stored in the storage medium.

The shutter mechanism 16 is driven and controlled by the camera control unit 5 via the shutter drive unit 17. The exposure time of the image sensor 6 is controlled according to the shooting seconds set by the photographer or the shooting seconds determined by the camera control unit 5.

The image blur correction unit 14 corrects the image blur by moving the image sensor 6 according to the control command of the camera control unit 5. The image blur correction unit 14 is an image pickup element in a second or third direction orthogonal to the photographing optical axis 4 of the interchangeable lens 2, for example, based on an output correction value calculated from the output of the shake detection unit 15 of the main body unit 1. Move 6 This makes it possible to correct image blur caused by camera shake due to camera shake of the photographer or the like.

The runout detection unit 15 detects the runout of the main body unit 1 in three directions, and outputs a plurality of runout detection signals to the camera control unit 5. For example, the runout detection unit 15 has an angular velocity sensor. The camera control unit 5 acquires the output of the shake detection unit 15, calculates the output correction value thereof, and controls the movement of the image sensor 6 with respect to the photographing optical axis 4.

In the configuration example of FIG. 1, image blur correction based on the movement control of the image sensor 6 has been described, but the present invention is not limited to this, and can be applied to the following embodiments.
(1) Optical image stabilization based on the drive control of the correction lens included in the interchangeable lens 2.
(2) Electronic image stabilization based on image processing (image area extraction processing) for captured images.
(3) Complex image stabilization by the configuration of FIG. 1 or the combination of any of (1) and (2).

Next, with reference to FIG. 2, the offset estimation unit 5002 included in the camera control unit 5 will be described in detail. FIG. 2 is a block diagram showing a functional configuration for explaining the operation of the camera control unit 5. The camera control unit 5 includes an adder 5001, 5003, an offset estimation unit 5002, an integration filter 5004, and a storage unit 5100. In the following, it is assumed that the calculation performed by the adder includes addition (subtraction) of negative values.

The adder 5001 acquires the output signal of the angular velocity sensor possessed by the runout detection unit 15 and the signal of the camera runout amount calculated by the vector calculation in the image processing unit 7, and calculates the difference between the two signals. The adder 5001 outputs a difference signal to the offset estimation unit 5002.

The data stored in the storage unit 5100 is input to the offset estimation unit 5002 in order to estimate the fluctuation of the output of the vibration detection unit 15 from the difference signal. The storage unit 5100 stores the data of the observed noise dispersion 5101 and the system noise dispersion 5102. These data are data calculated from a table showing the relationship of the angle error with respect to the measurement time interval of the output of the runout detection unit 15 at rest, and the details thereof will be described later.

The offset estimation unit 5002 executes the offset estimation process based on the input signal. That is, the offset estimation unit 5002 acquires the fluctuation signal superimposed on the output of the vibration detection unit 15. The fluctuation signal acquired by the offset estimation unit 5002 is sent to the adder 5003.

The adder 5003 acquires the fluctuation signal of the output of the runout detection unit 15 and the output signal of the runout detection unit 15 and performs subtraction. By subtracting the fluctuation signal from the output signal of the shake detection unit 15, it is possible to acquire the output signal of the shake detection unit 15 from which the influence of the fluctuation is removed or reduced. The integration filter 5004 acquires the output correction value of the runout detection unit 15 by performing time integration on the output signal of the adder 5003. This output correction value is input to the image blur correction unit 14 and used to control the amount of movement of the image sensor 6.

Next, the estimation process performed by the offset estimation unit 5002 will be described in detail. The offset is a fluctuation including a DC component included in the output of the runout detection unit 15. An example of configuring the offset estimation unit 5002 with a linear Kalman filter and performing filter processing is shown. The linear Kalman filter can be expressed by the following equations (1) to (7).

Equation (1) represents an operation model in a state-space representation, and equation (2) represents an observation model. The meaning of each symbol is as follows.
A: System matrix in the operation model.
B: Input matrix.
C: Output matrix in the observation model.
ε _t : Process noise.
δ _t : Observation noise.
t: Discrete time.
_ut : Control input.

式（３）は予測ステップにおける事前推定値を表し、式（４）は事前誤差共分散を表す。式（４）中の、Σ_ｘは動作モデルのノイズの分散を表す。

Equation (3) represents the pre-estimated value in the prediction step, and equation (4) represents the pre-error covariance. In equation (4), Σ _x represents the noise variance of the operation model.

式（５）はフィルタリングステップにおいてカルマンゲインの算出式を表し、上付き添え字のＴは転置行列を表している。さらに式（６）はカルマンフィルタによる事後推定値を表し、式（７）は事後誤差共分散を表す。また式（５）中のΣ_ｚは観測モデルのノイズの分散を表す。

Equation (5) represents the Kalman gain calculation equation in the filtering step, and the superscript T represents the transposed matrix. Further, Eq. (6) represents the post-estimation value by the Kalman filter, and Eq. (7) represents the post-error covariance. Further, Σ _z in Eq. (5) represents the noise variance of the observation model.

In the present embodiment, in order to estimate the fluctuation of the output of the runout detection unit 15, the offset value of the output of the runout detection unit 15 is set to _xt , and the output of the runout detection unit 15 is set to _zt . Further, assuming that ε _t is system noise and δ _t is observation noise, the model representing the offset value does not have _ut , which is the control input term in equation (1), and A in equations (1) and (2). It can be represented by the following linear linear model in which = C = 1.

で表す。また、式（５）における観測モデルのノイズの分散Σ_ｚを、

で表す。 The noise dispersion Σ _x of the operation model in the above equation (4) is the system noise variance.

It is represented by. In addition, the noise variance Σ _z of the observation model in Eq. (5) is determined.

It is represented by.

とすると、時刻ｔにおけるシステムノイズの事前誤差分散推定値は、

となる。さらに時刻ｔにおけるカルマンゲインを、

とし、振れ検出部１５の出力を、

とすると、以下の式でカルマンフィルタを構成することができる。

Pre-estimated offset,

Then, the preliminary error variance estimated value of the system noise at time t is

Will be. Furthermore, the Kalman gain at time t,

The output of the runout detection unit 15 is

Then, the Kalman filter can be constructed by the following equation.

と、振れ検出部１５により出力されるシステムノイズの分散

と、時刻「ｔ－１」での誤差分散推定値

により、オフセット事前推定値

および事前誤差分散推定値

が算出される。そして事前誤差分散推定値

および振れ検出部１５により出力される観測ノイズの分散

に基づいてカルマンゲイン

が算出される。 The offset estimation unit 5002 performs an estimation calculation using the calculation formulas from the above formulas (10) to (14). Offset estimation value at time "t-1" in the update cycle of the estimation operation

And the dispersion of the system noise output by the runout detection unit 15.

And the error variance estimate at time "t-1"

By offset pre-estimated value

And prior error variance estimates

Is calculated. And prior error variance estimates

And the dispersion of the observed noise output by the runout detection unit 15.

Kalman gain based on

Is calculated.

と、オフセット事前推定値

との誤差にカルマンゲイン

を乗じた値によってオフセット事前推定値

が修正され、オフセット推定値

が算出される。 By equation (13)
Output of the observed runout detection unit 15

And offset pre-estimated value

Kalman gain in error with

Offset pre-estimated value by multiplying by

Has been corrected and the offset estimate

Is calculated.

が修正されて、誤差分散推定値

が算出される。これらの演算によって事前推定更新と修正を演算周期ごとに繰り返すことでオフセット推定値が算出される。 Further, the preliminary error variance estimated value is calculated by Eq. (14).

Has been corrected and the error variance estimate

Is calculated. The offset estimation value is calculated by repeating the pre-estimation update and correction for each calculation cycle by these operations.

および観測ノイズの分散

がカルマンフィルタのパラメータに相当する。パラメータ値の設定方法として、従来の方法では、推定の収束速度とフィルタの安定性とを考慮して観測ノイズの分散

を定めるという指針しか示されていない。設定者はその意図した結果が得られるようにパラメータ値を設定するが、ドリフトといった慣性センサ出力の揺らぎが大きいジャイロセンサに対して、設定者がセンサ個体ごとに値を設定することは現実的ではない。例えば、揺らぎの大きい慣性センサを使用する場合にカルマンフィルタのパラメータ値をセンサ個体によらず一律に定める場合を想定する。この場合、個々の撮像装置ごとに像ブレ補正効果にばらつきが発生すると、同一機種でも画像品位に差異が生じる可能性がある。 Dispersion of system noise in the Kalman filter configured as described above

And dispersion of observed noise

Corresponds to the parameter of the Kalman filter. As a method of setting the parameter value, in the conventional method, the dispersion of the observed noise is taken into consideration in consideration of the estimated convergence speed and the stability of the filter.

Only the guideline to determine is shown. The setter sets the parameter value so that the intended result can be obtained, but it is not realistic for the setter to set the value for each sensor for the gyro sensor where the fluctuation of the inertial sensor output such as drift is large. do not have. For example, when using an inertial sensor with large fluctuations, it is assumed that the parameter values of the Kalman filter are uniformly set regardless of the individual sensor. In this case, if the image stabilization effect varies from one image pickup device to another, the image quality may differ even with the same model.

With reference to FIG. 3, a table based on statistical data showing the relationship of the angle error with respect to the measurement time interval of the output of the runout detection unit 15 and the allan dispersion used in the table will be described in detail. FIG. 3A shows an example of a table showing the relationship of the angle error with respect to the measurement time interval of the output of the runout detection unit 15. The value of the Allan variance σ (τ) corresponding to each measurement time interval τ and the slope between adjacent data are shown. The measurement time interval τ in this example is the sampling time Δt or a power of 2 thereof.

式（１６）のＫは、データ数ｍ個のときのグループの数を表す。式（１７）のτはデータ数ｍ個のグループにおける測定時間間隔τを表す。式（１５）では、データ数ｍ個のグループにおいて隣り合うグループの平均値の差の平均を算出する演算が行われる。よって、アラン分散は、データを測定時間間隔τで測定した際の測定データの標準偏差（以下、偏差ともいう）を示していることがわかる。 FIG. 3B is a log-log graph, the horizontal axis represents the measurement time interval τ, and the vertical axis represents the Allan variance σ (τ). Further, the solid line 301 shows the noise characteristics of the output signal of the runout detection unit 15 at rest. The allan variance σ (τ) is widely used as an index of the frequency stability of the output of an inertial sensor such as an angular velocity sensor, and represents the degree of variation in the output at the measurement time interval τ. It's called variance, but it's actually a standard deviation. However, σ (τ) is generally called the Allan variance. Allan variance σ when the sampling time Δt and the total number of N data (here, 2 ⁿ ) are divided into m (m = 1, 2, 4, 8, ..., N / 2) groups. (Τ) is obtained by the following equation (15).

K in the equation (16) represents the number of groups when the number of data is m. Τ in the equation (17) represents the measurement time interval τ in a group of several m of data. In equation (15), an operation is performed to calculate the average of the differences between the average values of adjacent groups in a group having m data. Therefore, it can be seen that the Allan variance indicates the standard deviation (hereinafter, also referred to as deviation) of the measured data when the data is measured at the measurement time interval τ.

Since the measurement time interval τ is the time interval for acquiring data, it can be rephrased as the sampling time in a group of several m of data. Therefore, the deviation of the measured data when the data is sampled at the measurement time interval τ can be read from the graph of the Allan variance in FIG. 3 (B). At this time, the measurement time interval τ of the data plotted at the left end of the solid line 301 is the sampling time Δt at the time of data acquisition, and a deviation of the measurement time interval τ shorter than that cannot be acquired. In the table shown in FIG. 3A, the value of the allan variance in τ when m = 1 to N / 2 in the equation (17) is shown. The solid line 301 in FIG. 3B shows the table data on a log-log graph.

From the graph showing the noise characteristics due to the Allan variance in FIG. 3B, it is possible to know the characteristics of the noise of the signal when the signal is measured at the sampling time τ. Generally, the noise included in the output of the inertial sensor includes white noise, 1 / f noise, and random walk noise, and the noise showing the main feature changes by changing the measurement time interval τ. In FIG. 3B, the straight line portion of the inclination “−1 / 2” shown in the first range 302 is a range showing the characteristics of white noise. Further, in FIG. 3B, the straight line portion of the slope “+1/2” shown in the second range 303 is a range showing the characteristics of the random walk noise. Although not shown, it is known that when the signal contains 1 / f noise, a linear range having a slope of 0 appears between the range 302 and the range 303.

In FIG. 3B, as shown by the solid line 301, there is one continuous curve, but in reality, the deviations between the white noise and the random walk noise contained in the noise of the signal, that is, the allan variance, are shown. It is the sum of the lines that represent it. Therefore, by extending each slope line in FIG. 3B, it is possible to read the Allan variance when the noise corresponding to the slope is sampled at the measurement time interval τ. The solid line 301 in FIG. 3B is a line showing the noise characteristics of the output signal of the runout detection unit 15 at rest, but the output signal itself can be regarded as observation noise. Therefore, the solid line 301 is a line representing the allan variance of the observed noise in Eq. (12), that is, the deviation σ _z with respect to the sampling time of the observed noise.

Further, the broken line 304 in FIG. 3B represents the random walk noise included in the output signal of the shake detection unit 15 at rest, that is, the dispersion of fluctuations. This is a line representing the allan variance of the system noise in Eq. (11), that is, the deviation σ _x with respect to the sampling time of the system noise.

Therefore, the value of the Allan variance of τ corresponding to the desired sampling time can be read from these lines. By using the acquired allan variance values in the equations (10) to (14), it is possible to set each variance corresponding to the parameter of the Kalman filter from the allan variance according to the inertial sensor output.

Next, with reference to FIG. 3, a method for acquiring each variance when the sampling time is Δt will be described. First, in the case of the deviation σ _z of the observed noise representing the output noise of the inertial sensor, the measurement time interval τ coincides with the sampling time Δt of the output noise. Therefore, the deviation σ _z of the observed noise can be obtained from the Allan variance when τ = Δt in the equation (15).

式（１８）の右辺に示すＫはレートランダムウォーク係数と呼ばれ、τ＝３のときに傾き１／２の線が取るアラン分散の値である。この値は、実線３０１で傾きが１／２になる範囲を延長した破線３０４の切片を算出することで求まる値である。図３（Ａ）のテーブルからＫ値を求める場合、アラン分散の傾きが１／２になる範囲の中から任意の１点を選び、式（１８）を用いてＫについて解けばよい。傾き１／２の範囲内であれば、システムノイズの偏差σ_ｘはアラン分散と一致する。尚、傾きが１／２と一致しない場合には、傾き０および１と区別できる範囲内で傾き１／２に対して許容誤差を設け、その範囲内から選択してもよい。 Further, in the case of the deviation σ _x of the system noise representing the fluctuation in the output noise of the inertial sensor, the allan variance can be obtained from the value when the measurement time interval τ = Δt in the following equation.

K shown on the right side of the equation (18) is called the rate random walk coefficient, and is the value of the allan variance taken by the line with a slope of 1/2 when τ = 3. This value is a value obtained by calculating the intercept of the broken line 304 which is an extension of the range where the slope becomes 1/2 on the solid line 301. When obtaining the K value from the table of FIG. 3A, any one point may be selected from the range in which the slope of the allan variance becomes 1/2, and K may be solved using the equation (18). Within the range of slope 1/2, the system noise deviation σ _x agrees with the Allan variance. If the inclination does not match 1/2, a tolerance may be provided for the inclination 1/2 within a range that can be distinguished from the inclinations 0 and 1, and the inclination may be selected from the range.

The system noise deviation σ _x and the observation noise deviation σ _z obtained by the equations (15), (18), and (19) are obtained by removing the amount of deflection of the main body 1 superimposed on the output of the deflection detection unit 15. The actual deviation obtained from the signal. Therefore, by using this method, it is possible to set the parameter value of the Kalman filter according to the individual even for the gyro sensor having a large fluctuation.

For the parameter values of the Kalman filter of the present embodiment, a process of calculating a table from the data acquired in advance in the stationary state of the angular velocity sensor of the runout detection unit 15 at a factory or the like is performed. The data of the dispersion of each noise calculated from the table is stored in the storage unit 5100. In addition to this method, there is a method of generating a table using the data of the angular velocity sensor acquired when the image pickup apparatus 1000 is in a stationary state and calculating the parameter value of the Kalman filter.

With reference to FIGS. 4 and 5, the process of acquiring the output correction value of the runout detection unit 15 by the offset estimation unit 5002 will be described in detail. The output correction value of the shake detection unit 15 is an output obtained by removing the offset from the output of the shake detection unit 15, and corresponds to a target value of the shake correction amount of the image shake correction unit 14. In the present embodiment, the offset is estimated with respect to the output of the runout detection unit 15 in the main body 1, but the target of the offset estimation may be the output of the runout detection unit 18 in the interchangeable lens 2.

FIG. 4 is a flowchart illustrating the process from the time when the power of the image pickup apparatus 1000 is turned on to the time before the start of shooting, and is started when the photographer turns on the power of the main body 1. Further, FIG. 5 is a diagram showing an output signal, the horizontal axis is the time axis, and the vertical axis is the angular velocity. FIG. 5A is a diagram showing an output signal 51 of the runout detection unit 15. FIG. 5B is a diagram showing a runout amount 52 of the main body unit 1 acquired by the motion vector information of the image processing unit 7. FIG. 5C shows a signal 53 obtained by removing the runout amount 52 of the main body 1 from the output signal 51 of the runout detection unit 15, and an estimated waveform 54 (see the white line portion) of the offset component acquired by the offset estimation unit 5002. It is a figure. FIG. 5D is a diagram showing a waveform 55 obtained by removing the estimated waveform 54 of the offset component from the output signal 51 of the runout detection unit 15.

In S401 of FIG. 4, the camera control unit 5 acquires the output signal 51 of the runout detection unit 15 of the main body unit 1 and proceeds to the process of S402. In S402, the camera control unit 5 performs a process of detecting the runout amount 52 of the main body unit 1 from the motion vector information by the image processing unit 7, and then proceeds to the process of S403. The motion vector information is runout detection information based on comparison between a plurality of images acquired by the image sensor 6. From this motion vector information, the amount of runout of the main body 1 can be detected.

In S403, the camera control unit 5 acquires a signal 53 from the output signal 51 of the runout detection unit 15 of the main body 1 detected in S401, in which the runout amount 52 of the main body 1 detected in S402 is removed or reduced. Then, the process proceeds to S404.

In S404, the offset estimation unit 5002 estimates the offset component included in the signal 53 acquired in S403. The offset estimation unit 5002 executes the offset component estimation process using the signal 53 acquired in S403 and the data of the observation noise dispersion 5101 and the system noise dispersion 5102 stored in advance in the storage unit 5100. Since the method of estimating the offset component using the Kalman filter and the method of obtaining the observed noise variance 5101 and the system noise variance 5102 are as described above, their description is omitted. The waveform 54 shown by the white line portion in FIG. 5C represents the time change of the offset component estimated by the offset estimation unit 5002.

After calculating the estimated waveform 54 of the offset component, the camera control unit 5 proceeds to S405 and subtracts the signal of the estimated waveform 54 of the offset component from the output signal 51 of the runout detection unit 15 of the main body unit 1 to obtain FIG. 5 (D). The angular velocity waveform 55 of is obtained. The angular velocity waveform 55 is a waveform representing the amount of runout of the main body 1 detected by the runout detection unit 15 because the estimated waveform 54 of the offset component is removed from the output signal 51.

Next, proceeding to S406, the camera control unit 5 performs time integration processing on the angular velocity waveform 55 acquired in S405 by the integration filter 5004. The output correction value of the runout detection unit 15 representing the runout due to the angular displacement amount of the main body unit 1 is acquired, and the process is completed.

In the present embodiment, it is possible to set the parameter value of the Kalman filter according to the fluctuation of each individual inertial sensor in the offset estimation by the filter processing by the above series of operations. Stabilization of image quality can be achieved by suppressing variations in the image stabilization effect of image pickup devices of the same model.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In this embodiment, an example of processing for switching the parameter value according to the time change of the allan variance is shown. The main difference between the present embodiment and the first embodiment is that, in the case of the present embodiment, the processing is performed in consideration of the case where the fluctuation included in the output of the runout detection unit 15 changes with the passage of time. With respect to the same matters as in the first embodiment, the reference numerals and symbols already used will be used, and detailed explanations thereof will be omitted and the differences will be mainly described. The method of omitting such a description is the same in the embodiments described later.

The operation of the camera control unit 5 will be described with reference to FIG. FIG. 6 is a block diagram showing a configuration example for explaining the operation of the camera control unit 5. The difference from FIG. 2 is that the table calculation unit 5201 and the determination unit 5202 in the storage unit 5100 are provided.

The adder 5001 of the camera control unit 5 calculates the difference between the output signal of the runout detection unit 15 and the signal of the camera shake amount acquired by the vector calculation of the image processing unit 7. The acquired difference data is input to the table calculation unit 5201. The details of the table calculation process will be described later.

The observed noise variance data and the system noise variance data acquired by the table calculation unit 5201 are stored in the storage unit 5100. At the time of storage, the determination unit 5202 in the storage unit 5100 compares the value of the system noise dispersion calculated by the table calculation unit 5201 with the value of the system noise dispersion 5102 stored in the storage unit 5100, and which value is used. Is determined to be stored in memory. Here, the reason why only the dispersion value of the system noise is compared is that it is only the system noise that the dispersion value changes under the influence of the fluctuation.

The data of the observed noise variance 5101 and the system noise variance 5102 stored in the storage unit 5100 and the difference data acquired by the adder 5001 are input to the offset estimation unit 5002. The offset estimation unit 5002 executes an estimation process of the offset component superimposed on the output of the runout detection unit 15.

The adder 5003 subtracts the offset component of the runout detection unit 15 acquired by the offset estimation unit 5002 from the output of the runout detection unit 15, so that the output signal of the runout detection unit 15 has the influence of fluctuation removed or reduced. To get. The integration filter 5004 performs time integration of this output signal, and the output correction value of the shake detection unit 15 is acquired and input to the image blur correction unit 14.

Next, with reference to FIG. 7, the calculation method of the table representing the allan variance in the table calculation unit 5201 will be described in detail. This table is a table showing the relationship of the angle error with respect to the measurement time interval of the difference waveform of the adder 5001. Hereinafter, a method for determining the measurement time required to acquire the allan variance and a method for updating the allan variance will be described.

FIG. 7 is a graph showing the difference waveform 71 of the adder 5001 immediately after the power of the image pickup apparatus 1000 is turned on. The horizontal axis is the time axis, and the vertical axis represents the angular velocity. The angular velocity sensor included in the runout detection unit 15 has a temperature drift called start drift immediately after the power is turned on, and as shown in the range 72, a low frequency component is superimposed on the sensor output. Since the effect of this starting drift becomes smaller with the passage of time, the superposition of low frequency components also disappears with the passage of time. Since the magnitude of the fluctuation due to the drift continues to change until the start-up drift is eliminated, the deviation of the system noise also changes with the passage of time.

In order to acquire the deviation of the system noise that changes with the passage of time, the table calculation unit 5201 collects the difference waveform data of the adder 5001 immediately after the power of the image pickup apparatus 1000 is turned on. After a predetermined time (denoted as s seconds) has elapsed, acquisition of the deviation of the system noise is started from the allan variance of the difference waveform data. Here, s seconds is the measurement time required to acquire the deviation of the system noise. When the deviation of the system noise is acquired, the deviation of the observed noise is also acquired. The maximum measurement time interval of the allan variance obtained by the measurement for s seconds is s / 2 seconds. However, since the Allan variance obtained at the maximum measurement time interval is a value obtained by the difference of only two groups, the accuracy as a value representing the deviation is lower than the predetermined accuracy. Therefore, the measurement time of s seconds is set to a time at which the accuracy required for obtaining the deviation of the system noise can be sufficiently obtained. In addition, since the magnitude of fluctuation changes with the passage of time, it is necessary to update the deviation value of the system noise according to the change. The time T shown in FIG. 7 represents the elapsed time from the time when the table calculation unit 5201 starts to acquire the difference waveform data from the adder 5001, and has a relationship of “T ≧ s” with respect to s. After the lapse of T seconds, the table calculation unit 5201 calculates the allan variance for T-Δs seconds at any time according to the sampling time of the runout detection unit 15, acquires the deviations of the system noise and the observation noise, and continues the update. ..

With reference to FIG. 8, a series of processes until the camera control unit 5 acquires the output correction value of the runout detection unit 15 by the offset estimation unit 5002 will be described in detail. FIG. 8 is a flowchart illustrating a process from when the power of the image pickup apparatus 1000 is turned on to before the start of shooting. For the same processes as those shown in the flowchart of FIG. 6, the same step numbers as those of FIG. 6 are used, and detailed description thereof will be omitted.

After the processing of S401 to S403, the process proceeds to the processing of S801. In S801, after the power of the image pickup apparatus 1000 is turned on, the camera control unit 5 compares the elapsed time T with the measurement time s seconds. The elapsed time T is measured by a timer in the camera control unit 5. With respect to the elapsed time T, it is determined whether or not the measurement time s seconds has elapsed. If it is determined that "T ≧ s", the process proceeds to S802, and if it is determined that "T <s", the determination process of S801 is repeated.

In S802, the table calculation unit 5201 calculates the Allan variance using the difference waveform data of the adder 5001 in the s second immediately before the elapsed time T, and acquires each variance from the deviations of the observed noise and the system noise. The calculation of the allan variance and the acquisition method of the variance of each noise are as described above. After the acquisition processing of the dispersion of the observed noise and the dispersion of the system noise, the process proceeds to the processing of S404.

In S803 next to S406, the determination unit 5202 acquires the difference value between the system noise dispersion acquired in S802 and the system noise dispersion 5102 stored in the storage unit 5100, and whether or not the difference value is equal to or less than the threshold value. Is determined. This threshold value is set to a value at which it can be determined that the dispersion value of the system noise acquired in S802 has converged to the dispersion value of the system noise in the storage unit 5100. If it is determined that the acquired difference value is larger than the threshold value, the process proceeds to S804, and if it is determined that the difference value is equal to or less than the threshold value, a series of processes is terminated. The method is not limited to the method of comparing the difference value between the dispersion value acquired this time and the dispersion value acquired last time with the threshold value, and any method may be used as long as it can determine the convergence to the dispersion value.

In S804, after the process of updating the data values of the observed noise dispersion 5101 and the system noise dispersion 5102 stored in the storage unit 5100 with the dispersion values acquired in S802 is executed, the process proceeds to S802.

In the present embodiment, when the fluctuation included in the output of the runout detection unit 15 changes with the passage of time, a process of updating the parameter value of the filter means (Kalman filter) at any time according to the output of the runout detection unit 15 is performed. Image quality can be stabilized by suppressing variations in the image stabilization effect of image pickup devices of the same model.

[Third Embodiment]
A third embodiment of the present invention will be described with reference to FIGS. 9 to 17. In this embodiment, an example of an image stabilization device having the same image stabilization device as that of the above embodiment and capable of suppressing deterioration of image stabilization performance in long-second exposure and super-telephoto shooting is shown. In the prior art, it is difficult to control the image stabilization according to the characteristics of the vibration detection sensor, and it is not possible to accurately control only the output error of the vibration detection sensor.

In the present embodiment, as a countermeasure against the deterioration of the image stabilization performance due to the output error of the angular velocity sensor, a blur image is obtained when the predicted error is equal to or more than the threshold value based on the allan dispersion, the exposure time, and the focal length. Generation prevention processing is performed. An example of an image pickup apparatus including an image pickup optical system and an image processing apparatus will be described below.

FIG. 9 is a block diagram showing the configuration of the image pickup apparatus 100. The image pickup apparatus 100 is a digital camera, and includes a main body 101 and an interchangeable lens 102 that can be attached to the main body 101. The main body 101 includes an image sensor, an image stabilization unit, a shutter mechanism, and the like inside.

The camera CPU 201 included in the main body 101 controls the control of the image pickup apparatus 100, and controls each component shown in FIG. 9 in response to an instruction operation by the photographer using the release button 104 or the like.

The light from the subject passes through the image pickup optical system 213 along the photographing optical axis 214 shown in FIG. 9 and is incident on the image pickup element 210. The image pickup device 210 is a CMOS (complementary metal oxide semiconductor) image sensor or the like, and outputs an electric signal corresponding to an optical image in response to an input luminous flux. The signal output from the image sensor 210 is image-processed by the image processing unit, and the generated image data is stored in the storage unit. The focal plane shutter 212 controls the shielding state of the light incident on the image pickup device 210. The focal plane shutter 212 and the image sensor 210 are driven according to the control signal of the camera CPU 201.

The main body 101 includes a vibration sensor 202. The angular velocity sensor included in the vibration sensor unit 202 detects the angular velocity from the vibration applied to the main body unit 101. The angular velocity detection signal is input to the offset correction unit 203. The offset correction unit 203 refers to the motion vector information acquired from the plurality of captured images, and corrects the offset generated in the angular velocity sensor.

The calculation unit 204 acquires the angular velocity signal corrected by the offset correction unit 203, performs first-order time integration and the like, and calculates a target value. The calculation unit 204 calculates the correction amount for the image stabilization mechanism from the signal converted into the angle signal. The control unit 205 controls the correction unit 211 that drives the image blur correction mechanism based on the correction amount (angle information) calculated by the calculation unit 204. The correction unit 211 drives the image pickup device 210 in a direction orthogonal to the photographing optical axis 214 as shown by the arrow 211 (a) according to the control signal of the control unit 205. It is possible to correct image blurring that occurs on the image pickup surface of the image pickup device 210 due to the shake of the main body 101.

The characteristic calculation unit 206 of the vibration sensor performs calculation processing of the noise characteristics of the vibration sensor based on the angular velocity signal corrected by the offset correction unit 203. The integration error calculation unit 207 of the vibration sensor calculates the integration error based on the noise characteristics calculated by the characteristic calculation unit 206 and the exposure time and focal length information set by the photographer, and determines the amount of image blur under the shooting conditions. Predict. Information on the exposure time and focal length set by the photographer is acquired from the camera CPU 201.

The determination unit 208 performs a determination process for generating a blurred image. For example, the determination unit 208 compares the predicted value of the amount of blurring of the image calculated by the integration error calculation unit 207 with the threshold value, and determines whether or not the predicted value is equal to or greater than the threshold value. The blur image prevention processing unit 209 suppresses blur image generation when the determination unit 208 determines from the prediction result of the image blur amount obtained by the integration error calculation unit 207 that the magnitude of the image blur is equal to or larger than the threshold value. Alternatively, a command for prevention is sent to the camera CPU 201. The camera CPU 201 executes processing for suppressing or preventing blur image generation according to a command from the blur image prevention processing unit 209.

10 to 12 are explanatory views regarding offset correction of the angular velocity sensor. FIG. 10 is a block diagram for explaining the operation of the offset correction unit 203. The offset correction unit 203 includes a motion vector detection unit 1301, a multiplication unit 1302, and a division unit 1304.

The multiplication unit 1302 acquires the output signal of the angular velocity sensor of the vibration sensor unit 202 and the motion vector information output from the motion vector detection unit 1301 and performs multiplication. The deviation 1303 (denoted as e) calculated by the multiplication unit 1302 corresponds to the offset amount superimposed on the angular velocity sensor. The division unit 1304 acquires the information of the deviation e and divides the output of the angular velocity sensor according to the deviation e to generate an angular velocity signal obtained by dividing the offset amount.

11 (A) and 11 (B) are graphs showing an example of the output waveform of the angular velocity sensor at static. The vertical axis represents the angular velocity signal, and the horizontal axis is the time axis. The graph line 305 in FIG. 11A shows a state in which the angular velocity sensor is offset. In the state where the offset is generated, the angular velocity sensor may output a signal having a fluctuation that does not originally occur even in a statically indeterminate state where the vibration of the image pickup apparatus is small. Therefore, when image stabilization is executed based on a detection signal including an unnecessary signal component, an image with image blur is acquired due to erroneous image stabilization. On the other hand, the graph line 306 in FIG. 11B shows the waveform of the angular velocity sensor subjected to the offset correction processing. The waveform of the graph line 306 is a waveform with less fluctuation than the waveform of the graph line 305 in FIG. 11 (A).

The signal related to the angular velocity sensor generated by the offset correction (see graph line 306), that is, the signal of the angular velocity sensor with the offset reduced can be output, and the deterioration of the image blur correction performance can be suppressed.

FIG. 12A shows the time change of the angle signal, the horizontal axis is the time axis, and the vertical axis is the angle axis. The waveform shown by graph line 307 shows an angle signal obtained by integration processing by the calculation unit 204 (FIG. 9) with respect to the waveform of the angular velocity signal after offset correction shown by graph line 306 in FIG. 11 (B). When a delay such as arithmetic processing in the offset correction occurs, the correction remaining of the angular velocity sensor occurs, and an error may occur in the output signal. The waveform shown by the graph line 307 in FIG. 12A shows the integrated value of the angular velocity signal including the output error, and the angular fluctuation that does not originally occur can be confirmed. This angular fluctuation corresponds to the post-integral error with respect to the integration time. The effect of the angular fluctuation generated by the output error of the angular velocity sensor will be described in detail with reference to FIG. 12 (B).

FIG. 12B is a shooting sequence diagram when still image shooting is performed by the image pickup apparatus 100. The meaning of each symbol is as follows.
RLS: A signal output when the release button 104 is pressed.
SIG_A: Front curtain drive signal of the focal plane shutter 212 in FIG.
SIG_B: Rear curtain drive signal of the focal plane shutter 212 in FIG.
SIG_deg: Represents an angular waveform obtained by integrating a signal including an output error of an angular velocity, and corresponds to the waveform of graph line 307 in FIG. 12 (A).
A plurality of times t1 to t4 are shown on the time axis. The first period Tv_A is the period from time t2 to time t3, and the second period Tv_B is the period from time t2 to time t4. It is assumed that the period length of Tv_B is larger than the period length of Tv_A.

When the user presses the release button 104 at the time t1 in FIG. 12B, the RLS signal changes, and the image pickup apparatus 100 shifts from the non-shooting state to the shooting state. After the RLS signal changes, the SIG_A signal changes after a certain period of time, and the driving of the front curtain is started. Time t2 indicates the timing of starting the front curtain drive of the focal plane shutter 212. The SIG_B signal changes after a certain period of time from the output time of the SIG_A signal, and the driving of the rear curtain of the focal plane shutter 212 is started.

Drive control of the focal plane shutter 212 is performed according to the exposure time set by the photographer. The time t3 in FIG. 12B indicates the timing of starting the rear curtain drive according to the setting of the first exposure time with respect to the shooting condition of several seconds such as camera shake. The period Tv_A is a period corresponding to the first exposure time from the time t2 to the time t3.

On the other hand, the time t4 in FIG. 12B indicates the timing of starting the rear curtain drive according to the setting of the second exposure time longer than the first exposure time. For example, the second exposure time is an exposure time assuming shooting performed with the image pickup apparatus 100 fixed to a tripod or the like. The period Tv_B is a period corresponding to the second exposure time from the time t2 to the time t4.

For example, when a user takes a handheld image with a lens having a focal length of 100 mm, if the exposure condition is relatively short (a few seconds) as shown in the period Tv_A, the output error of the angular velocity signal is the cause of the angular signal. Since the fluctuation is small, there is no effect on image stabilization. However, when the camera is fixed to a tripod or the like and shooting is performed under relatively long exposure conditions as in the period Tv_B, the fluctuation of the angular signal due to the output error of the angular velocity signal cannot be ignored. .. That is, if erroneous image stabilization is performed due to the influence of fluctuations in the angle signal, an image having image blur may be generated.

Further, even under the exposure conditions shown in the period Tv_A, when the user takes a picture with a super-telephoto lens having a large focal length, even a slight movement of the image stabilization mechanism is increased and the subject is taken. To. Therefore, an image having image blur may be generated due to the movement of the image blur correction mechanism based on the fluctuation of the angular signal due to the output error of the angular velocity signal.

As described above, in shooting with a long exposure time by fixing the tripod, shooting using a super-telephoto lens, and the like, image blurring due to the output error of the angular velocity sensor may occur. In order to suppress the deterioration of the image blur correction performance due to the output error of the angular velocity sensor, the angle fluctuation due to the output error of the angular velocity sensor is estimated in advance, and the amount of image blur according to the shooting conditions is predicted appropriately. It needs to be processed.

A method of estimating the angular fluctuation due to the output error of the angular velocity sensor will be described with reference to FIG. Graph line 401 in FIG. 13 (A) shows the Alan standard deviation with respect to the angular velocity sensor output after offset correction shown by graph line 306 in FIG. 11 (B). The horizontal axis of FIG. 13A represents the measurement time interval τ of the angular velocity sensor output signal after offset correction. The vertical axis of FIG. 13A represents the standard deviation value στ of the angular velocity sensor output signal after offset correction in the measurement time interval τ. Both axes are shown on a logarithmic scale.

The Alan standard deviation corresponds to a quantity that represents the noise characteristics of the angular velocity sensor. The Alan standard deviation is widely used as an index when evaluating an inertial sensor such as an angular velocity sensor. With the Alan standard deviation, the standard deviation value of the sensor output signal to be evaluated can be evaluated on the time axis. In the present embodiment, the angle fluctuation can be estimated by utilizing the feature. The allan standard deviation is generally known by the name of the allan variance, but in order to clearly indicate that it is the standard deviation as described above, it is referred to as the allan standard deviation in the present embodiment.

In FIG. 13A, the horizontal axis is the measurement time interval τ of the angular velocity sensor output signal after offset correction. Therefore, as shown in the following equation (20), the value obtained by multiplying the Alan standard deviation value στ according to the measurement time interval τ by the measurement time interval τ can be treated as the angular fluctuation obtained by integrating the angular velocity.
τ (sec) × στ (dps) = Estimated angle fluctuation (degree) ・ ・ ・ Equation (20)
In FIG. 13A, the unit of the measurement time interval τ is “second”, and the unit of the Alan standard deviation στ corresponding to τ is “degrees per second”, that is, the angle per second. As an example, at the position where the measurement time interval τ is 20 seconds on the graph line 401 in FIG. 13 (A), the Alan standard deviation value στ of the angular velocity sensor output signal after offset correction according to the measurement time interval is 0. It is .0016 dpi. In this case, the calculation result using the equation (20) is as follows.
20 (sec) x 0.0016 (dps) = Estimated angle fluctuation 0.032 (degree)
That is, the estimated angle fluctuation calculated by the integration for 20 seconds is 0.032 (degree).

The graph line 402 in FIG. 13 (B) shows the data when the angle fluctuation estimated value is obtained by the above method based on the graph line 401 shown in FIG. 13 (A). The horizontal axis represents the measurement time interval τ, and the vertical axis represents the angle fluctuation estimated value. Both axes are shown on a logarithmic scale. As described above, by using the Alan standard deviation, it is possible to evaluate the standard deviation value of the angular velocity sensor output signal after offset correction on the time axis. As shown by the graph line 402 in FIG. 13 (B), the angle fluctuation estimated value on the time axis can be calculated. That is, the graph line 402 represents the estimated value of the angle fluctuation with the passage of time. By using the curve data shown in the graph line 402, the estimated value of the angle fluctuation at the exposure time can be obtained. The data of the estimated angle fluctuation value in the exposure time is stored in advance in the storage unit in the image pickup apparatus as table data. Therefore, it is possible to predict the amount of image blur based on the shooting conditions set by the photographer.

A processing example will be described with reference to FIG. FIG. 14 is a flowchart illustrating a camera sequence that predicts the amount of image blur and suppresses the deterioration of the image blur correction performance. When the power of the image pickup apparatus 100 is turned on in S501, the process proceeds to S502. In S502, the offset correction unit 203 executes the offset correction process of the angular velocity sensor.

In S503, the characteristic calculation unit 206 calculates the Alan standard deviation of the angular velocity sensor output based on the output signal of the angular velocity sensor offset-corrected in S502. Table data of the angle fluctuation estimated value in the exposure time is created. In S504, the shooting instruction determination process is executed. The camera CPU 201 determines whether or not the photographer has pressed the release button 104 to perform a shooting instruction operation. When the shooting instruction operation is detected, the process proceeds to S505. If the shooting instruction operation is not detected, the process returns to S503 and continues to create table data.

In S505, the camera CPU 201 determines whether or not the image blur correction performance is deteriorated due to the output error of the angular velocity sensor. Specifically, it is a shooting condition with a long exposure time such as by fixing a tripod (exposure time is longer than the threshold value) or a shooting condition using a super-telephoto lens (focal length is larger than the threshold value). These imaging conditions correspond to the imaging conditions in which the characteristics of the vibration detection means are dominant. If it is determined in S505 that the shooting conditions are not met, the process proceeds to S506 to end the process. If it is determined in S505 that the shooting conditions are met, the process proceeds to S507.

In S507, the integration error calculation unit 207 calculates a predicted value of image blur based on the exposure conditions and focal length information set by the photographer. In the next S508, the determination unit 208 performs the determination process by comparing the predicted value of the image blur with the threshold value. If it is determined that the predicted value is equal to or less than the threshold value, the process proceeds to S506 and the operation is terminated. On the other hand, if it is determined in S508 that the predicted value is larger than the threshold value, the process proceeds to S509.

In S509, the blur image prevention processing unit 209 outputs a command to the camera CPU 201 to perform a process for warning that an image blur will occur. The camera CPU 201 executes a process of warning the generation of a blurred image by, for example, a message on the display screen or a voice on the speaker. After the processing of S509, the process proceeds to S506 to end the operation.

In the present embodiment, it is possible to provide an image pickup apparatus capable of causing a user to recognize a decrease in image stabilization performance due to an output error of an angular velocity sensor in long-second exposure, super-telephoto shooting, or the like. An angular error prediction table is created based on the output result after offset correction of the angular velocity sensor. At that time, the time required for offset correction may be insufficient, and sufficient correction may not be possible. In that case, an angle error prediction table may be created from the output signal of the angular velocity sensor before offset correction. The angular velocity sensor information before offset correction causes fluctuations due to offset, but even if the predicted value of image blur is calculated based on that information, it is possible to make the user recognize the deterioration of image blur correction performance. be.

[Modification 1 of the third embodiment]
A modification 1 of the third embodiment will be described with reference to FIG. The main difference from the third embodiment is that signal processing is performed on the angular velocity signal. FIG. 15 is a flowchart illustrating a specific camera sequence for suppressing deterioration of image stabilization performance in Modification 1. In FIG. 15, for the same processes (S501 to S508) as in FIG. 14, the step numbers already used are used, and detailed description thereof will be omitted. Such an abbreviation method is the same in the modified example described later.

If it is determined in S508 that the predicted value of image blur is larger than the threshold value, the process proceeds to S601. In S601, the blur image prevention processing unit 209 instructs the camera CPU 201 to execute a process of shifting the time constant of the high-pass filter (HPF) to the high frequency side with respect to the output signal of the angular velocity sensor. This process is a process of switching the signal processing method of the vibration detection means. As a result, the low frequency characteristic that causes the offset is cut, so that the angular fluctuation caused by the output error of the angular velocity sensor can be suppressed. If the process of shifting the time constant of the HPF to the high frequency side is performed, it becomes impossible to detect low frequency components such as camera shake, but the control of this modified example is for shooting under conditions such as a tripod fixed state where camera shake is unlikely to occur. Will be carried out at. Therefore, there is little harmful effect due to the process of shifting the HPF to the high frequency side.

According to this modification, it is possible to suppress the deterioration of the image stabilization performance by changing the signal processing method of the vibration detection means.

[Modification 2 of the third embodiment]
A modified example 2 of the third embodiment will be described with reference to FIG. The main difference from the third embodiment is that image processing is performed on the captured image. FIG. 16 is a flowchart showing a specific camera sequence for suppressing deterioration of image stabilization performance in Modification 2.

If it is determined in S508 that the predicted value of image blur is larger than the threshold value, the process proceeds to S701. In this case, it is known in advance that the image to be captured is an image with image blur. Therefore, before the user confirms the captured image, the blur image prevention processing unit 209 instructs the camera CPU 201 to execute the image processing for correcting the blur image in S701. Image processing that corrects blurred images includes image processing that enhances the sharpness of images. This operation corresponds to the operation of switching the image processing method.

According to this modification, it is possible to suppress the deterioration of the image stabilization performance by changing the image processing method.

[Modification 3 of the third embodiment]
A modified example 3 of the third embodiment will be described with reference to FIG. The main difference from the third embodiment is that the process of changing the exposure time is executed. FIG. 17 is a flowchart showing a specific camera sequence for suppressing the deterioration of the image stabilization performance in the modification 3.

If it is determined in S508 that the predicted value of image blur is larger than the threshold value, the process proceeds to S801. In S801, the blur image prevention processing unit 209 instructs the camera CPU 201 to execute the automatic setting of the exposure time. Based on the angle error prediction table, a process of automatically setting a short exposure time so that the predicted value of image blur is equal to or less than the threshold value is executed, and shooting is performed with the set exposure time. This process corresponds to the process of switching the exposure conditions of the image pickup apparatus.

According to this modification, it is possible to suppress the deterioration of the image stabilization performance by automatically changing the exposure conditions.

[Fourth Embodiment]
A fourth embodiment of the present invention will be described with reference to FIG. An example of an image pickup device capable of suppressing deterioration of image stabilization performance even when various lens devices are attached to the main body of the image pickup device and used in an interchangeable lens camera system is shown. The process of calculating the Allan variance data based on the output of the vibration sensor and the motion vector will be described.

FIG. 18 is a block diagram relating to the control of the image pickup apparatus 100 including the main body 101 and the interchangeable lens 102. The interchangeable lens 102 includes a lens CPU 1202, an electric contact 1203, an image pickup optical system 213, a lens driving unit 1215, and a vibration sensor 1216.

When the main body 101 and the interchangeable lens 102 are mechanically connected, the camera CPU 1201 and the lens CPU 1202 are electrically connected via the electric contact 1203, and can communicate with each other. The lens CPU 1202 controls the lens driving unit 1215 according to a signal from the vibration sensor 1216 and a command from the camera CPU 1201. The lens driving unit 1215 drives the aperture of the imaging optical system 213 and the image blur correction lens according to the command of the lens CPU 1202. The image pickup optical system 213 is provided with a focus lens, a zoom lens, a shift lens, an aperture, and the like on the photographing optical axis 214, and forms an image of light from a subject.

The vibration sensor 1216 detects the vibration of the interchangeable lens 102, and the detection signal is transmitted to the lens CPU 1202. The vibration sensor 1216 is an inertial sensor such as an angular velocity sensor. The main body 101 includes a camera CPU 1201, an image sensor 210, a focal plane shutter 212, an image processing unit 1204, a storage unit 1205, a motion vector detection unit 1206, and an allan dispersion calculation unit 1207.

The camera CPU 1201 controls the control of the entire system. The camera CPU 1201 controls each component of the interchangeable lens camera system in response to an operation of a shooting instruction by the photographer using the release button 104 or the like.

The focal plane shutter 212 controls the shielding state of the light incident on the image pickup device 210 based on the command of the camera CPU 1201. The image pickup element 210 performs photoelectric conversion on the light formed through the image pickup optical system 213 and outputs an electric signal corresponding to the optical image to the image processing unit 1204.

The image processing unit 1204 has an A / D converter, a white balance adjustment circuit, a gamma correction circuit, an interpolation calculation circuit, and the like inside. The image processing unit 1204 executes a predetermined image processing and generates digital data of a storage image (video). Further, the image processing unit 1204 compresses still image data, moving image data, and audio data by using a known method.

The storage unit 1205 includes a semiconductor storage device that stores a program or the like executed by the camera CPU 1201, a memory card that stores various data generated by the image processing unit 1204, and the like. The semiconductor storage device is, for example, EEPROM (Electrically Erasable Programmable Read-Only Memory).

The motion vector detection unit 1206 detects a motion vector from a plurality of image signals having different imaging times processed by the image processing unit 1204. In the motion vector detection, the motion of an image can be detected by comparing the feature points of two images that are consecutive in time. Since the motion vector detection method is a well-known technique, a detailed explanation thereof will be omitted.

The allan dispersion calculation unit 1207 calculates the difference between the detection signal of the vibration sensor 1216 and the detection signal of the motion vector detection unit 1206, and calculates the allan dispersion using the difference. At this time, the detection signal of the vibration sensor 1216 and the detection signal of the motion vector detection unit 1206 are converted into angular velocities and then the difference calculation is performed. When converting the motion vector into an angular velocity, information such as the focal length of the interchangeable lens 102, the shooting interval (or frame rate) of the image used for detecting the motion vector, and the pixel pitch of the image pickup element 210 is used. The allan variance data calculated by the allan variance calculation unit 1207 can be referred to by the camera CPU 1201.

It is assumed that the image pickup device 100 is not fixed to a tripod or the like, for example, a shake occurs in the image pickup device 100 during preparation for handheld shooting. In this case, by removing the vibration signal of the image pickup device 100 detected by the motion vector detection unit 1206 from the detection signal of the vibration sensor 1216, it is possible to detect the noise and drift inherent in the vibration sensor 1216. The camera CPU 1201 can more accurately perform offset correction of the runout detection signal and signal processing for suppressing drift.

In the present embodiment, the camera CPU 1201 can acquire the data of the allan variance peculiar to the vibration sensor 1216 from the allan variance calculation unit 1207 even after the interchangeable lens 102 is attached to the main body portion 101. Therefore, it is possible to provide an image pickup device capable of suppressing deterioration of the image stabilization performance when various lens devices are attached to the main body portion. Although the lens device is shown as an example of the external device mounted on the main body of the image pickup device, the present invention can be applied to a system in which an external device such as an accessory having a vibration sensor can be mounted on the main body.

[Fifth Embodiment]
The image pickup apparatus 300 of the fifth embodiment of the present invention will be described with reference to FIG. Differences from the image pickup apparatus 100 of the fourth embodiment will be mainly described. The difference between FIGS. 19 and 18 is that the correction unit 211 and the correction control unit 1208 are provided. Further, the main body 101 includes a first vibration sensor 1209, and the interchangeable lens 102 includes a second vibration sensor 1216. That is, the vibration sensor 1209 detects the vibration of the main body 101 and outputs the detection signal to the camera CPU 1201. The vibration sensor 1216 detects the vibration of the interchangeable lens 102 and outputs the detection signal to the lens CPU 1202.

The correction control unit 1208 controls the correction unit 211 according to a control command from the camera CPU 1201, and the correction unit 211 drives the image sensor 210. The drive mechanism unit of the image pickup element 210 is configured so that the image pickup element 210 can be translated in a plane orthogonal to the photographing optical axis 214 and can be rotated around the photographing optical axis 214. The correction control unit 1208 sends a control command to the correction unit 211 to drive the image pickup device 210 so that the vibration detected by the vibration sensor 1209 is canceled out.

The Allan variance calculation unit 1307 determines whether or not the magnitude of the detection signal of the vibration sensor 1209 is equal to or less than a predetermined threshold value. As a predetermined threshold value, a threshold value for the detection signal of the vibration sensor 1209, which is used when it is determined that the image pickup apparatus 300 is supported by the tripod, for example, is set. When the magnitude of the detection signal of the vibration sensor 1209 is equal to or less than the threshold value, the allan dispersion calculation unit 1307 calculates the allan dispersion using the detection signal of the vibration sensor 1216. In the method of determining whether or not the image pickup device is supported by the tripod by using the detection signal of the vibration sensor, the state where the magnitude of the detection signal of the vibration sensor is smaller than the predetermined threshold value is longer than the predetermined threshold time. There is a method of judging whether or not it has continued for a long time. Since this technique is well known, detailed explanation is omitted. The same applies to the determination of whether or not the image pickup device is placed on a desk or the like.

It is assumed that the image pickup apparatus 300 is in a static state after the interchangeable lens 102 is attached to the main body 101. The statically indeterminate state is a state in which there is almost no runout, such as when the image pickup device is attached to a tripod or the like or when it is placed on a desk. In such a case, it is possible to acquire the detection signal of the vibration sensor 1216 and calculate the allan variance. That is, since it is possible to calculate the allan variance when the main body 101 of the image pickup apparatus 300 is not vibrating, it is possible to detect the noise and drift inherent in the vibration sensor 1216. The lens CPU 1202 can more accurately perform offset correction of the runout detection signal and signal processing for suppressing drift.

According to the present embodiment, it is possible to suppress a deterioration in image stabilization performance by an imaging system in which various lens devices can be attached to the main body of the imaging device. In this embodiment, an example is shown in which the detection signal of the vibration sensor 1209 originally used for controlling the correction unit 211 is used to determine the statically indeterminate state of the image pickup apparatus 300. Not limited to this, the statically indeterminate state of the image pickup apparatus 300 may be determined using the detection signals of other vibration sensors provided in the main body 101.

According to the above-described embodiment, it is possible to appropriately set the parameter value of the filter processing based on the statistical data and perform offset correction for the fluctuation of the inertial sensor for each individual, and the image blur correction effect of the image pickup apparatus can be performed. Variation can be suppressed.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and modifications can be made within the scope of the gist thereof. It is possible to carry out with a configuration in which the above-described embodiments and modifications are appropriately combined. Further, in the above embodiment, an example is shown in which an Allan variance data table is used as statistical data showing the relationship of the angle error with respect to the measurement time interval of the runout detection output. Not limited to this, it can be applied to embodiments using various statistical processes.

5 Camera control unit 7 Image processing unit 14 Image stabilization unit 15 Image stabilization unit 5002 Offset estimation unit 5100 Storage unit

Claims (20)

An image stabilization device provided with image stabilization means for correcting image blur of an image.
An estimation means that acquires a vibration detection signal output by the detection means and a motion vector between a plurality of captured images and estimates the offset component of the detection signal by filtering.
A control means for controlling the image blur correction means by a signal in which an offset component estimated by the estimation means is suppressed with respect to the detection signal is provided.
The estimation means is an image blur correction device characterized in that the parameter value of the filter processing is changed by using statistical data showing the relationship of the angle error with respect to the measurement time interval of the output of the detection means. The image stabilization device according to claim 1, wherein the estimation means estimates the offset component by a Kalman filter. The image blur correction device according to claim 1 or 2, wherein the estimation means changes the parameter value by using the standard deviation data related to the offset component as the statistical data. The image blur correction device according to claim 2, wherein the parameter value is a dispersion value of observation noise and a dispersion value of system noise. An update means for acquiring the detection signal and the motion vector and updating the statistical data is provided.
The image stabilization device according to any one of claims 1 to 4, wherein the estimation means acquires the updated statistical data and changes the parameter value. A storage means for storing data of the dispersion value of the observed noise and the dispersion value of the system noise, and
The image shake correction device according to claim 4, further comprising an updating means for acquiring the detection signal and the motion vector and updating the dispersion value of the system noise stored in the storage means. 6. The updating means is characterized in that the dispersion value of the system noise is updated when the difference value between the dispersion value of the system noise acquired this time and the dispersion value of the system noise acquired last time is larger than the threshold value. Image blur correction device according to. The image blur according to any one of claims 1 to 7, wherein the estimation means changes the parameter value by using the statistical data generated from the detection signal acquired in a stationary state. Compensator. An image pickup apparatus comprising the image stabilization apparatus according to any one of claims 1 to 8. A calculation means for calculating the noise characteristics of the signal in which the offset component is suppressed, and
Using the noise characteristics calculated by the calculation means, a calculation means for generating data showing the relationship of the integration error with respect to the integration time with respect to the detection signal, and
A process of acquiring data generated by the calculation means, an exposure time of an image sensor, and a focal length of an image pickup optical system, and executing a process of suppressing or preventing blur image generation when the integration error is equal to or greater than a threshold value. The image pickup apparatus according to claim 9, further comprising means. The processing means is one or more of a process of warning the generation of a blurred image, a process of switching a signal processing method of the detection signal, a process of switching an image processing method for an captured image, and a process of switching exposure conditions. 10. The image pickup apparatus according to claim 10. The imaging device according to claim 10 or 11, wherein the calculation means calculates a standard deviation related to the detection signal and generates table data. The image pickup apparatus according to any one of claims 10 to 12, wherein the processing means executes the processing when a predetermined shooting condition is satisfied and the integration error is equal to or larger than a threshold value. .. The imaging device according to claim 13, wherein the calculation means predicts image blurring of an captured image from information on an exposure time and a focal length when the shooting conditions are satisfied. The imaging device according to claim 13, wherein the photographing condition is a photographing condition in which the exposure time is longer than the threshold value or the focal length is larger than the threshold value. The detecting means has an angular velocity sensor and has an angular velocity sensor.
One of claims 10 to 15, wherein the calculation means multiplies the standard deviation value corresponding to the measurement time interval of the detection signal by the measurement time interval to calculate data representing an angle variation. The imaging device according to. An external device having a detection means for detecting vibration can be attached to the main body.
The feature is that the external device is provided with a calculation means for calculating the statistical data by calculating the difference between the detection signal of the detection means included in the external device and the motion vector in a state of being mounted on the main body. 9. The imaging device according to claim 9. An external device can be attached to the main body,
The first detecting means for detecting the vibration of the main body, and the
When the magnitude of the detection signal of the first detection means is equal to or less than the threshold value in the state where the external device is attached to the main body, the detection signal of the second detection means provided in the external device is used. The image pickup apparatus according to claim 9, further comprising a calculation means for calculating the statistical data. The image pickup apparatus according to any one of claims 9 to 18, wherein the image shake correction means performs image shake correction by moving an image pickup device or a correction lens or image processing. It is a control method executed by an image stabilization device provided with image stabilization means for correcting image blur of an image.
An estimation step of acquiring a vibration detection signal output by the detection means and a motion vector between a plurality of captured images and estimating the offset component of the detection signal by filtering.
It has a control step of controlling the image blur correction means by a signal that suppresses an offset component estimated in the estimation step with respect to the detection signal.
The control method is characterized in that in the estimation step, a process of changing a parameter value of the filter process is performed using statistical data showing the relationship of an angle error with respect to a measurement time interval of the output of the detection means.

Images