JP2021033233A - Image blur correction device and control method thereof, program, and imaging apparatus including image blur correction device - Google Patents

Abstract

To make it possible to precisely correct a shifting blur caused by the runout of an imaging apparatus.SOLUTION: The image blur correction device includes: a motion vector detector that detects a motion vector from two images acquired by an imaging device at regular time intervals; an acquisition unit that acquires runout signals from a runout detector that detects the runout of the imaging apparatus; a comparison unit that finds a correlation between the motion vector and the runout signal; and a correction amount acquisition unit that acquires the image blur correction amount based on the comparison result by the comparison unit and the runout signal.SELECTED DRAWING: Figure 3

Description

The present invention relates to a technique for correcting image blurring caused by shake of an image pickup apparatus.

In recent years, many image pickup devices have come to be equipped with an image blur correction mechanism (image plane image blur correction mechanism) that corrects image blur caused by the shake of the device by moving the image sensor in parallel. Further, in order to realize higher image blur correction performance, many proposals have been made regarding the method of image blur correction control.

Patent Document 1 describes a method of stopping correction when the signal level is small with respect to image blur (angle blur) when the camera rotates and image blur (shift blur) when the camera moves in parallel. It is disclosed. According to the method described in Patent Document 1, it is possible to prevent erroneous correction of image blur in a state where there is no runout such as when installed on a tripod. As a result, it is possible to provide the user with an appropriate image.

Patent Document 2 discloses a method in which the center of rotation is estimated from the values of one accelerometer and two angular velocity meters, and shift blur correction is performed using the values of the angular velocity meters during exposure.

Japanese Unexamined Patent Publication No. 2017-194529 Japanese Unexamined Patent Publication No. 2012-888466

However, even when the methods disclosed in Patent Document 1 and Patent Document 2 are used, the shift blur may not be corrected well depending on the user.

That is, according to Patent Document 1, control can be appropriately switched depending on the state of the camera (difference between stationary state such as tripod installation and handheld state), but it cannot be said that it corresponds to individual differences of users. .. Further, Patent Document 2 does not disclose a method for determining the certainty of the estimated rotation center.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to enable accurate correction of shift blur caused by runout of an image pickup apparatus.

The image blur correction device according to the present invention includes a motion vector detecting means for detecting a motion vector from two images acquired by an image sensor at regular time intervals, and a runout signal from a shake detecting means for detecting the runout of the image pickup device. The acquisition means for acquiring the image blur correction amount, the comparison means for obtaining the correlation between the motion vector and the runout signal, and the correction amount acquisition means for acquiring the image blur correction amount based on the comparison result by the comparison means and the runout signal. , It is characterized in that.

According to the present invention, it is possible to accurately correct the shift blur caused by the runout of the image pickup apparatus.

The figure which shows the structure of the single-lens reflex type digital camera with interchangeable lenses. An exploded perspective view of the image blur correction mechanism in the image sensor unit. The figure which shows the structure of the image blur correction control part. The figure explaining the internal structure of the output correction part. The figure explaining the correction method in the case of a signal with high correlation. The figure explaining the correction method in the case of a signal with low correlation. The figure explaining the operation of the motion vector detection part. A flowchart illustrating the flow of imaging operation of a digital camera.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate explanations are omitted.

FIG. 1 is a diagram showing a configuration of a single-lens reflex type digital camera 100 with interchangeable lenses, which is an embodiment of the image pickup apparatus of the present invention. FIG. 1A is a central sectional view of the digital camera 100 in one embodiment, and FIG. 1B is a block diagram showing an electrical configuration of the digital camera 100.

In FIG. 1A, the digital camera 100 includes a camera body 1 and a lens 2 that is detachably attached to the camera body 1. The lens 2 includes a photographing optical system 3 composed of a plurality of lenses centered on the optical axis 4. The lens 2 includes a lens driving unit 13 so that the light flux from the subject transmitted through the lens 2 can be well imaged on the image sensor unit 6 of the camera body 1. The lens driving unit 13 receives a control signal from the lens system control circuit 12 and drives the photographing optical system 3. The photographing optical system 3 includes a focus adjusting unit, an aperture driving unit, and the like.

Further, the camera body 1 includes a shutter mechanism 16, an image sensor unit 6, a rear display unit 9a, an electronic viewfinder (hereinafter referred to as EVF) 9b, and a shake detection unit 15. An electrical contact 11 that electrically connects the camera body 1 and the lens 2 is arranged between the camera body 1 and the lens 2. In FIG. 1 (b), the lens 2 has a lens system as an electrical configuration. It includes a control circuit 12, a lens drive unit 13, and a memory 25. The lens system control circuit 15 controls the entire lens 2 by executing a program stored in the memory 25. The lens driving unit 13 receives a control signal from the lens system control circuit 12 and drives the photographing optical system 3.

Further, as an electrical configuration, the camera body 1 includes a camera system control circuit 5, an image sensor unit 6, an image processing unit 7, a memory 8, a display unit 9 including a rear display unit 9a and an EVF 9b, an operation detection unit 10, and an operation detection unit 10. A runout detection unit 15 and a runout correction drive unit 14 are provided. The camera system control circuit 5 controls the entire digital camera 100 by executing a program stored in the non-volatile memory unit of the memory 8. The image sensor unit 6 includes an image sensor that photoelectrically converts a subject image and outputs an image signal, and an image blur correction mechanism that moves the image sensor in a direction perpendicular to the optical axis (including tilting). The image processing unit 7 performs image processing necessary for the image signal output from the image sensor of the image sensor unit 6. The memory 8 has a non-volatile memory unit for storing a program and a volatile memory unit for temporarily storing image data. The operation detection unit 10 accepts a user's operation. The display unit 9 displays captured images, information indicating the state of the camera, and the like. The runout detection unit 15 detects the runout amount of the digital camera 100. The shake correction driving unit 14 drives the image blur correction mechanism of the image sensor unit 6 to move the image sensor in a plane perpendicular to the optical axis 4 to perform image shake correction. The image sensor position detection unit 21 detects the moving position of the image sensor in the image sensor 6.

From a functional point of view, the digital camera 100 including the camera body 1 and the lens 2 has an imaging means, an image processing means, a recording / reproducing means, and a control means.

The image pickup means includes a photographing optical system 3, an image sensor unit 6, and a shutter mechanism 16, and the image processing means includes an image processing unit 7. Further, the recording / reproducing means includes a memory 8 and a display unit 9. The display unit 9 includes a rear display unit 9a and an EVF9b, which is a small display panel (not shown) for displaying shooting information provided on the upper surface of the camera body 1. The control means includes a camera system control circuit 5, an operation detection unit 10, a shake detection unit 15, a shake correction drive unit 14, a lens system control circuit 12, and a lens drive unit 13. The lens system control circuit 12 drives a focus lens (not shown), an aperture, a zoom, and the like.

The runout detection unit 15 includes a plurality of inertial sensors, and can detect the movement and rotation of the digital camera 100. Specifically, a vibrating gyro or the like is used. The shake correction driving unit 14 drives the image blur correction mechanism in the image sensor unit 6 to shift or tilt the image sensor on a plane perpendicular to the optical axis 4.

To explain each of the above means in more detail, the image pickup means is an optical processing system that forms an image of light from an object on the image pickup surface of the image pickup device via the photographing optical system 3. Since information on the focus evaluation amount / appropriate exposure amount can be obtained from the image sensor, the photographing optical system 3 is adjusted based on this information. As a result, an object light having an appropriate amount of light can be exposed on the image sensor in a focused state. The shutter mechanism 16 controls the exposure state and the light-shielding state of the image sensor by traveling the shutter curtain. The shutter mechanism 16 includes at least a curtain (mechanical rear curtain) for blocking the subject image, and the exposure is completed by the shutter mechanism 16. Further, in the present embodiment, the image sensor has a mode (electronic front curtain) in which the timing of the start of exposure is controlled by resetting the electric charge for each line prior to the traveling of the rear curtain of the shutter mechanism 16. In the electronic front curtain mode, exposure control is performed by synchronously operating the charge reset (electronic front curtain) of the image sensor and the rear curtain of the shutter mechanism 16. Since the electronic front curtain is described in many prior art documents, detailed description thereof will be omitted.

The image processing unit 7 constituting the image processing means has an A / D converter, a white balance adjustment circuit, a gamma correction circuit, an interpolation calculation circuit, and the like inside, and generates an image for recording. The color interpolation processing unit 7 is provided in the image processing unit 7, and performs color interpolation (demosiking) processing from the signals of the Bayer array to generate a color image. In addition, the image processing unit 7 compresses still images, moving images, sounds, and the like using a predetermined method.

The memory 8 includes an actual storage unit. The camera system control circuit 5 outputs image data to the storage unit of the memory 8 and displays an image to be presented to the user on the display unit 9.

The camera system control circuit 5 constituting the control means generates and outputs a timing signal or the like at the time of imaging. The imaging system, image processing system, and recording / playback system are controlled according to external operations. For example, the camera system control circuit 5 detects the pressing of the shutter release button (not shown) to control the driving of the image sensor in the image sensor unit 6, the operation of the image processing unit 7, the compression processing, and the like. Further, the state of each segment of the display unit 9 that displays information is controlled. Further, the rear display unit 9a has a touch panel and is connected to the operation detection unit 10.

Next, the adjustment operation of the photographing optical system will be described. An image processing unit 7 is connected to the camera system control circuit 5, and an appropriate focus is obtained based on a signal from the image sensor provided in the image sensor unit 6 and an operation of the photographer detected by the operation detection unit 10. Find the position and aperture position. The camera system control circuit 5 issues a command to the lens system control circuit 12 via the electrical contact 11, and the lens system control circuit 12 controls a focal length changing unit and an aperture driving unit (not shown). Further, in the mode of performing image shake correction, the shake correction driving unit 14 is controlled based on the signal obtained from the shake detection unit 15 and the detection information of the image sensor position detection unit 21. The image sensor unit 6 includes, for example, an image blur correction mechanism having a magnet and a flat plate coil. Further, the image sensor position detection unit 21 includes, for example, a magnet and a Hall element.

Here, the flow of control for image blur correction in the present embodiment will be briefly described. In the present embodiment, the image blur correction control unit is roughly divided into a shake detection unit 15 that detects camera shake, a shake correction drive unit 14 that performs an image blur correction operation, and a shake correction drive unit from the signals of the shake detection unit 15. It is composed of a camera system control circuit 5 that generates 14 target values and performs drive control. First, the operation detection unit 10 detects an operation (ON of the switch SW1) in which the release button (not shown) is pushed down halfway to enter the preparatory shooting operation. This is an aiming operation that determines the so-called composition. At this time, in order to facilitate composition determination, image blur correction is performed using the shake correction driving unit 14. That is, image blur correction is realized by controlling the runout correction drive unit 14 based on the signal from the runout detection unit 15. After that, the operation detection unit 10 detects an operation (ON of the switch SW2) for entering the shooting operation by completely pressing the release button. At this time, image blur correction is performed using the shake correction driving unit 14 in order to suppress blurring of the subject image acquired by exposure. The image blur correction operation is stopped after a certain period of time has passed after the exposure.

FIG. 2 is an exploded perspective view showing the mechanical configuration of the image blur correction mechanism in the image sensor unit 6. In FIG. 2, the vertical line indicates a direction parallel to the optical axis 4. In FIG. 2, the non-moving member (= fixed portion) is designated by a reference numeral in the 100s, and the moving member (= movable portion) is designated by the reference numeral in the 200s. Further, the balls sandwiched between the fixed portion and the movable portion are designated by a code in the 300 series.

In FIG. 2, the image blur correction mechanism is fixed to the upper yoke 101, screws 102a, 102b, 102c, upper magnets 103a, 103b, 103c, 103d, 103e, 103f, auxiliary spacers 104a, 104b, main spacers 105a, 105b, 105c. A part rolling plate 106a, 106b, 106c, a lower magnet 107a, 107b, 107c, 107d, 107e, 107f, a lower yoke 108, a screw 109a, 109b, 109c, and a base plate 110 are provided. The image blur correction mechanism further includes an FPC 201, mounting positions 202a, 202b, 202c, a movable PCB 203, a movable portion rolling plate 204a, 204b, 204c, a coil 205a, 205b, 205c, a movable frame 206, and balls 301a, 301b, 301c. .. The mounting positions 202a, 202b, and 202c indicate positions where a plurality of position detecting elements constituting the image sensor position detecting unit 21 are mounted.

The upper yoke 101, the upper magnets 103a, 103b, 103c, 103d, 103e, 103f, the lower magnets 107a, 107b, 107c, 107d, 107e, 107f, and the lower yoke 108 form a magnetic circuit, forming a so-called closed magnetic path. There is. The upper magnets 103a, 103b, 103c, 103d, 103e, and 103f are adhesively fixed to the upper yoke 101 in a state of being attracted to them. Similarly, the lower magnets 107a, 107b, 107c, 107d, 107e, and 107f are adhesively fixed to the lower yoke 108 in a state of being attracted to them. The upper magnets 103a, 103b, 103c, 103d, 103e, 103f and the lower magnets 107a, 107b, 107c, 107d, 107e, 107f are magnetized in the optical axis direction (vertical direction in FIG. 2), respectively, and adjacent magnets (upper and lower magnets in FIG. 2) are magnetized. The positional relationship of the magnets 103a and 103b) is magnetized in different directions. Further, the opposing magnets (positional relationships such as magnets 103a and 107a) are magnetized in the same direction. By doing so, a magnetic flux having a strong density in the optical axis direction is generated between the upper yoke 101 and the lower yoke 108.

Since a strong suction force is generated between the upper yoke 101 and the lower yoke 108, the main spacers 105a, 105b, 105c and the auxiliary spacers 104a, 104b are configured to maintain an appropriate distance. The appropriate spacing referred to here is that the coils 205a, 205b, 205c and FPC201 are arranged between the upper magnets 103a, 103b, 103c, 103d, 103e, 103f and the lower magnets 107a, 107b, 107c, 107d, 107e, 107f. The intervals are such that an appropriate gap can be secured at times. The main spacers 105a, 105b, 105c are provided with screw holes, and the upper yoke 101 is fixed to the main spacers 105a, 105b, 105c by the screws 102a, 102b, 102c. Rubber is installed on the body of the main spacers 105a, 105b, 105c to form a mechanical end (so-called stopper) of the movable part.

The base plate 110 is provided with holes so as to avoid the lower magnets 107a, 107b, 107c, 107d, 107e, and 107f, and the surface of the magnet is configured to protrude from the holes. That is, the base plate 110 and the lower yoke 108 are fixed by the screws 109a, 109b, 109c, and the lower magnets 107a, 107b, 107c, 107d, 107e, 107f, which are larger in thickness direction than the base plate 110, protrude from the base plate 110. It is fixed to do.

The movable frame 206 is formed of magnesium die-cast or aluminum die-cast, and is lightweight and has high rigidity. Each element of the movable portion is fixed to the movable frame 206 to form the movable portion. In the FPC 201, the position detection element constituting the image pickup element position detection unit 21 is attached to the surface on the side that cannot be seen from FIG. 2 at the positions indicated by the attachment positions 202a, 202b, and 202c. As the position detecting element, for example, a Hall element or the like can be used so that the position can be detected by using the above-mentioned magnetic circuit. Since the Hall element is small, it is arranged so as to be nested inside the windings of the coils 205a, 205b, 205c.

An image sensor 6, coils 205a, 205b, 205c and a Hall element (not shown) are connected to the movable PCB 203. Electrical communication with the outside is performed via a connector on the movable PCB 203.

The fixed portion rolling plates 106a, 106b, 106c are adhesively fixed to the base plate 110, and the movable portion rolling plates 204a, 204b, 204c are adhesively fixed to the movable frame 206, and the rolling surfaces of the balls 301a, 301b, 301c are adhered to the base plate 110. Form. By separately providing the rolling plate, it becomes easy to design the surface roughness and hardness in a preferable state.

By passing an electric current through the coil with the above configuration, a force according to Fleming's left-hand rule is generated in the coils 205a, 205b, 205c, and the movable part can be moved. Further, by using the signal of the Hall element which is the position detection element described above, the image blur correction operation of the image sensor can be feedback-controlled. By appropriately controlling the value of the signal of the Hall element, the movable frame 206 can be translated and rotated around the optical axis in a plane orthogonal to the optical axis 4.

By driving the signals of the Hall elements arranged at the mounting positions 202b and 202c in opposite phases while keeping the signals of the Hall elements arranged at the mounting positions 202a constant, a rotational motion of approximately four optical axes is generated. Can be done.

The magnetic flux densities detected at the mounting positions 202a, 202b, and 202c are the magnetic flux densities in the optical axis direction. The characteristics of a magnetic circuit including upper magnets 103a, 103b, 103c, 103d, 103e, 103f and lower magnets 107a, 107b, 107c, 107d, 107e, 107f, etc. are generally non-linear. Therefore, the magnetic flux densities detected at the mounting positions 202a, 202b, and 202c do not necessarily have a constant resolution in the entire drive range. There are positions where the change in magnetic flux density is steep and gentle, and the steeper the position, the higher the detection resolution. In the magnetic circuit shown in FIG. 2, the change in magnetic flux density is the largest at the boundary position of the magnet (for example, the boundary position between the magnets 103a and 103b), and the detection resolution is high.

FIG. 3 is a diagram showing a configuration of an image blur correction control unit 300 including a camera system control circuit 5, an image processing circuit 7, a shake detection unit 15, and a shake correction drive unit 14. FIG. 3A is a block diagram showing the configuration of the image blur correction control unit 300, and FIG. 3B is a table showing concretely the signals to be compared in the comparison unit. In FIG. 3, the same functional parts as those in FIG. 2 are assigned the same numbers.

In FIG. 3, the image blur correction control unit 300 includes a shake detection unit 15, and the shake detection unit 15 has a vibration gyro 15a and an accelerometer 15b. Further, the image blur correction control unit 300 includes an integrator / gain adjuster 21a and 21b, a target generator 22 in the preliminary shooting operation, a coordinate converter 23, a motion vector detection unit 24, an adder 25, and a first band passage filter. It includes 26a and 26b, and a comparison unit 27. Further, the image blur correction control unit 300 includes an integrator 28, second band pass filters 29a and 29b, a radius of gyration estimator 30, an image magnification information storage unit 31, an output correction unit 32, and a target generator 33 in shooting operation. To be equipped. The difference between the pre-shooting operation and the shooting operation here is whether or not the acquired image is finally recorded in the non-volatile memory. The case of recording is called a shooting operation, and the case of not recording is called a shooting preliminary operation. Among the components of FIG. 3, the motion vector detection unit 24 is included in the image processing unit 7.

The operation of each part of the image blur correction control unit 300 will be described with reference to FIG. The vibration gyro 15a is an angular velocity meter (rotational runout detection). The accelerometer 15b (translational runout detection) detects the acceleration associated with the translation of the camera. Similar to the case of Patent Document 2, the radius of gyration is estimated in the present embodiment as well. That is, the output of the accelerometer 15b is integrated by the integrator 28 to obtain a translational velocity signal, and the velocity signal of the band used for estimating the radius of gyration is further extracted through the second band pass filter 29b. On the other hand, the signal from the vibration gyro 15a also extracts the angular velocity signal of the band used for estimating the radius of gyration through the second band pass filter 29a. The turning radius estimator 30 compares the above-mentioned speed signal and the angular velocity signal (divides the speed signal by the angular velocity signal), and calculates and estimates the turning radius. Since the method of obtaining the turning radius is described in Patent Document 2, detailed description thereof will be omitted here.

Here, the accelerometer 15b can detect at least two axes of acceleration in the in-plane direction of the image sensor. For example, the longitudinal direction of the image sensor is x, the lateral direction is y, and the acceleration in the biaxial directions of x and y is detected. The integrated velocity signals also have values in the biaxial directions of x and y, and are shown as output signals vX and vY of the second bandpass filter 29b in FIG. 3A. The vibrating gyro 15a can detect the angular velocities in the three axes of Pitch around the x-axis, Yaw around the y-axis, and Roll around the z-axis. This signal is shown as the output signal ωP / Y / R of the vibrating gyro 15a in FIG. 3A. As a specific configuration, both the vibration gyro 15a and the accelerometer 15b may be provided with a plurality of sensors capable of detecting one axis, or a sensor capable of detecting multiple axes (such as a three-axis accelerometer) may be provided.

Due to such a configuration, the radius of gyration estimator 30 is a multi-input / multi-output system. This is schematically shown in FIG. 3 (b). The input of the angular velocity is described in the horizontal direction, and the three-dimensional signals of ωP, ωY, and ωR are shown. The velocity input is shown vertically and the vX, vY two-dimensional signals are shown. ΔX and δY will be described later. As a result, there are six signal combinations. The combination of ωP and vX is shown as I, the combination of ωY and vX is shown as II, and the combination of ωR and vY is shown as VI as shown in the table below. The radius of gyration is estimated for each of these. Therefore, the output of the radius of gyration estimator 30 is a six-dimensional signal.

Next, the operation of the comparison unit in the preliminary shooting operation will be described.

In the image pickup apparatus of the present embodiment, when an image is displayed on the display unit 9 in the preliminary shooting operation (so-called live view), an image blur correction operation is performed. In FIG. 3A, the signal of the vibration gyro 15a is converted into an angle signal via the integrator / gain adjuster 21a, and then input to the target generator 22 in the preparatory operation for photographing, whereby the image sensor unit 6 is subjected to. The target signal C1 / 2/3 of each actuator is generated. This target signal is the target value of the position detection element described with reference to FIG. The amount of energization of the coils 205a, 205b, 205c is adjusted by the feedback control so that the position detection element becomes the target signal. As a result, the image blur correction operation is realized.

At this time, the target generator 22 in the pre-shooting operation may have characteristics that can realize the ease of aiming, and appropriate control is performed so that a stroke in the shooting operation can be left. Here, considering the subject image on the image sensor, only the blur after the image blur is suppressed by the above image blur correction operation remains from the blur that originally occurred. That is, the remaining blur is observed. Images are acquired at regular intervals by the image sensor, and the motion vector detection unit 24 measures the amount of motion between two images having a time interval. The operation of the motion vector detection unit 24 will be described later with reference to FIG. The motion vector detection unit 24 observes the so-called residual amount of blurring.

The coordinate converter 23 converts the target signal C1 / 2/3 of the position detection element into the movement amount of the image pickup element 6. That is, the driving amount in each coil is converted into the movement of the image sensor 6. This amount is the amount for which image blur correction has been performed. In the adder 25, the total amount of runout acting on the digital camera 100 can be obtained by adding the output of the coordinate converter 23 and the output of the motion vector detection unit 24. By passing the obtained amount of runout acting on the digital camera 100 through the first band-passing filter 26a, only the band used for obtaining the correlation is extracted. At the same time, the output signal of the vibrating gyro 15a is passed through the first band-passing filter 26b, and similarly, only the band used for obtaining the correlation is extracted. These signals are input to the comparison unit 27 to obtain the correlation. The specific method of obtaining the correlation will be described later with reference to FIGS. 4 and 5.

Consider the input / output of the comparison unit 27 in the same manner as the turning radius estimator 30. The signal input to the comparison unit 27 through the first bandpass filter 26b is three-dimensional in ωP, ωY, and ωR. On the other hand, the output of the motion vector detection unit 24 is a deviation (to be exact, a difference) between the two images in the X and Y directions. The output of the first bandpass filter 26a has the same dimension as the output of the motion vector detection unit 24, and the amount of motion (= difference value) between the two images in the two directions of x and y is output. This is illustrated as δX and δY in FIG. 3A.

That is, the comparison unit 27 has the same multi-input / multi-output system as the turning radius estimator 30. This is schematically shown in FIG. 3 (b). The input of the angular velocity is described in the horizontal direction, and the three-dimensional signals of ωP, ωY, and ωR are shown. The input from the motion vector detection unit 24 is described in the vertical direction, and two-dimensional signals of δX and δY are shown. As a result, there are six signal combinations. The combination of ωP and δX is shown as I, the combination of ωY and δX is shown as II, and as shown in the table below, there are sequential combinations, and the combination of ωR and δY is shown as VI. Correlation is calculated for each of these. Therefore, the output of the comparison unit 27 is a six-dimensional signal.

Finally, a signal processing method in the shooting operation will be described. The basic operation is the same as in Patent Document 2. In Patent Document 2, after passing the signal of the angular velocity meter through the HPF / integration filter and the gain adjustment, the gain is adjusted by the output correction unit in accordance with the turning radius. At this time, the gain is adjusted in consideration of the shooting magnification from the zoom and focus information.

Also in this embodiment, the signal of the vibration gyro 15a (corresponding to the angular velocity meter) is converted into an angle signal by using the integrator / gain adjuster 21a (HPF / integrator filter, corresponding to the gain adjustment), and the output correction unit 32 Adjust the gain corresponding to the turning radius. Information about the radius of gyration is obtained from the radius of gyration estimator 30. In addition to the information regarding the radius of gyration, the image magnification information is acquired from the image magnification information storage unit 31, and the corresponding gain adjustment is performed. Since the relationship between the radius of gyration and the image magnification and the gain correction is disclosed in Patent Document 2, detailed description thereof will be omitted here.

One feature of this embodiment is that the output of the comparison unit 27 is given to the output correction unit 32. The control gain in the axial direction from the inertial sensor is changed depending on whether the comparison unit 27 determines (determines) that the correlation is high or low. The control gain is the ratio of the amount of correction to the detected shift runout by the image blur correction mechanism, and is sometimes called the correction ratio. The method of adjusting the gain using the correlation will be described in detail later with reference to FIG.

To explain a specific example, when it is determined that the correlation between ωY and δX is high (when the combination of II in FIG. 3B shows a high correlation), the image accompanying the translation in the X direction using ωY. Actively correct blur. On the contrary, when the correlation is low, the degree of image blur correction due to translation in the X direction using ωY is reduced. If the correlation is low, the degree of image blur correction may be set to 0, and the correction may be stopped for the axis with low correlation. As a result, the axis with high correlation is corrected more than the axis with low correlation, and appropriate shift blur correction is realized. In particular, even if the state of correlation changes depending on the state of the digital camera 100, such as individual differences and posture differences, it is possible to deal with it.

The information θP', θY', and θR'from the output correction unit 32 are sent to the target generator 33 in the photographing operation to generate the target signal C1 / 2/3 of each actuator. This target signal becomes the target value of the position detection element described with reference to FIG. The amount of energization of the coils 205a, 205b, 205c is adjusted by the feedback control so that the position detection element becomes the target signal. As a result, the image blur correction operation is realized.

At this time, it is desired that the target generator 33 in the photographing operation completely removes the blur. As described above, when the correction by the output correction unit 32 is appropriately performed, the shift blur correction is performed in a combination having a high correlation and an effect. As a result, when considering the subject image on the image sensor, both the angle blur and the shift blur are suppressed by the above image blur correction operation from the originally generated blur. As a result, a high-quality image can be provided.

The internal configuration of the output correction unit 32 will be described with reference to FIG. In FIG. 4, the same functional parts as those in FIG. 3 are designated by the same reference numerals.

The output correction unit 32 includes a correction gain calculator 32a, a correction gain adjuster 32b, gain units 32c1, 32c2, 32c3, 32c4, 32c5, 32c6, and adders 32d1, 32d2, 32d3, 32d4.

In FIG. 4, the radius of gyration obtained by the radius of gyration estimator 30 is shown as L1, L2, L3, L4, L5, L6, respectively. Subscripts 1 to 6 correspond to the Roman numerals shown in the table of FIG. 3 (b). Similarly, the correlations obtained by the comparison unit 27 are shown as C1, C2, C3, C4, C5, and C6, respectively. Subscripts 1 to 6 correspond to the Roman numerals shown in the table of FIG. 3 (b).

The correction gain calculator 32a determines the gain correction amount in each axial direction based on the method described in Patent Document 2. The gain determined from the turning radius L1 and the image magnification β is G1, the gain determined from the turning radius L2 and the image magnification β is G2, and so on, G1 to G6 are shown.

The correction gain adjuster 32b receives C6 from the correlation C1 obtained by the comparison unit 27 and adjusts the correction gain. As described above, when it is determined that the correlation is higher than the predetermined value, the value of the correction gain calculator 32a is output as it is. On the other hand, when it is determined that the correlation is equal to or less than a predetermined value, the value of the correction gain calculator 32a is output as a smaller value. This cuts the output from the sensor, which has low correlation and may cause harmful effects.

The output of the correction gain adjuster 32b is set in the gain units 32c1, 32c2, 32c3, 32c4, 32c5, 32c6, respectively. In the calculation of the gain units 32c1, 32c2, 32c3, 32c4, 32c5, 32c6 and the adders 32d1, 32d2, 32d3, 32d4, θP, θY, θR and θP', θY', and θR'are as shown in the following equation (1). Become a relationship.

θP'= θP + G1 * θP + G2 * θY + G3 * θR
θY'= θY + G4 * θP + G5 * θY + G6 * θR (1)
θR'= θR
If the image blur correction of shift blur does not work effectively, the correlation in the comparison unit 27 becomes low. In that case, the gains G1 to G6 become small values. As is clear from the equation (1), for example, when G1 to G6 = 0, θP'= θP, θY'= θY, θR'= θR, and the image blur correction of shift blur is not performed and only the angle blur is corrected. Will be done.

On the other hand, when the image blur correction of shift blur works effectively, the correlation in the comparison unit 27 becomes high. In that case, the gains G1 to G6 have values. As is clear from the equation (1), in this case, the target value (image blur correction amount) for correcting both the angle blur and the shift blur can be obtained by adding.

A method of obtaining the correlation in the comparison unit 27 will be described with reference to FIGS. 5 and 6. FIG. 5 shows an example when the correlation is high, and FIG. 6 shows an example when the correlation is low.

In the graphs of FIGS. 5A and 6A, the horizontal axis represents time and the vertical axis represents signal level. In the graphs of FIGS. 5 (b) and 6 (b), the horizontal axis represents the output of the motion vector detection unit 24, and the vertical axis represents the output of the inertial sensor. In FIGS. 5 and 6, 41 is the output of the motion vector detection unit 24, 42 is the output of the inertial sensor, and 43 is the approximate line when linearly approximated to the first principal component direction in the case of principal component analysis. 44 indicates the direction of the second principal component when the principal component analysis is performed, and 45 indicates the range in which the data is distributed.

First, consider the signal of FIG. In the example shown in FIG. 5A, as the output 41 of the motion vector detection unit 24 increases, the output 42 of the inertial sensor also increases, and a waveform close to the similarity is obtained. That is, it can be said that the positive correlation is high. In order to determine the correlation, the correlation coefficient may be calculated by the output 41 of the motion vector detection unit 24 and the output 42 of the inertial sensor. In determining the height of the correlation in the present embodiment, the correlation does not necessarily have to be a positive correlation, and even if it is a negative correlation, it may be determined that the correlation is high if the correlation is high. In terms of the correlation coefficient, it can be said that the correlation is high when the absolute value is close to 1.

Such signals having a higher correlation than a predetermined value are represented by the vertical and horizontal axes as shown in FIG. 5 (b). Since the other variable changes in a similar shape to one variable, a signal distribution 45 is formed around the approximate straight line 43. At this time, when the principal component analysis is performed by the output 41 of the motion vector detection unit 24 and the output 42 of the inertial sensor, the axial direction indicated by 43 as the first principal component and the axial direction indicated by 44 as the second principal component are obtained. Be done. Obviously, the variance in the direction indicated by 43 is large, and the variance in the direction indicated by 44 is small. By obtaining the ratio of the variances of the first principal component and the second principal component, it may be used as an index of the correlation between signals. That is, when the dispersion of the first principal component is sufficiently larger than the variance of the second principal component, it can be judged that the correlation is high.

Next, consider the signal of FIG. In the example shown in FIG. 6A, it is not seen that the output 41 of the motion vector detection unit 24 and the output 42 of the inertial sensor are interlocked with each other. In such a case, it can be judged that the correlation is low. When the correlation coefficient is obtained in the same manner as in the example of FIG. 5, the value is close to zero.

When such signals with low correlation are represented by the vertical and horizontal axes, as shown in FIG. 6 (b). Since the other variable changes completely differently from one variable, the signal distribution 45 is not concentrated around the approximate straight line 43. In such a case, when principal component analysis is performed using the output 41 of the motion vector detection unit 24 and the output 42 of the inertial sensor, the axial direction indicated by 43 as the first principal component is the axial direction indicated by 44 as the second principal component. Is obtained. In this case, unlike FIG. 5, the variance in the direction shown in 43 and the variance in the direction shown in 44 show similar values. In other words, the dispersion ratio of the first principal component and the second principal component is close to 1. That is, when the ratio of the variance of the first principal component to the variance of the second principal component is close to 1, it can be judged that the correlation is low.

The operation of the motion vector detection unit 24 will be described with reference to FIG. 7. The motion vector detection unit 24 measures the amount of motion between the two images. 7 (a) is an image acquired one sample before the calculation, FIG. 7 (b) is an image acquired at the time of calculation, and FIG. 7 (c) is FIG. 7 (a) to make the movement easy to understand. And the image in which the image of FIG. 7B is superimposed is shown.

In FIG. 7, 81 indicates the boundary of the acquired image, 82a and 82b indicate the subject, 83 indicates the region used for motion vector calculation, and 84 indicates the obtained motion vector. Many methods have been proposed for obtaining motion vectors, but here, a method using block matching will be described in order to make the explanation easier to understand. There are other methods such as using feature points, but in the present embodiment, any method may be used as long as the motion vector can be stably acquired.

The motion vector detection unit 24 determines a region to be a template in the image acquired one sample before shown in FIG. 7A. In FIG. 7A, the template area is shown by 83a. It is convenient to set the template area 83a so as to include a characteristic portion of the subject 82a. Next, a region similar to the region 83a is searched for in the image acquired at the time of calculation shown in FIG. 7 (b). For example, the XOR (exclusive OR) between two images may be calculated to find the region whose value is the minimum. The resulting region is shown in 83b. That is, in FIG. 7B, it is determined that the image in the region 83b is the most similar to the image in the region 83a. As shown in FIG. 7C, the vector 84 from the region 83a to the region 83b is a motion vector, and indicates the amount of motion between the two images.

As shown in FIG. 7, a so-called motion vector can be obtained by acquiring images at regular intervals by the image sensor 6 and comparing two images acquired at different times.

The flow of the imaging operation in this embodiment will be described with reference to FIG.

Step S100 is a step indicating the start of operation, and the power of the digital camera 100 is turned on.

In step S110, it is confirmed whether or not the power of the digital camera 100 has been turned off. When the power is turned off, the process proceeds to step S120 to stop the operation. If not, the process proceeds to step S130.

In step S130, it is determined whether or not the user holds the digital camera 100. Since the camera shake is affected by the posture difference (how to hold the digital camera 100, etc.) in addition to the individual difference of the user, it is preferable to perform the gain estimation operation after deciding the composition and holding the camera. For example, it may be determined that the shutter release button is pressed halfway down to enter the preparatory shooting operation (switch SW1 is turned on), or the signal level of the inertial sensor is used to determine the position. If it is determined that the user is ready, the process proceeds to step S140, and if it is determined that the user is not ready, the process returns to step S110.

In step S140, the gain estimation operation, which is the main part of the present embodiment, is performed. This content will be described with reference to FIG. 8 (b). Then, the process proceeds to step S150.

In step S150, it is determined whether or not the operation (switch SW2 ON) for entering the shooting operation by completely pressing the release button has been performed. If the switch SW2 is turned on, the process proceeds to step S160. If not, the process returns to step S110.

In step S160, a shooting operation is performed, and in the image blur correction control unit 300, the shake correction driving unit 14 is controlled based on the output of the target generator 33 in the shooting operation.

In FIG. 8B, step S200 is a step indicating the start of the gain estimation operation.

In step S210, the signal of the image sensor is read out. Here, an image is acquired from the image sensor in the live view state.

In step S220, the motion vector is calculated. The image obtained in step S210 is the current image and corresponds to FIG. 7B. The motion vector is calculated by comparing with the image acquired one sample before, which is stored in the memory 8. Further, the image obtained in step S210 is held in the memory 8 so as to be compared at the next timing.

In step S230, the amount of blur is calculated. Specifically, the filters 26a and 26b are used to extract signals in the required band from the motion vector from the motion vector detection unit 24 and the signal from the vibration gyro 15a.

In step S240, the comparison unit 27 compares the output of the filter 26a with the output of the filter 26b. By obtaining the correlation coefficient and the like in the comparison unit 27, the correlation between the output of the inertial sensor and the output of the motion vector detection unit 24 is obtained.

In step S250, the gain of the output correction unit 32 is determined based on the correlation obtained by the comparison unit 27. Specifically, the turning radius estimator 30 shown in FIG. 3 is operated to obtain the turning radius, and the correction gain calculator 32a and the correction gain adjuster 32b shown in FIG. 4 are operated to obtain gains G1 to G6. Ask. The gain set here is used in step S160, and the output of the target generator 33 in the shooting operation is appropriately adjusted (correction amount acquisition).

In step S260, the process returns to the original step S140.

As described above, according to the present embodiment, it is possible to perform appropriate shift blur correction in response to individual differences, posture differences, etc. of the user, and as a result, it is possible to provide a high-quality image. ..

(Modification example)
In the above-described embodiment, an image pickup device including an image blur correction mechanism for moving the image sensor in a direction perpendicular to the optical axis as an image blur correction means has been described. However, the image blur correction means is not limited to this. For example, in the case of image shake correction during moving image shooting or live view display, image shake correction may be performed using so-called electronic image stabilization in which the position of the region for reading the image signal from the image sensor is changed for each frame. Further, the image blur may be corrected by driving the lens constituting the photographing optical system.

(Other embodiments)
The present invention also supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads the program. It can also be realized by the processing to be executed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

1: Camera body 2: Lens 3: Shooting optical system 5: Camera system control circuit, 6: Image pickup element unit, 12: Lens system control circuit, 13: Lens drive unit, 14: Shake correction drive unit, 15: Runout detection unit, 21: Image pickup element position detection unit

Claims (13)

A motion vector detecting means for detecting a motion vector from two images acquired by an image sensor at regular time intervals, and a motion vector detecting means.
An acquisition means for acquiring a runout signal from a runout detection means for detecting the runout of an image pickup apparatus, and an acquisition means for acquiring a runout signal.
A comparison means for obtaining the correlation between the motion vector and the runout signal, and
A correction amount acquisition means for acquiring an image blur correction amount based on the result of comparison by the comparison means and the runout signal, and
An image blur correction device characterized by being equipped with. The image blur correction device according to claim 1, wherein the runout signal is a rotational runout signal indicating a rotational runout of the image pickup apparatus, and the comparison means obtains a correlation between the motion vector and the rotational runout signal. .. The acquisition means further acquires a signal indicating the first translational runout from the translational runout detecting means for detecting the translational runout of the image pickup apparatus, and the image blur correction device further acquires the rotational runout signal and the first translation. The image blur correction device according to claim 2, further comprising an estimation means for estimating the radius of gyration of the image pickup device based on a signal indicating runout. The calculation means for calculating the second translational runout of the image pickup apparatus based on the radius of gyration estimated by the estimation means is further provided, and the correction amount acquisition means has the second translation based on the result of the comparison. The image blur correction device according to claim 3, wherein the ratio for correcting the runout is changed. When the comparison means determines that the motion vector and the runout signal have a high correlation, the correction amount acquisition means corrects the second translational runout as compared with the case where the correlation is determined to be low. The image blur correction device according to claim 4, wherein the ratio is increased. The method according to any one of claims 1 to 5, wherein the comparison means obtains the correlation between the motion vector and the runout signal by obtaining the correlation coefficient between the motion vector and the runout signal. Image blur correction device. The comparison means obtains the correlation between the motion vector and the runout signal by performing principal component analysis of the motion vector and the runout signal and obtaining the ratio of the variances of the first principal component and the second principal component. The image blur correction device according to any one of claims 1 to 5, which is characterized. The image blur correction according to any one of claims 1 to 7, wherein the comparison means compares a plurality of inertial sensors of the runout detecting means with components of the motion vector in a plurality of directions. apparatus. The image blur correction device according to any one of claims 1 to 8, further comprising a control means for controlling image blur correction by the image blur correction means based on the image blur correction amount. With the image sensor
The image blur correction device according to claim 9,
The image blur correction means controlled by the control means is provided.
The image blur correction means is an image pickup apparatus characterized in that the image pickup element is moved in a direction perpendicular to the optical axis. With the image sensor
The image blur correction device according to claim 9,
The image blur correction means controlled by the control means is provided.
The image blur correction means is an image pickup apparatus characterized in that the position of a region for reading an image signal from the image pickup element is changed. A motion vector detection process that detects a motion vector from two images acquired by an image sensor at regular time intervals, and a motion vector detection process.
The acquisition process of acquiring the runout signal from the runout detection means for detecting the runout of the image pickup device, and
A comparison step for obtaining the correlation between the motion vector and the runout signal, and
A correction amount acquisition step of acquiring an image blur correction amount based on the comparison result in the comparison step and the runout signal, and a correction amount acquisition step.
A method for controlling an image blur correction device, which comprises. A program for causing a computer to execute each step of the control method according to claim 12.