JP6053422B2 - Imaging apparatus and imaging method - Google Patents

Description

  The present invention relates to a technique for correcting image shake in an imaging apparatus.

  2. Description of the Related Art In recent years, techniques for correcting image blur caused by shake of an image pickup apparatus have advanced. Along with this, the image pickup device rotates around the optical axis, thereby rotating the picked-up image, and image shake that corrects image shake or the like that the picked-up image is distorted in a trapezoid shape when the image pickup device is tilted with respect to the subject. Correction functions are becoming popular.

  As a method for correcting the image blur caused by the rotation of the captured image, for example, there is a method for correcting the image blur by mechanically rotating the image sensor as disclosed in Patent Document 1. Further, for example, as a method of correcting image blur that rotates a captured image and image blur that is distorted into a trapezoidal shape, the image deformation amount of the captured image is calculated as in Patent Document 2, and the image is corrected so as to cancel the image deformation amount. A method of deforming is known.

JP 2006-71743 A JP 2011-146260 A

  However, the conventional example has the following problems.

  When the image shake is corrected by detecting the shake of the imaging device from the output of the angular velocity sensor, the shake around the axis (YAW and PITCH directions) on the plane perpendicular to the optical axis of the imaging optical system, and the optical axis Is detected by the angular velocity sensor. At this time, the angular velocity sensor is subject to assembly errors when mounted on the electronic circuit board. Due to this error, the output of the angular velocity sensor in the ROLL or PITCH direction also fluctuates when a large shake such as panning the imaging device in the YAW direction occurs.

  In this case, when the image is rotated (ROLL direction) or trapezoidal distortion (tilt) in the vertical direction (PITCH direction) occurs when panning in the YAW direction, the image is very conspicuous and uncomfortable for the user. There was a possibility of becoming an unnatural image.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to minimize the adverse effect on the other axis caused by the deviation of the detection axis of the angular velocity sensor and to obtain a good image blur correction effect. An imaging device is provided.

An imaging apparatus according to the present invention includes a shake around a first axis perpendicular to an optical axis, a shake around a second axis perpendicular to the first axis and the optical axis, and a shake around the optical axis. Based on the information about the shake about the first axis and the information about the shake about the second axis detected by the shake detection means for detecting the shake, the correction amount of the image shake due to the shake in the translation direction of the captured image is calculated. Translation correction amount calculation means for calculating, and rotation correction amount calculation for calculating a correction amount of image shake due to shake around the optical axis of the captured image based on information about shake around the optical axis detected by the shake detection means and means, based on the correction amount of the image blur is more operation on the translational compensation amount calculating means and the prior SL rotational compensation amount calculating means, first control means for controlling the image blur correcting means for correcting an image blur, Time when the shake information about the first axis is larger than a predetermined threshold When it has been determined that it has been panned or tilted when it has continued for a predetermined time or more, and when it has been determined that it has been panned or tilted, it has not been determined that it has been panned or tilted And a second control means for reducing a correction amount of image blur around the optical axis of the captured image .
In addition, the imaging apparatus according to the present invention includes a shake around a first axis perpendicular to the optical axis, a shake around a second axis perpendicular to the first axis and the optical axis, and the optical axis. Correction of image blur due to shake in the translational direction of the captured image based on information on shake around the first axis and information on shake around the second axis detected by the shake detection means for detecting around shake Translation correction amount calculating means for calculating the amount, and rotation correction for calculating a correction amount of the image shake due to the shake around the optical axis of the captured image based on the information about the shake around the optical axis detected by the shake detection means A first control unit that controls an image blur correction unit that corrects an image blur based on an amount calculation unit, and an image blur correction amount calculated by the translation correction amount calculation unit and the rotation correction amount calculation unit; Limiting the amount of image blur correction around the optical axis of the captured image Two control means, wherein the second control means is arranged around the first axis when the information about the shake about the first axis continues for a predetermined time or longer than the predetermined threshold. The high-pass filter of the rotation correction amount calculation means is used to reduce the amount of correction of image blur around the optical axis of the captured image, compared to the case where the time when the shake information is greater than the predetermined threshold does not continue for a predetermined time. It is characterized by increasing the cut-off frequency.
In addition, the imaging apparatus according to the present invention includes a shake around a first axis perpendicular to the optical axis, a shake around a second axis perpendicular to the first axis and the optical axis, and the optical axis. Correction of image blur due to shake in the translational direction of the captured image based on information on shake around the first axis and information on shake around the second axis detected by the shake detection means for detecting around shake Translation correction amount calculating means for calculating the amount, and rotation correction for calculating a correction amount of the image shake due to the shake around the optical axis of the captured image based on the information about the shake around the optical axis detected by the shake detection means. A first control unit that controls an image blur correction unit that corrects an image blur based on an amount calculation unit, and an image blur correction amount calculated by the translation correction amount calculation unit and the rotation correction amount calculation unit; Limiting the amount of image blur correction around the optical axis of the captured image Two control means, wherein the second control means is arranged around the first axis when the information about the shake about the first axis continues for a predetermined time or longer than the predetermined threshold. In order to reduce the correction amount of the image blur around the optical axis of the captured image compared to the case where the time when the shake information is longer than the predetermined threshold does not continue for a predetermined time or longer, the integration unit in the rotation correction amount calculation unit The time constant of is shortened.
In addition, the imaging apparatus according to the present invention includes a shake around a first axis perpendicular to the optical axis, a shake around a second axis perpendicular to the first axis and the optical axis, and the optical axis. Correction of image blur due to shake in the translational direction of the captured image based on information on shake around the first axis and information on shake around the second axis detected by the shake detection means for detecting around shake Translation correction amount calculating means for calculating the amount, and rotation correction for calculating a correction amount of the image shake due to the shake around the optical axis of the captured image based on the information about the shake around the optical axis detected by the shake detection means. A first control unit that controls an image blur correction unit that corrects an image blur based on an amount calculation unit, and an image blur correction amount calculated by the translation correction amount calculation unit and the rotation correction amount calculation unit; Limiting the amount of image blur correction around the optical axis of the captured image Two control means, wherein the second control means is arranged around the first axis when the information about the shake about the first axis continues for a predetermined time or longer than the predetermined threshold. Compared to the case where the time when the shake information is longer than the predetermined threshold does not continue for a predetermined time or more, the calculated value of the rotation correction amount calculating means is used to reduce the correction amount of the image shake around the optical axis of the captured image. It is characterized by reducing the limit of.

  According to the present invention, it is possible to provide an imaging apparatus that can minimize the degree of adverse effects on other axes caused by the deviation of the detection axis of the angular velocity sensor and obtain a good image blur correction effect.

1 is a block diagram illustrating a configuration example of a video camera as an example of an imaging apparatus according to an embodiment of the present invention. The figure explaining a pinhole camera model. The block diagram which shows the structure of the image deformation amount calculating part which concerns on 1st and 2nd embodiment. FIG. 5 is a view for explaining rotation correction in an image deforming unit according to the first embodiment. The flowchart which shows an example of the process of the rotation control change part which concerns on 1st Embodiment, and the process of the tilt control change part which concerns on 3rd Embodiment. The graph for demonstrating the process of the rotation control change part which concerns on 1st Embodiment, and the process of the tilt control change part which concerns on 3rd Embodiment. 10 is a flowchart illustrating an example of processing of a photographer movement determination unit according to the second and fourth embodiments. The graph for demonstrating the process of the photographer movement determination part 301 which concerns on 2nd and 4th embodiment. The block diagram which shows the structure of the image deformation amount calculating part which concerns on 3rd and 4th embodiment. FIG. 10 is a view for explaining tilt correction in an image deforming unit according to a third embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(First embodiment)
FIG. 1 is a block diagram showing a configuration of a digital video camera as an example of an imaging apparatus provided with an image shake correction apparatus according to the first embodiment of the present invention. Hereinafter, each component of the imaging apparatus 100 in FIG. 1 and an example of the operation thereof will be described in detail.

  The angular velocity sensor 102 detects a shake of the imaging apparatus 100 (a shake around two axes on a plane perpendicular to the photographing optical axis and a shake around the optical axis) as an angular velocity signal, and the angular velocity signal is converted into an A / D converter 103. To supply. If the two axes at this time are two axes (X axis and Y axis) perpendicular to the optical axis (Z axis) of the imaging optical system, the angular velocity detection directions in this case are the YAW and PITCH directions, The direction around the axis is the ROLL direction. The A / D converter 103 digitizes each angular velocity signal from the angular velocity sensor 102 and supplies it to the image deformation amount calculation unit 200 inside the μCOM 101 as angular velocity data.

  The imaging optical system 106 performs operations such as zooming and focusing, and forms a subject image on the imaging element 107. The zoom encoder 104 detects the position (zoom position) of the variable magnification optical system 105 in the imaging optical system 106 and outputs it to the image deformation amount calculation unit 200 inside the μCOM 101.

  The image deformation amount calculation unit 200 calculates an image deformation amount for correcting shake of the captured image using the angular velocity data and the output of the zoom encoder 104, and sets the calculated image deformation amount in the image deformation unit 110. . Details of the processing of the image deformation amount calculation unit 200 will be described later.

  The imaging element 107 converts the subject image formed by the imaging optical system 106 into an electrical signal as a captured image signal, and supplies the electrical signal to the signal processing unit 108. The signal processing unit 108 generates a video signal (video signal) compliant with, for example, the NTSC format from the signal obtained by the image sensor 107 and supplies it to the image memory 109.

  The image deformation unit 110 corrects the shake of the captured image by deforming the image stored in the image memory 109 based on the image deformation amount calculated by the image deformation amount calculation unit 200, and the recording control unit 111 and the display Output to the control unit 113. The display control unit 113 outputs the video signal supplied from the image deformation unit 110 and causes the display device 114 to display an image. The display control unit 113 drives the display device 114, and the display device 114 displays an image by a liquid crystal display element (LCD) or the like. Further, the recording control unit 111 outputs the video signal supplied from the image transformation unit 110 to the recording medium 112 when the recording of the video signal is instructed by an operation unit (not shown) used for instructing the start or end of recording. Let me record. The recording medium 112 is an information recording medium such as a semiconductor memory or a magnetic recording medium such as a hard disk.

  The image transformation unit 110 performs image transformation using geometric transformation such as projective transformation. Specifically, the pixel coordinates in the untransformed image (image stored in the image memory 109) are (X0, Y0) (however, the center of the captured image corresponding to the optical axis of the imaging optical system 106 is the origin). ), The pixel coordinates in the transformed image (the output image of the image transformation unit 110) are represented as (X1, Y1) and expressed in homogeneous coordinates, and can be described as (Equation 1).

  The left side and the right side of (Formula 1) show an equivalence relationship (the meaning does not change even if an arbitrary magnification is applied to the left side or the right side), and in the normal equal sign, (Formula 2) (Formula 3).

  In (Expression 1), a 3 × 3 matrix is generally called a projective transformation matrix, and the elements h1 to h8 of the matrix are set by the image deformation amount calculation unit 200. In the following description, the image transformation of the image transformation unit 110 is described as using projective transformation, but any transformation method such as affine transformation may be used.

  Next, details of processing performed by the image deformation amount calculation unit 200 will be described. The image deformation amount calculation unit 200 uses the shake angle of the imaging device calculated from the output of the angular velocity sensor 102 and the focal length of the imaging optical system 106 calculated from the zoom encoder 104 to perform image deformation of the image deformation unit 110. Calculate the amount. Specifically, the projective transformation matrix of (Expression 1) is calculated.

  Here, a method for calculating the projective transformation matrix using the shake angle and the focal length of the imaging optical system 106 will be described below. FIG. 2A illustrates the projection of the subject on the imaging surface by the imaging apparatus using a pinhole camera model. In FIG. 2A, the origin (0, 0, 0) of the XYZ space coordinates is the pinhole position in the pinhole camera model. If the imaging surface is arranged behind the pinhole position, the image projected on the imaging surface is inverted, so that the image is virtually pinned in FIG. The imaging surface I is arranged before the hole position.

  The distance in the Z-axis direction between the origin (0, 0, 0) of the XYZ space coordinates and the imaging surface I is the focal length f. Coordinates on the imaging plane I are defined as uv plane coordinates, and the origin (0, 0) of the uv plane coordinates is assumed to coincide with (0, 0, f) in the XYZ space coordinates. The coordinates P (u, v) on the uv plane coordinates are coordinates when the subject A (X, Y, Z) on the XYZ space coordinates is projected onto the imaging plane I. At this time, the coordinate P can be expressed by (Formula 4).

  (Expression 4) can be expressed by (Expression 5) when homogeneous coordinates are used.

  Since the fourth column element of the 3 × 4 matrix of (Expression 5) remains 0 in this embodiment, (Expression 5) is the same as (Expression 6).

  FIG. 2B shows the pinhole camera model shown in FIG. In FIG. 2B, coordinates obtained by rotating the XYZ space coordinates in FIG. 2A by R are defined as X′Y′Z ′ space coordinates. It is assumed that the origin (0, 0, 0) of the X′Y′Z ′ space coordinates coincides with the XYZ space coordinates. That is, FIG. 2B is a simplified representation of a state in which a rotational shake R occurs in the imaging apparatus and no shift shake occurs using a pinhole camera model.

  In the pinhole camera model of FIG. 2B, the imaging plane I ′ is arranged at a focal distance f at a distance from the origin (0, 0, 0), as in FIG. 2A. The coordinates on the imaging plane I ′ are defined as u′v ′ plane coordinates, and the origin (0,0) of the u′v ′ plane coordinates is (0,0, f) in the X′Y′Z ′ space coordinates. It shall be consistent with The coordinates P ′ (u ′, v ′) on the u′v ′ plane coordinates are the same as the subject A ′ (X ′, Y ′, Z ′) on the X′Y′Z ′ space coordinates on the imaging plane I ′. The coordinates when projected. Note that the position of the subject A in FIG. 2A and the subject A ′ in FIG. 2B in the world coordinate system is the same position (that is, the subject has not moved). At this time, the coordinate P ′ can be expressed by (Expression 7) similarly to (Expression 6) when homogeneous coordinates are used.

  Further, since the positions of the subject A and the subject A ′ in the world coordinate system are the same, the relationship between the coordinates of both can be expressed by (Equation 8).

  Furthermore, when (Expression 6) and (Expression 7) are modified and substituted into (Expression 8), (Expression 9) can be derived.

  (Equation 9) shows the correspondence of the position of the subject image on the imaging surface before and after the pinhole camera rotates R times. That is, when an R rotation shake is applied to the imaging apparatus, the equation indicates where the pixel on the imaging surface moves from where. Therefore, in order to perform image blur correction, conversion for returning the pixel movement amount when the imaging apparatus shakes may be performed. That is, according to (Equation 10), it is only necessary to perform conversion to restore the pixel position that has moved due to the addition of the R rotation shake to the imaging apparatus.

  Therefore, assuming that the shake of the image pickup apparatus 100 in FIG. 1 is R, the focal length of the image pickup optical system 106 is f, and the projective transformation matrix for performing image shake correction is H, H becomes (Expression 11).

  If the angular shake amount in the YAW direction of the imaging apparatus is θy, the angular shake amount in the PITCH direction is θp, and the angular shake amount in the ROLL direction is θr, R can be expressed by (Equation 12).

  H in (Expression 11) is decomposed into deformation components of translation t →, scaling s (constant), rotation r (matrix), shear k (matrix), and tilt ν → by using (Expression 13). be able to.

tx… Horizontal translation amount ty… Vertical translation amount θ… Rotation angle vx… Horizontal tilt amount vy… Vertical tilt amount α… Shear anisotropic magnification φ… Shear direction angle However, t → and ν → , Respectively, in the formula

Represents.

  From (Equation 11), (Equation 12), and (Equation 13), when equations for each deformation component are solved, (Equation 14) to (Equation 21) are obtained.

  Here, when the shake angle of the image pickup device is γ, when approximated as cos γ = 1, sin γ tan γ = 0, sin γ sin γ = 0, (Expression 14) to (Expression 21) are expressed as (Expression 22) to (Expression 29). Can be approximated by

  Hereinafter, the components of the image deformation amount calculation unit 200 and the operation of an example thereof will be described in detail with reference to the block diagram of FIG. In the present embodiment, among the deformation components of the image deformation performed by the image deformation unit 110, the tilt, shear, and enlargement / reduction components are not shown in the configuration of FIG. ) Or (Equation 25) to (Equation 29).

  In the block diagram of FIG. 3, blocks 201 to 204 are blocks for calculating a correction amount for correcting translational image blur (image blur in the horizontal and vertical directions of the image). Blocks 211 to 213 are blocks for calculating a correction amount for correcting image rotation.

  First, blocks 201 to 204 for calculating the translation blur correction amount will be described. Among the outputs from the A / D converter 103 described above, HPF (high pass filter) 201 is supplied with angular velocity data in the YAW direction or the PITCH direction. The HPF 201 has a function of changing its characteristics in an arbitrary frequency band, and outputs a signal in a high frequency band by blocking low frequency components included in the angular velocity data.

  The integrator 202 has a function capable of changing its characteristics in an arbitrary frequency band, integrates the output from the HPF 201, and supplies the integrated output to the focal length multiplication unit 203. The focal length calculation unit 231 calculates the focal length of the imaging optical system 106 from the output of the zoom encoder 104 described above, and calculates the focal length used for the calculation of the focal length multiplication unit 203. The focal length multiplication unit 203 multiplies the output of the integrator 202 by the focal length f calculated by the focal length calculation unit 231 and supplies the result to the saturation prevention control unit 204.

  The saturation prevention control unit 204 performs control to limit the output from the focal length multiplication unit 203 so that the output from the focal length multiplication unit 203 does not exceed a predetermined value (hereinafter referred to as a limiter). Further, the saturation prevention control unit 204 changes the cutoff frequency of the HPF 201 to a high frequency side or shortens the time constant of the integrator 202 when the output from the focal length multiplication unit 203 approaches the limiter. Take control. The output of the saturation prevention control unit 204 becomes a final translation correction amount (translation correction amount calculation) and is supplied to the image deformation amount synthesis unit 230.

  Next, blocks 211 to 213 for calculating a rotational shake correction amount (rotational correction amount calculation) will be described. Among the outputs from the A / D converter 103 described above, the angular velocity data in the ROLL direction is supplied to the HPF 211. The HPF 211 has a function of changing its characteristics in an arbitrary frequency band, and blocks a low frequency component included in the angular velocity data and outputs a signal in a high frequency band.

  The integrator 212 has a function of changing its characteristics in an arbitrary frequency band, integrates the output from the HPF 211, and supplies the integrated output to the saturation prevention control unit 213. The saturation prevention control unit 213 performs control to limit the output from the integrator 212 so that the output from the integrator 212 does not exceed a predetermined limiter. Further, when the output from the integrator 212 approaches the limiter, the saturation prevention control unit 213 changes the cutoff frequency of the HPF 211 to the high frequency side, shortens the time constant of the integrator 212, or the like. Take control. The output of the saturation prevention control unit 213 becomes a final rotation correction amount and is supplied to the image deformation amount synthesis unit 230.

  The image deformation amount synthesizing unit 230 performs an operation for synthesizing the correction amounts such as translation, rotation, and other not-illustrated tilts output from the saturation prevention control units 204 and 213. Specifically, the projective transformation matrix of (Expression 1) is calculated according to (Expression 13). The image deformation amount synthesis unit 230 outputs the calculated values of the elements of the projective transformation matrix to the image deformation unit 110. The image deforming unit 110 performs image blur correction by image deformation based on the output from the image deformation amount combining unit 230.

  4A, 4B, and 4C are diagrams for explaining the rotation correction in the image deformation unit 110. FIG. FIG. 4A shows a captured image when there is no shake in the imaging apparatus. FIG. 4B shows a captured image when a shake in the ROLL direction has occurred in the imaging apparatus. The image deforming unit 110 uses the output from the image deformation amount combining unit 230 based on the output from FIG. The image reading range from the image memory 109 is rotated as indicated by the dotted line portion. FIG. 4C shows an output image output by rotating the image reading range as indicated by the dotted line portion in FIG. 4B, and the subject imaged at an inclination in FIG. The shape has been corrected. When the shake in the ROLL direction is correctly detected, the rotation of the image can be corrected as shown in FIG.

  The rotation control changing unit 300 changes the control of the HPF 211, the integrator 212, and the saturation prevention control unit 213 based on the output of the A / D converter 103 in the YAW direction or the PITCH direction. Hereinafter, the processing of the rotation control change unit 300 will be described with reference to FIGS. 5 and 6.

  The flowchart of FIG. 5 shows an example of processing of the rotation control change unit 300. The process of the flowchart of FIG. 5 is repeatedly executed at a predetermined cycle such as 60 Hz.

  First, in S100, if the variable SENSOR_OUT1 is a signal generated based on the output of the A / D converter 103 in the YAW direction or the PITCH direction, it is determined whether or not SENSOR_OUT1 is larger than the threshold AMP_TH1. The signal generated based on the output of the A / D converter 103 in the YAW direction or PITCH direction may be the output of the A / D converter 103 itself, or the output of the A / D converter 103 may be Thus, it may be a signal that has been subjected to various filtering processes or gained. That is, any signal may be used as long as the signal is generated based on the output of the A / D converter 103. If it is determined in S100 that SENSOR_OUT1 is greater than AMP_TH1, the process proceeds to S101.

  In S101, the value of the counter COUNTER1 is incremented, and the process proceeds to S102. In S102, it is determined whether or not the counter COUNTER1 is larger than the threshold TIME_TH1. If it is determined in S102 that COUNTER1 is equal to or less than TIME_TH1, the processing in FIG. 5 ends. If it is determined in S102 that COUNTER1 is greater than TIME_TH1, the process proceeds to S103. The process of S103 is performed when a shake of a predetermined time (TIME_TH1) or more and a predetermined magnitude (AMP_TH1) or more is applied in the YAW direction or PITCH direction, that is, the panning or tilting operation is performed on the imaging apparatus 100. It will be when it is done.

  In S103, the rotation correction amount (correction amount in the ROLL direction) is controlled to be small. Controls such as limiting the rotation correction amount are performed by increasing the cut-off frequency of the HPF 211, shortening the time constant of the integrator 212, and narrowing the limiter of the saturation prevention control unit 213. The three types of control of the HPF 211, the integrator 212, and the saturation prevention control unit 213 may be performed by any one of them, or may be performed by combining a plurality of types of control. In addition to the above three types of control, other methods may be used as long as the control is performed so that the rotation correction amount is reduced. For example, it may be configured to forcibly input a signal such that the rotation correction amount approaches zero. Further, when the input value to the saturation prevention control unit 213 exceeds a certain threshold value, the cutoff frequency of the HPF 211 is increased, the time constant of the integrator 212 is shortened, and the rotation is reduced by decreasing the threshold value. The correction amount may be limited. After the process of S103, the process of FIG. If it is determined in S100 that SENSOR_OUT1 is equal to or less than AMP_TH1, the process proceeds to S104. In S104, the value of the counter COUNTER1 incremented in S101 is cleared, and the process proceeds to S105.

  In S105, processing is performed to restore the cutoff frequency of the HPF 211 changed in S103, the time constant of the integrator 212, or the limiter of the saturation prevention control unit 213. In addition, when control for limiting the amount of rotation correction is performed, it is canceled. After S105, the process in FIG. 5 ends.

  FIG. 6 is a graph for explaining the effect on the calculation of the rotation correction amount when the rotation control change unit 300 performs the process of FIG. In FIG. 6, a fixed subject (non-moving subject) is photographed during a period from time 0 to T0, and then a large movement such as panning is performed from time T0 to T2 to the imaging apparatus, and then the image is fixed again. It shows changes in various signals when returning to shooting of the subject.

  FIG. 6A is a graph showing a change over time of SENSOR_OUT1, which is a signal generated based on the output of the A / D converter 103 in the YAW direction or the PITCH direction. SENSOR_OUT1 indicates a state in which the output increases during T0 to T2, which is a time zone in which a large movement is applied to the imaging apparatus, and camera shake in the YAW direction or the PITCH direction is detected at other times.

  FIG. 6B is a graph showing a change of SENSOR_OUT2 with time, assuming that the output in the ROLL direction of the A / D converter 103 is SENSOR_OUT2. FIG. 6B shows the detection of the angular velocity sensor in spite of no movement in the ROLL direction during the time T0 to T2 when a large movement is applied to the imaging apparatus in the YAW direction or the PITCH direction. It shows a state in which an output is generated due to the deviation when there is a deviation of the axis. In the time other than T0 to T2, only the shake in the ROLL direction is detected.

  FIG. 6C is a graph showing the change of CALC_OUT1 over time, assuming that the output signal of the saturation prevention control unit 213 is CALC_OUT1 when it is assumed that the processing of the flowchart of FIG. It is. FIG. 6C is a graph of FIG. 6B, in which the output generated due to the deviation of the detection axis of the angular velocity sensor between the times T0 and T2 is not completely attenuated by the time constants of the HPF 211 and the integrator 212. It shows how it is integrated.

  FIGS. 4D and 4E are diagrams for explaining an image when it is assumed that the rotation is corrected by the image deforming unit 110 based on the signal of FIG. 6C. In FIG. 4D, although the image pickup apparatus is not shaken in the ROLL direction, the picked-up image is represented by the dotted line portion in FIG. 4D based on the signal in FIG. An image reading range from the image memory 109 is rotated. FIG. 4E shows an output image output by rotating the image reading range as indicated by the dotted line in FIG. 4D, and the subject that was not inclined in FIG. ing.

  FIG. 6D is a graph showing the change of CALC_OUT2 with time, assuming that the output signal of the saturation prevention control unit 213 when the processing of the flowchart of FIG. 5 is performed by the rotation control changing unit 300 is CALC_OUT2. In FIG. 6A, the determination of S100 in FIG. 5 is Yes at the timing of time T0 when the value of SENSOR_OUT1 becomes larger than AMP_TH1. Further, the determination in S102 is Yes at the timing of time T1 when COUNTER1 becomes larger than TIME_TH1. Therefore, the process of S103 is performed in the period from time T1 to time T2 when the determination of S100 changes to No. In the process of S103, when the cutoff frequency of the HPF 211 is increased, the attenuation rate in the low frequency band is increased, so that the output generated due to the shift of the detection axis in FIG. 6B can be attenuated. Further, when the time constant of the integrator 212 is shortened, the speed at which the output of the integrator 212 converges to zero increases, so that the output generated by the shift of the detection axis in FIG. 6B can be attenuated. When the limiter of the saturation prevention control unit 213 is narrowed, the time for the output of the integrator 212 to reach the limiter is increased, the cutoff frequency of the HPF 211 is likely to be increased, and the time constant of the integrator 212 is likely to be shortened. As a result, the output generated by the shift of the detection axis in FIG. 6B can be attenuated. As a result, the output of the saturation prevention control unit 213 that has continued to increase from time T0 in FIG. 6C can be attenuated as shown in FIG. 6D. Then, rotation correction is performed by the image deforming unit 110 based on the signal of FIG. 6D, so that the imaging device does not move in the ROLL direction as shown in FIG. 4E. It is possible to prevent a phenomenon in which image rotation correction is performed.

  As described above, in the first embodiment of the present invention, when a large movement such as panning occurs in the YAW or PITCH direction with respect to the imaging apparatus 100, the correction is based on the output of the angular velocity sensor in the ROLL direction. It was decided to limit the amount calculation. As a result, it is possible to minimize the influence of the signal in the ROLL direction caused by the deviation of the detection axis of the angular velocity sensor and obtain a good image blur correction effect.

  In the present embodiment, the rotation correction method has been described by taking the image deformation as an example. However, a method of mechanically rotating the image sensor may be used.

(Second Embodiment)
An imaging apparatus according to the second embodiment of the present invention will be described below. In the present embodiment, the configuration of the imaging apparatus is the same as that in FIG. 1, and the internal configuration of the image deformation amount calculation unit 200 is different. Therefore, the description of FIG. Further, the image deformation amount calculation unit 200 in the present embodiment is different from the first embodiment in that the photographer movement determination unit indicated by a broken line in FIG. 3 is added to the configuration of the first embodiment shown in FIG. 301 is added.

  The photographer movement determination unit 301 captures an image while the imaging device 100 moves such as the photographer walks from a signal generated based on the output of the A / D converter 103 in the YAW direction, the PITCH direction, or the ROLL direction. Judge whether or not you are doing. The photographer movement determination unit 301 changes the processing performed by the rotation control change unit 300 when it is determined that the photographer is shooting while moving.

  The flowchart in FIG. 7 shows an example of processing of the photographer movement determination unit 301. The process of the flowchart of FIG. 7 is repeatedly executed at a predetermined cycle such as 60 Hz.

  As an overview of the processing in FIG. 7, the image generated when the signal generated based on the output of the A / D converter 103 swings a predetermined number of times in the positive and negative directions exceeding a predetermined amplitude during a predetermined period. It is determined that the person is moving, and the processing of the rotation control change unit 300 is changed. In other words, the photographer movement determination unit 301 determines that the photographer is moving when it is determined that the change in the signal based on the output of the angular velocity sensor 102 is greater than the predetermined frequency and amplitude. The processing of the rotation control change unit 300 is changed. Hereinafter, the process of FIG. 7 will be described.

  First, in S200, the value of the counter COUNTER2 is incremented, and the process proceeds to S201. In S201, it is determined whether or not the flag SIGN_FLAG is 0. In S201, when SIGN_FLAG is 0, the process proceeds to S202, and when SIGN_FLAG is 1, the process proceeds to S206.

  In S202, if the variable SENSOR_OUT3 is a signal generated based on the output of the A / D converter 103 in the YAW direction, the PITCH direction, or the ROLL direction, it is determined whether SENSOR_OUT3 is larger than the threshold AMP_TH2. The signal generated based on the output of the A / D converter 103 in the YAW direction, the PITCH direction, or the ROLL direction may be the output of the A / D converter 103 or the A / D converter 103. The output may be a signal obtained by performing various filtering processes or applying gain or the like. That is, any signal may be used as long as the signal is generated based on the output of the A / D converter 103. If it is determined in S202 that SENSOR_OUT3 is greater than AMP_TH2, the process proceeds to S203. If it is determined that SENSOR_OUT3 is equal to or less than AMP_TH2, the process proceeds to S210.

  In S203, the variable OVER_COUNT for counting the number of times that the amplitude of SENSOR_OUT2 exceeds the threshold AMP_TH2 is incremented, and the process proceeds to S204. In S204, the value of the flag SIGN_FLAG is changed to 1, and when the processing of FIG. 7 is performed next, it is determined as No in S201. In S205, the value of the counter COUNTER2 incremented in S200 is cleared, and the process proceeds to S210.

  In S206, it is determined whether or not the SENSOR_OUT3 is smaller than a threshold value (0-AMP_TH2). If it is determined in S206 that SENSOR_OUT3 is smaller than (0-AMP_TH2), the process proceeds to S207. If it is determined that SENSOR_OUT3 is equal to or greater than (0-AMP_TH2), the process proceeds to S210.

  In S207, the variable OVER_COUNT is incremented as in S203, and the process proceeds to S208. In S208, the value of the flag SIGN_FLAG is changed to 0, and when the processing in FIG. 7 is performed next, it is determined again as Yes in S201. In S209, the value of the counter COUNTER2 incremented in S200 is cleared as in S205, and the process proceeds to S210.

  In S210, it is determined whether or not the counter COUNTER2 incremented in S200 is greater than or equal to the threshold TIME_TH2. If it is determined in S210 that COUNTER2 is greater than or equal to TIME_TH2, the process proceeds to S214, and if it is determined that COUNTER2 is smaller than TIME_TH2, the process proceeds to S211. In S211, it is determined whether or not the value of OVER_COUNT is equal to or greater than a threshold value COUNT_TH. In S211, if OVER_COUNT is greater than or equal to COUNT_TH, the process proceeds to S212, and if the value of OVER_COUNT is smaller than COUNT_TH, the process proceeds to S216. In S212, the value of OVER_COUNT is limited to COUNT_TH so that OVER_COUNT does not overflow, and the process proceeds to S213.

  The process of S213 is performed when the value of OVER_COUNT is incremented in S203 and S207 and reaches COUNT_TH before COUNTER2 reaches TIME_TH2. That is, when the output of SENSOR_OUT3 fluctuates more than OVER_COUNT times with an amplitude greater than AMP_TH2 in the plus or minus direction (when the output fluctuates more than a predetermined number of times with an amplitude greater than a threshold within a predetermined period). At this time, the imaging apparatus determines that the photographer is shooting while moving (for example, walking).

  In S213, the process of S103 in FIG. 5 performed by the rotation control change unit 300 is prohibited or limited. Prohibiting the process of S103 means returning the cut-off frequency change of the HPF 211, the time constant change of the integrator 212, and the limiter change of the saturation prevention control unit 213 performed in S103. In addition, if control for limiting the correction amount of rotation is being performed, it is canceled. Limiting the processing of S103 means reducing the amount of increasing the cutoff frequency of the HPF 211, the amount of shortening the time constant of the integrator 212, and the amount of narrowing the limiter of the saturation prevention control 213. Further, for example, when the cut-off frequency of the HPF 211 is changed and the time constant of the integrator 212 is changed, some of the restrictions on the plurality of rotation correction amounts in S103 are prohibited such that one of them is not changed. It may be configured to do. After S213, the process in FIG. 7 ends.

  The process of S214 is performed when COUNTER2 reaches TIME_TH2 without being cleared in S205 or S209. That is, when the output of SENSOR_OUT3 continues to change with an amplitude of AMP_TH2 or less. At this time, the imaging apparatus determines that the photographer is photographing in a state where the photographer is not moving (a state where the photographer stops or sits down). In S214, the value of COUNTER2 is limited to TIME_TH2 so that COUNTER2 does not overflow, and the process proceeds to S215. In S215, the value of OVER_COUNT is cleared, and the process proceeds to S216.

  The process of S216 is executed when OVER_COUNT has not reached COUNT_TH, which is a threshold value for determining whether or not the photographer is still moving while taking the image through S214 and S215. In S216, the execution of the process in S103 of FIG. 5 performed by the rotation control change unit 300 is permitted. That is, the process of prohibiting or limiting the process of S103 in S213 is canceled, and the rotation control change unit 300 returns to the state of operating according to the flowchart of FIG. After S216, the process in FIG. 7 ends.

  FIG. 8 is a graph for explaining the processing of the photographer movement determination unit 301. FIG. 8 shows changes in various signals when a fixed subject is shot while the photographer is stationary during the period from time 0 to T10, and when the photographer is shooting while walking from time T10. .

  FIG. 8A is a graph showing a change over time of SENSOR_OUT3 which is a signal generated based on the output of the A / D converter 103 in the YAW direction, the PITCH direction, or the ROLL direction. SENSOR_OUT3 shows a state in which the output increases due to vibration during walking from T10, and relatively small hand shake at the time before T10 is detected.

  In the flowchart of FIG. 7, when COUNT_TH = 2, the photographer movement determination unit 301 takes a picture while the photographer moves while the magnitude of the output of SENSOR_OUT3 exceeds AMP_TH2 in the plus direction and the minus direction. It is determined that it exists.

  FIG. 8B is a graph showing a change of SENSOR_OUT2 with time, assuming that the output in the ROLL direction of the A / D converter 103 is SENSOR_OUT2. When the photographer moves and performs shooting, the output of the angular velocity sensor 102 increases in all of the YAW direction, the PITCH direction, and the ROLL direction. Therefore, in the process of FIG. 5, the frequency at which the process of S103 is performed increases. This is because, as described in the first embodiment, the influence of an increase in the output component due to the shake in the YAW direction or the PITCH direction, which is superimposed on SENSOR_OUT2 due to the shift of the detection axis, is reduced.

  However, on the other hand, the signal of SENSOR_OUT2 itself, which is the output in the ROLL direction of the A / D converter 103, also increases from T10 when the photographer starts walking, as shown in FIG. 9B. In this state, if the image rotation correction amount is limited in accordance with the processing of S103, the effect of correcting the shake in the ROLL direction during walking is weakened, and the video is difficult to see. Therefore, this is avoided by prohibiting or limiting the execution of the process of S103 in the process of S213 by the photographer movement determination unit 301.

  As described above, in the second embodiment of the present invention, when a large movement such as panning occurs in the YAW or PITCH direction with respect to the imaging apparatus 100, the correction is based on the output of the angular velocity sensor in the ROLL direction. The following processing is further added to the first embodiment that limits the amount calculation. That is, from the output of the angular velocity sensor, it is determined whether or not shooting is being performed while the photographer is moving such as walking. If the determination is that shooting is being performed while moving, it occurs in the ROLL direction. Priority is given to the correction of shake that is. As a result, a good image blur correction effect can be obtained even with respect to a shake of a captured image caused by a large shake when the photographer moves.

  In the second embodiment, the rotation correction method has been described by taking the image deformation as an example. However, a method of mechanically rotating the image sensor may be used.

(Third embodiment)
An imaging apparatus according to the third embodiment of the present invention will be described below. In the present embodiment, the configuration of the imaging apparatus is the same as that in FIG. 1, and the internal configuration of the image deformation amount calculation unit 200 is different. Therefore, the description of FIG.

  Hereinafter, the components of the image deformation amount calculation unit 200 and the operation of an example thereof will be described in detail with reference to the block diagram of FIG. In FIG. 9, the same components as those in FIG. 3 are denoted by the same reference numerals, and description thereof is omitted. In the present embodiment, among the deformation components of the image deformation performed by the image deformation unit 110, the translation, rotation, shear, and enlargement / reduction components are not essential, and are not shown in the configuration of FIG. 16), (Equation 19) to (Equation 21), or (Equation 24), and (Equation 27) to (Equation 29).

  In the block diagram of FIG. 9, blocks 220 to 223 are blocks for calculating a correction amount for correcting a horizontal tilt caused by a shake in the YAW direction to the imaging apparatus (a tilt correction amount calculation unit). ). Blocks denoted by reference numerals 224 to 227 are blocks (tilt correction amount calculation unit) for calculating a correction amount for correcting a vertical tilt caused by a shake in the PITCH direction to the imaging apparatus.

  First, blocks 220 to 223 for calculating the horizontal tilt correction amount will be described. Among the outputs from the A / D converter 103, the HPF 220 is supplied with angular velocity data in the YAW direction. The HPF 220 has a (variable) function that can change its characteristics in an arbitrary frequency band, and outputs a signal in a high frequency band by cutting off a low frequency component included in the angular velocity data.

  The integrator 221 has a function of changing its characteristics in an arbitrary frequency band, integrates the output from the HPF 220, and supplies the integrated output to the focal length division unit 222. The focal length division unit 222 divides the output of the integrator 221 by the focal length f calculated by the focal length calculation unit 231 and supplies the result to the saturation prevention control unit 223. The reason for dividing the output of the integrator 221 by the focal length f is (Equation 25).

  The saturation prevention control unit 223 performs control to limit the output from the focal length division unit 222 so that the output (calculated value) from the focal length division unit 222 does not exceed the limiter (limit value). In addition, when the output from the focal length division unit 222 approaches the limiter, the saturation prevention control unit 223 changes the cutoff frequency of the HPF 220 to the high frequency side, shortens the time constant of the integrator 221, etc. Take control. The output of the saturation prevention control unit 223 is the final horizontal correction amount and is supplied to the image deformation amount combining unit 230.

  Next, blocks 224 to 227 for calculating the tilt correction amount in the vertical direction will be described. Among the outputs from the A / D converter 103, the HPF 224 is supplied with angular velocity data in the PITCH direction. The HPF 224 has a function of changing its characteristics in an arbitrary frequency band, and outputs a signal in a high frequency band by cutting off a low frequency component included in the angular velocity data.

  The integrator 225 has a function capable of changing its characteristic in an arbitrary frequency band, integrates the output from the HPF 224, and supplies it to the focal length division unit 226. The focal length division unit 226 divides the output of the integrator 225 by the focal length f calculated by the focal length calculation unit 231 and supplies the result to the saturation prevention control unit 227. The reason for dividing the output of the integrator 225 by the focal length f is (Equation 26).

  The saturation prevention control unit 227 performs control to limit the output from the focal length division unit 226 so that the output from the focal length division unit 226 does not exceed the limiter. In addition, when the output from the focal length division unit 226 approaches the limiter, the saturation prevention control unit 227 changes the cutoff frequency of the HPF 224 to the high frequency side, shortens the time constant of the integrator 225, or the like. Take control. The output of the saturation prevention control unit 227 becomes the final vertical tilt correction amount and is supplied to the image deformation amount combining unit 230.

  The image deformation amount synthesizing unit 230 performs calculations for synthesizing correction amounts such as translation and rotation (not shown) output from the saturation prevention control units 223 and 227 in the horizontal and vertical directions. Specifically, the projective transformation matrix of (Expression 1) is calculated according to (Expression 13). The image deformation amount synthesis unit 230 outputs the calculated values of the elements of the projective transformation matrix to the image deformation unit 110. The image deforming unit 110 performs image blur correction by image deformation based on the output from the image deformation amount combining unit 230.

  FIGS. 10A, 10B, and 10C are diagrams for explaining correction of vertical tilt by the geometric transformation processing in the image deforming unit 110. FIG. FIG. 10A shows a captured image when there is no shake in the imaging apparatus. FIG. 10B shows a captured image when a shake in the PITCH direction occurs in the imaging apparatus and a vertical tilt deformation occurs. Based on the output from the image deformation amount synthesizing unit 230, the image deforming unit 110 sets the image reading range from the image memory 109 as a trapezoidal range as indicated by the dotted line in FIG. FIG. 10C shows an output image in which the image reading range is output as a trapezoid as shown by the dotted line in FIG. 10B, and the subject imaged in a trapezoidal shape in FIG. 10B. Has been corrected to its original shape. When the shake is correctly detected by the angular velocity sensor 102, it is possible to correct the tilt of the image as shown in FIG.

  The tilt control changing unit 302 controls the HPF 220, the integrator 221, the saturation prevention control unit 223, the HPF 224, the integrator 225, and the saturation prevention control unit 227 based on the output of the A / D converter 103 in the YAW direction or the PITCH direction. To change. Hereinafter, the processing of the tilt control change unit 302 will be described with reference to FIGS. 5 and 6. 5 and 6 have been described as processing of the rotation control change unit 300 in the first embodiment, but the rotation control change unit 300 and the tilt control change unit 302 have different signals to be processed, The processing steps are similar. Therefore, in the present embodiment, FIGS. 5 and 6 will be described as processing performed by the tilt control change unit 302. FIG. Moreover, in the following description, the case where the process for the shake signal in the PITCH direction is changed based on the shake signal in the YAW direction will be described. In addition, since it is the same when changing the process with respect to the shake signal of a YAW direction based on the shake signal of a PITCH direction, description is abbreviate | omitted.

  The flowchart of FIG. 5 shows an example of processing of the tilt control change unit 302. The process of the flowchart of FIG. 5 is repeatedly executed at a predetermined cycle such as 60 Hz.

  First, in S100, if the variable SENSOR_OUT1 is a signal generated based on the output of the A / D converter 103 in the YAW direction, it is determined whether or not SENSOR_OUT1 is larger than the threshold AMP_TH1. The signal generated based on the output of the A / D converter 103 in the YAW direction may be the output of the A / D converter 103 itself, or various signals may be output with respect to the output of the A / D converter 103. It is also possible to use a signal that has been subjected to the filtering process of FIG. That is, any signal may be used as long as the signal is generated based on the output of the A / D converter 103. If it is determined in S100 that SENSOR_OUT1 is greater than AMP_TH1, the process proceeds to S101.

  In S101, the value of the counter COUNTER1 is incremented, and the process proceeds to S102. In S102, it is determined whether or not the counter COUNTER1 is larger than the threshold TIME_TH1. If it is determined in S102 that COUNTER1 is equal to or less than TIME_TH1, the processing in FIG. 5 ends. If it is determined in S102 that COUNTER1 is greater than TIME_TH1, the process proceeds to S103. The process of S103 is performed when a shake for a predetermined time (TIME_TH1) or more and a predetermined magnitude (AMP_TH1) or more is applied in the YAW direction (the time when the shake is greater than a predetermined threshold continues for a predetermined time or more) That is, when the panning operation is performed on the imaging apparatus 100.

  In step S103, control is performed so that the tilt correction amount is reduced. Control is performed such as increasing the cutoff frequency (cutoff frequency) of the HPF (high pass filter) 224, shortening the time constant of the integrator 225, and narrowing the limiter of the saturation prevention control unit 227. Note that the three types of control of the HPF 224, the integrator 225, and the saturation prevention control unit 227 may be performed alone, or may be performed by combining a plurality of types of control. In addition to the above three types of control, other methods may be used as long as the control is performed so that the tilt correction amount is reduced. For example, it may be configured to forcibly input a signal that causes the tilt correction amount to approach zero. Further, when the input value to the saturation prevention control unit 227 exceeds a certain threshold value, the cutoff frequency of the HPF 224 is increased, the time constant of the integrator 225 is shortened, and the threshold value is decreased. The amount of correction may be limited. After the process of S103, the process of FIG.

  If it is determined in S100 that SENSOR_OUT1 is equal to or less than AMP_TH1, the process proceeds to S104. In S104, the value of the counter COUNTER1 incremented in S101 is cleared, and the process proceeds to S105.

  In S105, processing is performed to restore the cutoff frequency of the HPF 224 changed in S103, the time constant of the integrator 225, or the limiter of the saturation prevention control unit 227. In addition, when control for limiting the amount of rotation correction is performed, it is canceled. After S105, the process in FIG. 5 ends.

  FIG. 6 is a graph for explaining the effect of the tilt control change unit 302 on the calculation of the tilt correction amount in the vertical direction when the process of FIG. 5 is performed. In FIG. 6, a fixed subject is photographed from time 0 to T0, and then a large movement such as panning is applied to the imaging apparatus from time T0 to T2, and then the shooting returns to the fixed subject. The change of various signals at the time is shown.

  FIG. 6A is a graph showing a change with time of SENSOR_OUT1, which is a signal generated based on the output of the A / D converter 103 in the YAW direction. SENSOR_OUT1 indicates a state in which the output increases during T0 to T2, which is a time zone in which a large movement is applied to the imaging apparatus, and camera shake in the YAW direction is detected at other times.

  FIG. 6B is a graph showing changes in SENSOR_OUT2 with time, assuming that the output of the A / D converter 103 in the PITCH direction is SENSOR_OUT2. FIG. 6B shows the deviation of the detection axis of the angular velocity sensor, although no movement in the PITCH direction occurs during the time T0 to T2 when a large movement is applied to the imaging device with respect to the YAW direction. Shows how the output is generated. At times other than T0 to T2, camera shake in the PITCH direction is detected.

  FIG. 6C is a graph showing the change of CALC_OUT1 over time, assuming that the output signal of the saturation prevention control unit 227 is CALC_OUT1 when the processing of the flowchart of FIG. 5 is not performed by the tilt control change unit 302. It is. FIG. 6C is a graph of FIG. 6B, in which the output generated due to the deviation of the detection axis of the angular velocity sensor between times T0 and T2 cannot be attenuated by the time constants of the HPF 224 and the integrator 225. It shows how it is integrated.

  FIGS. 10D and 10E are diagrams for explaining an image when it is assumed that correction of vertical tilt is performed by the image deformation unit 110 based on the signal of FIG. 6C. . In FIG. 10D, the image pick-up image does not have a shake in the PITCH direction, but the picked-up image is represented by a dotted line portion in FIG. 10D based on the signal in FIG. The image reading range from the image memory 109 is changed to a trapezoidal shape. FIG. 10E shows an output image output by changing the image reading range into a trapezoidal shape as indicated by a dotted line portion in FIG. 10D, and is not deformed into a trapezoidal shape in FIG. The subject is transformed into a trapezoidal shape and output.

  FIG. 6D is a graph showing the change of CALC_OUT2 with time, assuming that the output signal of the saturation prevention control unit 227 when the process of the flowchart of FIG. 5 is performed by the tilt control change unit 302 is CALC_OUT2. In FIG. 6A, the determination of S100 in FIG. 5 is Yes at the timing of time T0 when the value of SENSOR_OUT1 becomes larger than AMP_TH1. Further, the determination in S102 is Yes at the timing of time T1 when COUNTER1 becomes larger than TIME_TH1. Therefore, the process of S103 is performed in the period from time T1 to time T2 when the determination of S100 changes to No. In the process of S103, if the cutoff frequency of the HPF 224 is increased, the attenuation rate in the low frequency band increases, so that the output generated due to the shift of the detection axis in FIG. 6B can be attenuated. Further, if the time constant of the integrator 225 is shortened, the speed at which the output of the integrator 225 converges to zero increases, so that the output generated by the shift of the detection axis in FIG. 6B can be attenuated. When the limiter of the saturation prevention control unit 227 is narrowed, the time for the output of the integrator 225 to reach the limiter is increased, the cutoff frequency of the HPF 224 is likely to be increased, and the time constant of the integrator 225 is likely to be shortened. As a result, the output generated by the shift of the detection axis in FIG. 6B can be attenuated. As a result, the output of the saturation prevention control unit 227 that has continued to increase from time T0 in FIG. 6C can be attenuated as shown in FIG. 6D. Then, by correcting the image deformation unit 110 based on the signal shown in FIG. 6D, the image pickup apparatus is picked up even though the image pickup apparatus is not moving in the PITCH direction as shown in FIG. It is possible to prevent a phenomenon in which the vertical tilt of the image is corrected.

  As described above, the tilt control changing unit 302 also performs control to change the horizontal tilt correction amount calculation process based on the shake signal in the PITCH direction. However, in the above description, the YAW direction and the PICH direction are switched. Since the processing is similar, the description thereof is omitted.

  As described above, in the third embodiment of the present invention, when a large movement such as panning occurs in the YAW direction with respect to the imaging apparatus 100, correction of tilt based on the output of the angular velocity sensor in the PITCH direction is performed. It was decided to limit the amount calculation. In addition, when a large movement such as tilting occurs in the PITCH direction with respect to the imaging apparatus 100, the calculation of the tilt correction amount based on the output of the angular velocity sensor in the YAW direction is limited. As a result, it is possible to minimize the influence of the signal error caused by the deviation of the detection axis of the angular velocity sensor and obtain a good image blur correction effect.

(Fourth embodiment)
An imaging apparatus according to the fourth embodiment of the present invention will be described below. In the present embodiment, the configuration of the imaging apparatus is the same as that in FIG. 1, and the internal configuration of the image deformation amount calculation unit 200 is different. Therefore, the description of FIG. Further, the image deformation amount calculation unit 200 in the present embodiment is similar to the configuration of the third embodiment shown in FIG. 9 as well, and differs from the third embodiment in the configuration of the third embodiment. In addition, a photographer movement determination unit 301 indicated by a broken line in FIG. 9 is added.

  The photographer movement determination unit 301 captures an image while the imaging device 100 moves such as the photographer walks from a signal generated based on the output of the A / D converter 103 in the YAW direction, the PITCH direction, or the ROLL direction. Judge whether or not you are doing. The photographer movement determination unit 301 changes the processing performed by the tilt control change unit 302 when it is determined that the photographer is shooting while moving. Since the process of the photographer movement determination unit 301 is the same as that described in the flowchart of FIG. 7 except for the processes of S213 and S216 in the second embodiment, detailed description other than the processes of S213 and S216 will be described. Omitted.

  In S213, the process of S103 in FIG. 5 performed by the tilt control change unit 302 is prohibited or limited. Prohibiting the processing of S103 is to return the cut-off frequency change of HPF 220 or HPF 224, the time constant change of integrator 221 or integrator 225, the limiter change of saturation prevention control unit 223 or saturation prevention control unit 227 performed in S103. That means. In addition, when the control for limiting the correction amount of the tilt is performed, the control is canceled. To limit the processing of S103, the amount by which the cutoff frequency of HPF 220 or HPF 224 is increased, the amount by which the time constant of integrator 221 or integrator 225 is shortened, the limiter of saturation prevention control unit 223 or saturation prevention control unit 227 is set. This means reducing the amount of narrowing. Further, for example, when the cut-off frequency of the HPF 220 or HPF 224 is changed and the time constant of the integrator 221 or the integrator 225 is changed, one change is not performed. Of the restrictions, a configuration may be adopted in which restrictions on some tilt correction amounts are prohibited.

  In S216, the execution of the process of S103 in FIG. 5 performed by the tilt control change unit 302 is permitted. That is, the process of prohibiting or limiting the process of S103 in S213 is canceled, and the tilt control change unit 302 returns to the state of operating according to the flowchart of FIG. After S216, the process in FIG. 7 ends.

  Next, the processing of the photographer movement determination unit 301 when the photographer performs photographing while walking will be described with reference to FIG. In the second embodiment, SENSOR_OUT2 in FIG. 8B has been described as an output in the ROLL direction of the A / D converter 103. However, in this embodiment, the SENSOR_OUT2 of the A / D converter 103 is described. The description will be given as an output in the YAW direction or the PITCH direction. In FIG. 9, as described above, a fixed subject is photographed while the photographer is stationary for a period of time 0 to T10, and various signals when the photographer is photographed while walking from time T10. Showing changes.

  FIG. 8A is a graph showing a change over time of SENSOR_OUT3 which is a signal generated based on the output of the A / D converter 103 in the YAW direction, the PITCH direction, or the ROLL direction. SENSOR_OUT3 shows a state in which the output increases due to vibration during walking from T10, and relatively small hand shake at the time before T10 is detected.

  In the flowchart of FIG. 7, when COUNT_TH = 2, the photographer movement determination unit 301 takes a picture while the photographer moves while the magnitude of the output of SENSOR_OUT3 exceeds AMP_TH2 in the plus direction and the minus direction. It is determined that it exists.

  FIG. 8B is a graph showing changes in SENSOR_OUT2 with time, assuming that the output of the A / D converter 103 in the YAW direction or the PITCH direction is SENSOR_OUT2. When the photographer moves and performs shooting, the output of the angular velocity sensor 102 increases in all of the YAW direction, the PITCH direction, and the ROLL direction. Therefore, in the process of FIG. 5, the frequency at which the process of S103 is performed increases. This is because, as described in the second embodiment, the influence of the increase in the output component due to the shake in the other axis direction, which is superimposed on SENSOR_OUT2 due to the shift of the detection axis, is reduced.

  However, SENSOR_OUT2, which is the output of the A / D converter 103 in the YAW direction itself or the output in the PITCH direction itself, increases from T10 when the photographer starts walking, as shown in FIG. 8B. In this state, if the correction amount of the tilt of the image is limited in accordance with the processing of S103, the tilt correction effect during walking is weakened and the video is difficult to see. Therefore, in the process of S213 by the photographer movement determination unit 301, this is avoided by prohibiting or limiting (relaxing) the execution of the process of S103.

  As described above, in the fourth embodiment of the present invention, when a large movement such as panning occurs in the YAW or PITCH direction with respect to the imaging apparatus 100, correction based on the output of the angular velocity sensor in the other direction. The following processing is further added to the third embodiment that limits the amount calculation. That is, it is determined from the output of the angular velocity sensor whether or not shooting is being performed while the photographer is walking, and when the determination is that shooting is being performed, the YAW direction or the PITCH direction is determined. Priority is given to the correction of shakes occurring in. As a result, a good blur correction effect can be obtained even with respect to the shake of the captured image caused by a large shake when the photographer moves.

  Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included. A part of the above-described embodiments may be appropriately combined.

  For example, although the present embodiment has been described using a digital video camera as an example, a digital camera or a surveillance camera may be used. Also, a digital single-lens reflex camera may be used. For example, a shake detection sensor is mounted on the interchangeable lens, and the imaging apparatus side receives shake information in the PITCH and YAW directions as appropriate for correcting and limiting rotational shake and tilt in the roll direction. It may be a configuration. Moreover, the electronic device provided with the camera unit like a mobile phone, a smart phone, and a portable game machine may be sufficient.

Claims (10)

A shake detection unit that detects a shake around a first axis perpendicular to the optical axis, a shake around a second axis perpendicular to the first axis and the optical axis, and a shake around the optical axis. A translation correction amount computing means for computing a correction amount of image blur due to a shake in the translation direction of the captured image based on the information on the shake around the first axis and the information on the shake around the second axis. ,
A rotational compensation amount calculating means on the basis of the shake information of the detected light axis, calculates a correction amount of the image blur due to shake of the optical axis of the captured image by the shake detection means,
Based on the correction amount of the image blur is more operation on the translational compensation amount calculating means and the prior SL rotational compensation amount calculating means, first control means for controlling the image blur correcting means for correcting an image blur,
When a time greater than a predetermined threshold value for the shake information about the first axis continues for a predetermined time or more, it is determined that the camera is panned or tilted, and the camera is determined to be panned or tilted. A second control means for reducing a correction amount of image blur around the optical axis of the captured image, as compared to a case where it is not determined that the panned or tilted state is obtained,
An imaging apparatus comprising: A shake detection unit that detects a shake around a first axis perpendicular to the optical axis, a shake around a second axis perpendicular to the first axis and the optical axis, and a shake around the optical axis. A translation correction amount computing means for computing a correction amount of image blur due to a shake in the translation direction of the captured image based on the information on the shake around the first axis and the information on the shake around the second axis. ,
A rotational compensation amount calculating means on the basis of the shake information of the detected light axis, calculates a correction amount of the image blur due to shake of the optical axis of the captured image by the shake detection means,
Based on the correction amount of the image blur is more operation on the translational compensation amount calculating means and the prior SL rotational compensation amount calculating means, first control means for controlling the image blur correcting means for correcting an image blur,
Second control means for limiting a correction amount of image blur around the optical axis of the captured image;
Equipped with a,
The second control means is configured such that when the information about the shake about the first axis is longer than the predetermined threshold for a predetermined time or more, the information about the shake about the first axis is more than the predetermined threshold. In order to reduce the amount of image blur correction around the optical axis of the captured image, the high-pass filter cut-off frequency of the rotation correction amount calculation means is increased compared to the case where a large time does not continue for a predetermined time or longer. An imaging device that is characterized. A shake detection unit that detects a shake around a first axis perpendicular to the optical axis, a shake around a second axis perpendicular to the first axis and the optical axis, and a shake around the optical axis. A translation correction amount computing means for computing a correction amount of image blur due to a shake in the translation direction of the captured image based on the information on the shake around the first axis and the information on the shake around the second axis. ,
A rotational compensation amount calculating means on the basis of the shake information of the detected light axis, calculates a correction amount of the image blur due to shake of the optical axis of the captured image by the shake detection means,
Based on the correction amount of the image blur is more operation on the translational compensation amount calculating means and the prior SL rotational compensation amount calculating means, first control means for controlling the image blur correcting means for correcting an image blur,
Second control means for limiting a correction amount of image blur around the optical axis of the captured image;
Equipped with a,
The second control means is configured such that when the information about the shake about the first axis is longer than the predetermined threshold for a predetermined time or more, the information about the shake about the first axis is more than the predetermined threshold. Compared to a case where a large time does not continue for a predetermined time or longer, the time constant of the integrating means in the rotation correction amount calculating means is shortened in order to reduce the amount of image blur correction around the optical axis of the captured image. An imaging device. A shake detection unit that detects a shake around a first axis perpendicular to the optical axis, a shake around a second axis perpendicular to the first axis and the optical axis, and a shake around the optical axis. A translation correction amount computing means for computing a correction amount of image blur due to a shake in the translation direction of the captured image based on the information on the shake around the first axis and the information on the shake around the second axis. ,
A rotational compensation amount calculating means on the basis of the shake information of the detected light axis, calculates a correction amount of the image blur due to shake of the optical axis of the captured image by the shake detection means,
Based on the correction amount of the image blur is more operation on the translational compensation amount calculating means and the prior SL rotational compensation amount calculating means, first control means for controlling the image blur correcting means for correcting an image blur,
Second control means for limiting a correction amount of image blur around the optical axis of the captured image;
Equipped with a,
The second control means is configured such that when the information about the shake about the first axis is longer than the predetermined threshold for a predetermined time or more, the information about the shake about the first axis is more than the predetermined threshold. Compared with a case where a large time does not continue for a predetermined time or more, in order to reduce a correction amount of image blur around the optical axis of the captured image, a limit of a calculation value of the rotation correction amount calculation unit is reduced. An imaging device. The imaging apparatus further includes a determination unit that determines whether or not the photographer is shooting while moving based on the information about the shake about the first axis ,
The second control unit prohibits or relaxes control of a correction amount of image blur around the optical axis of the captured image when the determination unit determines that the photographer is shooting while moving. The imaging apparatus according to any one of claims 1 to 4, wherein the imaging apparatus is characterized in that The determination means moves the photographer when the signal calculated based on the shake information about the first axis changes with a larger amplitude than a predetermined amplitude for a predetermined number of times within a predetermined period. The imaging apparatus according to claim 5 , wherein it is determined that shooting is being performed. A shake detection unit that detects a shake around a first axis perpendicular to the optical axis, a shake around a second axis perpendicular to the first axis and the optical axis, and a shake around the optical axis. A translation correction amount calculating step for calculating a correction amount of image blur due to a shake in the translation direction of the captured image based on the information about the shake about the first axis and the information about the shake about the second axis, ,
A rotation correction amount calculating step for calculating a correction amount of image shake due to shake around the optical axis of the captured image based on information about shake around the optical axis detected by the shake detection unit ;
On the basis of the translational motion correction amount calculating step and before Symbol rotational compensation amount correction amount of the shake more calculated images to the computing step, a first control step of controlling the image blur correcting means for correcting an image blur,
When a time greater than a predetermined threshold value for the shake information about the first axis continues for a predetermined time or more, it is determined that the camera is panned or tilted, and the camera is determined to be panned or tilted. A second control step that reduces a correction amount of image blur around the optical axis of the captured image, compared to a case where the panned or tilted state is not determined.
An imaging method comprising: A shake detection unit that detects a shake around a first axis perpendicular to the optical axis, a shake around a second axis perpendicular to the first axis and the optical axis, and a shake around the optical axis. A translation correction amount calculating step for calculating a correction amount of image blur due to a shake in the translation direction of the captured image based on the information about the shake about the first axis and the information about the shake about the second axis, ,
A rotation correction amount calculating step for calculating a correction amount of image shake due to shake around the optical axis of the captured image based on information about shake around the optical axis detected by the shake detection unit ;
On the basis of the translational motion correction amount calculating step and before Symbol rotational compensation amount correction amount of the shake more calculated images to the computing step, a first control step of controlling the image blur correcting means for correcting an image blur,
A second control step for limiting the amount of image blur correction around the optical axis of the captured image ,
In the second control step, when the information on the shake about the first axis is longer than the predetermined threshold for a predetermined time or more, the information on the shake about the first axis is more than the predetermined threshold. In order to reduce the amount of image blur correction around the optical axis of the captured image compared to the case where a large time does not continue for a predetermined time or longer, the cutoff frequency of the high-pass filter in the rotation correction amount calculation step is increased. A characteristic imaging method . A shake detection unit that detects a shake around a first axis perpendicular to the optical axis, a shake around a second axis perpendicular to the first axis and the optical axis, and a shake around the optical axis. A translation correction amount calculating step for calculating a correction amount of image blur due to a shake in the translation direction of the captured image based on the information about the shake about the first axis and the information about the shake about the second axis, ,
A rotation correction amount calculating step for calculating a correction amount of image shake due to shake around the optical axis of the captured image based on information about shake around the optical axis detected by the shake detection unit ;
On the basis of the translational motion correction amount calculating step and before Symbol rotational compensation amount correction amount of the shake more calculated images to the computing step, a first control step of controlling the image blur correcting means for correcting an image blur,
A second control step for limiting the amount of image blur correction around the optical axis of the captured image ,
In the second control step, when the information on the shake about the first axis is longer than the predetermined threshold for a predetermined time or more, the information on the shake about the first axis is more than the predetermined threshold. Compared to a case where a large time does not continue for a predetermined time or longer, the time constant of the integration means in the rotation correction amount calculation step is shortened in order to reduce the image blur correction amount around the optical axis of the captured image. An imaging method . A shake detection unit that detects a shake around a first axis perpendicular to the optical axis, a shake around a second axis perpendicular to the first axis and the optical axis, and a shake around the optical axis. A translation correction amount calculating step for calculating a correction amount of image blur due to a shake in the translation direction of the captured image based on the information about the shake about the first axis and the information about the shake about the second axis, ,
A rotation correction amount calculating step for calculating a correction amount of image shake due to shake around the optical axis of the captured image based on information about shake around the optical axis detected by the shake detection unit ;
On the basis of the translational motion correction amount calculating step and before Symbol rotational compensation amount correction amount of the shake more calculated images to the computing step, a first control step of controlling the image blur correcting means for correcting an image blur,
A second control step for limiting the amount of image blur correction around the optical axis of the captured image ,
In the second control step, when the information on the shake about the first axis is longer than the predetermined threshold for a predetermined time or more, the information on the shake about the first axis is more than the predetermined threshold. Compared to a case where a large time does not continue for a predetermined time or more, in order to reduce an image blur correction amount around the optical axis of the captured image, a limit of a calculation value in the rotation correction amount calculation step is reduced. Imaging method .

Images