JP6257289B2 - Image processing apparatus, imaging apparatus including the same, and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus that corrects image shake accompanying image sensor shake.

  Image orientation correction lens (image stabilization optical system) for image stabilization placed in the optical system, because the change in the angle of view affects the image as a motion change due to rotation and translational attitude changes of the imaging device (camera) By controlling and driving this, it is possible to suitably suppress blurring. Moreover, blurring can be more suitably suppressed by performing geometric deformation processing with a high degree of freedom such as projective transformation by signal processing.

  However, if the drive amount of the image blur correction optical system is increased in order to correct a large shake of the image pickup device, the influence of image distortion and resolution deterioration due to decentration aberration due to the image blur correction optical system being off the optical axis center is avoided. It becomes impossible. Furthermore, since the optical properties of the decentration aberration are complicated, the correction processing for the image distortion of the aberration including the decentration aberration cannot be corrected practically, and the image quality deterioration due to the remaining correction or overcorrection cannot be avoided.

  Further, in the case where the image pickup device mounted on the image pickup apparatus is a rolling shutter type in which image information is sequentially read out in units of one line such as a CMOS sensor, a shift occurs in the exposure period for each line. When taking an image with a rolling shutter sensor, if camera shake occurs during the exposure period from the top line to the bottom line of the screen or the image blur correction optical system is decentered, The image of the subject is distorted by the difference in exposure period for each line. Hereinafter, this distortion is referred to as rolling shutter distortion.

  As a technique for controlling the camera shake suppression while correcting the image quality deterioration due to rolling shutter distortion with respect to such a problem, Patent Document 1 is known.

Japanese Patent No. 3185152

  In the image pickup apparatus described in Patent Document 1, the amount of shake of the image pickup apparatus, the drive amount of the image blur correction optical system, and the focal length are used to correct rolling shutter distortion caused by the movement of the image pickup apparatus. By doing this, we propose more accurate correction. However, in blur correction with decentration aberration, in addition to rolling shutter distortion due to the movement of the imaging device, rolling shutter distortion due to decentering of the image blur correction optical system occurs during the exposure time, so the influence is ignored. I can't do that. Therefore, in Patent Document 1, for example, when the image blur correction optical system is largely shifted, there is a problem that the rolling shutter distortion cannot be corrected satisfactorily.

  In view of the above problems, an object of the present invention is to provide an image processing apparatus that is advantageous for correcting rolling shutter distortion, an imaging apparatus including the image processing apparatus, and an image processing method.

An image processing apparatus according to an aspect of the present invention is an image processing apparatus that corrects an image shake accompanying a shake of an imaging apparatus, and an image acquisition unit that acquires an image generated by an imaging element included in the imaging apparatus. A first distortion calculating means for calculating a first distortion generated in the image due to a shake of the image pickup apparatus and an image shake correction optical system for correcting the shake of the image pickup apparatus during the exposure time of the image pickup apparatus; Second distortion calculating means for calculating second distortion generated in the image due to decentering, first distortion obtained from the first distortion calculating means, and second distortion calculating means. Correction means for correcting the image based on the second distortion information.

  According to the present invention, it is possible to provide an image processing apparatus, an image processing method, and an imaging apparatus that are advantageous for correcting rolling shutter distortion.

1 is a block diagram illustrating a configuration of an imaging apparatus that is Embodiment 1 of the present invention. FIG. 3 is a flowchart illustrating the operation of the imaging apparatus according to the first embodiment. It is a schematic diagram of distortion by movement of an imaging device. It is a schematic diagram of distortion generation. It is an outline figure of shape calculation of distortion by a motion of an imaging device. It is a schematic diagram of distortion due to the movement of the image blur correction optical system. It is a block diagram which shows the structure of the imaging device which is Example 2 of this invention. 6 is a flowchart illustrating an operation of the imaging apparatus according to the second embodiment.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 shows the configuration of an image pickup apparatus that is Embodiment 1 of the present invention. In FIG. 1, reference numeral 101 denotes an image pickup optical system for forming a subject image, and reference numeral 102 denotes an image shake correction lens (image shake correction optical system) for blur correction mounted inside the image pickup optical system 101. The image blur correction optical system 102 is driven in a direction different from the optical axis of the imaging optical system. Reference numeral 103 denotes an imaging element such as a CMOS sensor that photoelectrically converts a subject image formed by the imaging optical system 101. The image sensor 103 of the present invention is a rolling shutter sensor that sequentially reads image information in units of one line. A development processing unit 104 forms a video signal from an electrical signal output from the image sensor 103. The development processing unit 104 includes an A / D conversion unit, an auto gain control unit (AGC), and an auto white balance unit (AWB) (not shown), and converts an analog signal output from the image sensor 103 into a digital signal.

  The imaging device 103 and the development processing unit 104 constitute an imaging system that generates an image.

  A memory 105 temporarily stores and holds one frame or a plurality of frame images of the video signal formed by the development processing unit 104.

  A shake detection unit 106 includes a detection device such as an angular velocity sensor (gyro sensor) that acquires information on the movement of the imaging device such as camera shake or camera work. The shake detection unit 106 mainly functions as a shake detection unit that detects the shake of the imaging apparatus. In addition to using the gyro sensor, the method for acquiring the movement information of the image pickup apparatus is not only a method using a sensing device such as an acceleration sensor (shift sensor) but also a movement from an input image and a past reference image. There is a so-called motion vector detection method. Reference numeral 107 denotes an optical system drive amount detection unit that acquires the amount driven by the image blur correction optical system 102 mainly for correcting blur. In other words, the optical system drive amount detection unit 107 functions as a drive amount detection unit that detects the drive amount of the image shake correction optical system 102 that corrects the shake of the imaging apparatus. Reference numeral 108 denotes an imaging parameter acquisition unit (imaging parameter acquisition means) that acquires imaging parameters such as a focal length and a shutter speed during imaging.

  Reference numeral 109 denotes a first distortion calculation unit (first distortion calculation means), which is motion information (shake information) of the imaging device obtained from the shake detection unit 106 and image shake correction obtained from the optical system drive amount detection unit 107. The shape of the rolling shutter distortion is calculated using the drive amount (drive information) of the optical system. Reference numeral 110 denotes a second distortion calculation unit (second distortion calculation unit), which is image blur correction optical system drive information obtained from the optical system drive amount detection unit 107 and imaging obtained from the imaging parameter acquisition unit 108. The shape of the rolling shutter distortion is calculated using the parameter information.

  A geometric deformation processing unit 111 has a function as an image acquisition unit that acquires a captured image generated by the imaging system (the image sensor 103 and the development processing unit 104) via the memory 105. The geometric deformation processing unit 111 (correction unit) corrects the rolling shutter distortion for the input image based on the rolling shutter distortion shape obtained from the first distortion calculation unit 109 and the second distortion calculation unit 110. The geometric deformation process is applied. Then, the image output unit 112 records the video with the corrected rolling shutter distortion on a recording medium (not shown) and displays it on a display (not shown). The first distortion calculation unit 109, the second distortion calculation unit 110, and the geometric deformation processing unit 111 constitute an image processing unit (image processing apparatus) that corrects image distortion. The operation of the imaging apparatus of the present invention configured as described above will be described with reference to the flowchart shown in FIG.

  In FIG. 2, in step S <b> 201, the subject image formed by the imaging optical system 101 is output as an analog signal corresponding to the subject luminance in the imaging element 103, and a video signal is generated by performing processing of the development processing unit 104. At this time, when the shake detection unit 106 detects the movement of the image pickup apparatus, the image shake correction optical system 102 is driven in a direction to cancel the detected movement of the shake. Thereby, it is possible to optically correct the movement of blur of the imaging apparatus. The development processing unit 104 converts an analog signal into, for example, a 12-bit digital signal by an A / D conversion unit (not shown). Further, the digital video signal subjected to signal level correction and white level correction by AGC and AWB (not shown) is stored and held in the memory 105. In the imaging apparatus of this embodiment, frame images are sequentially generated at a predetermined frame rate, and the frame images stored and held in the memory 105 are input to the geometric deformation processing unit 111. The frame images stored and held in the memory 105 are also updated sequentially.

  In step S202, movement information of the imaging device, drive information of the image blur correction optical system, and shooting parameter information of the imaging device are acquired. First, the shake detection unit 106 acquires motion information of the imaging device. When a gyro sensor is used to acquire motion information, motion information in the pan, tilt, and roll directions of the imaging device can be obtained. Although the gyro sensor is mentioned here as the motion information acquisition means, the present invention is not particularly limited to this, and other means for obtaining motion information of the imaging apparatus such as an acceleration sensor or a magnetic sensor may be used. Next, the optical system drive amount detection unit 107 acquires how much the image blur correction optical system 102 has been driven in order to correct the blur of the imaging apparatus. For example, when the image blur correction optical system 102 shifts perpendicularly to the optical axis, the shift amount is acquired, and when the blur is corrected by pan or tilt movement, the rotation angle is acquired. Become. Further, the shooting parameter acquisition unit 108 acquires various shooting parameter information at the time of shooting. At this time, the imaging parameter to be acquired is a parameter that affects the shape of the rolling shutter distortion, and in this embodiment, focal length information is acquired as an example.

  In step S203, the first distortion calculation unit 109 uses the shake information obtained from the shake detection unit 106 and the image shake correction optical system drive information obtained from the optical system drive amount detection unit 107 to move the imaging apparatus. The shape of rolling shutter distortion caused by the above is calculated.

  FIG. 3 shows a schematic diagram of rolling shutter distortion caused by the movement of the imaging apparatus. In the figure, reference numeral 301 denotes an image pickup apparatus, which is assumed to have a blur in the horizontal direction. Reference numeral 302 schematically represents an image taken at that time, and it is assumed that a rectangular object as shown by 303 is captured. At this time, if the imaging device 301 is moved in the horizontal direction during the exposure time, the subject 303 existing in the image is distorted into a parallelogram as shown at 304 and is photographed.

  In order to explain the cause of such distortion, FIG. 4 shows the relationship between each line of the image and time t when the image is taken with a rolling shutter type imaging device. Each line 401 in a certain frame forms image information by sequentially performing exposure and reading for each line in accordance with the vertical synchronization signal VD. Since the image sensor 103 of the present invention is a rolling shutter type sensor in which image information is sequentially read out in units of one line, a shift occurs in the exposure period for each line, resulting in distortion of the subject image (rolling shutter distortion). ) Occurs. As shown in FIG. 4, when the delay time for readout between lines is τ and the total number of lines in the image is H, a deviation of the readout time of (H−1) τ occurs between the first row and the last row. Therefore, if the imaging device moves during the exposure time, the readout time differs for each line. Therefore, an image at the position of the imaging device at the time of readout for each line is taken, and finally one image is formed. Appear as distortion of the subject.

  At this time, if the image blur correction optical system is driven to correct the blur of the image pickup apparatus, the movement of the image shake correction optical system cancels the movement of the image pickup apparatus. Therefore, as indicated by reference numeral 305 in FIG. The rolling shutter distortion that has occurred in the subject 304 is corrected. Here, if the blur of the image pickup apparatus can be completely corrected by driving the image blur correction optical system, the rolling shutter distortion generated in the subject 304 is also completely corrected, and a rectangular shape having an original shape such as 303 is obtained. Imaged. However, in actuality, the image cannot be completely corrected depending on the frequency and size of the blur of the image pickup apparatus, and rolling shutter distortion may remain as a correction remainder as indicated by 305. In this step, the shape (first distortion) is estimated for such a rolling shutter distortion correction remainder, and information for correcting the geometric deformation processing is generated.

  FIG. 5 shows a schematic diagram of shape calculation of rolling shutter distortion. FIG. 5A shows the rolling shutter distortion that occurs when only the movement of the imaging apparatus is considered. When the imaging apparatus moves in the horizontal direction during the exposure time, the rectangular object 502 captured in the captured image 501 is distorted in a parallelogram shape as indicated by 503. The shape of the distortion at that time can be calculated from the motion information of the imaging device during the exposure time obtained from the shake detection unit 106. Reference numeral 504 in FIG. 5A indicates the amount of movement of the image pickup apparatus during the exposure time. The horizontal axis indicates time t, and the vertical axis indicates the amount of movement p1 (t) of the image pickup apparatus at time t. Here, if the total number of lines in the image 501 is H, and the shift amount of the read start time per line is τ, the time for reading the last line is t = τ (H−1). That is, reference numeral 504 indicates how much the image pickup apparatus has moved while being read to the last line with reference to the top line of the image, and the value corresponds to the amount of deviation between the lines of the image 501 at the same time. ing. Therefore, it is possible to calculate the shape of the rolling shutter distortion by using the motion information of the imaging device. However, for example, when the motion information of the imaging device is acquired from the gyro sensor, the value is angle information. Therefore, in order to correspond to the amount of deviation between the lines on the image, the angle is on the image. It is necessary to convert the number of pixels corresponding to the movement in

  On the other hand, FIG. 5B shows a schematic diagram of rolling shutter distortion that occurs when only the image blur correction optical system is driven in a direction in which the movement of the image pickup apparatus is corrected. Since the driving is performed in a direction to cancel out the blur during the exposure time, the rolling shutter distortion 505 caused by the movement is shaped so that the rolling shutter distortion caused by the movement of the imaging device shown in FIG. Become. The shape of the rolling shutter distortion due to the driving of the image blur correction optical system can be calculated using the drive information of the image blur correction optical system during the exposure time obtained from the optical system drive amount detection unit 107. Reference numeral 506 in the figure shows drive information of the image blur correction optical system, where the horizontal axis indicates time t and the vertical axis indicates the amount of motion p2 (t) of the image blur correction optical system at time t. Accordingly, it is possible to calculate the shape of the rolling shutter distortion depending on the driving amount of the image blur correction optical system in the same manner as in FIG. However, for example, when the image blur correction optical system shifts vertically with respect to the optical axis, since the value is obtained as distance information, in order to take correspondence with the amount of deviation between lines on the image. It is necessary to convert how many pixels the distance corresponds to on the image.

The remaining shape of the rolling shutter distortion after the blur of the image pickup apparatus is corrected by the image blur correction optical system is determined from the shape of the rolling shutter distortion caused by the movement of the image pickup apparatus, and the rolling shutter distortion caused by driving the image shake correction optical system. The shape is subtracted. That is, when the rolling shutter distortion shape p1 (t) caused by the movement of the imaging apparatus and the rolling shutter distortion shape p2 (t) of the image blur correction optical system are used, the remaining correction shape p3 (t) is expressed by the following equation. It is represented by
p3 (t) = p1 (t) −p2 (t) (Formula 1)
In the present embodiment, as an example of rolling shutter distortion caused by the imaging apparatus, a case where horizontal movement is performed at a constant speed during the exposure time has been described. However, the present invention is not limited to this. For example, the same can be said even when the speed or moving direction of the imaging apparatus changes during the exposure time. In addition, since it is difficult to cancel out the rolling shutter distortion caused by the movement of the image pickup apparatus by a roll or a tilt by the image blur correction optical system, in such a case, the rolling shutter distortion is determined only from the movement information of the image pickup apparatus. The shape of is calculated. Then, the rolling shutter distortion shape p3 (t) caused by the imaging device obtained in this step is transmitted to the geometric deformation processing unit 111.

  In step S204, the second distortion calculation unit 110 calculates the shape (second distortion) of the rolling shutter distortion that occurs when the image blur correction optical system is decentered during the exposure time in order to correct the blur of the imaging apparatus. To do. For calculating the shape of the rolling shutter distortion, the driving amount of the image blur correction optical system obtained from the optical system driving amount detection unit 107 and the photographing parameter obtained from the photographing parameter acquisition unit 108 are used. FIG. 6 shows a schematic diagram of rolling shutter distortion that occurs when only the image blur correction optical system is driven. In the figure, as an example, the rolling shutter distortion when the image blur correction optical system 601 inside the imaging optical system of the imaging apparatus is driven in the horizontal direction is shown. At this time, the rectangular subject 603 shown in the image 602 is distorted in shape as shown by 604 due to the influence of rolling shutter distortion caused by the drive of the image blur correction optical system and decentering. This rolling shutter distortion is caused when the image blur correction optical system is driven and the amount of eccentricity changes during the exposure time. Here, when the shape of the distortion changes depending on the direction and magnitude of the eccentricity, the shape of the eccentric distortion changes for each line by driving the image blur correction optical system during the exposure time. The shape of the Rollin shutter distortion when viewed as a single image is asymmetric and non-linear.

  Further, this rolling shutter distortion is caused by driving (eccentricity) of the image blur correction optical system regardless of the rolling shutter distortion caused by the movement of the image pickup apparatus. The shape of rolling shutter distortion caused by driving the image blur correction optical system also changes depending on the focal length of the imaging optical system 101. Therefore, in order to calculate the shape of the rolling shutter distortion, the drive information of the image blur correction optical system obtained from the optical system drive amount detection unit 107 and the focal length information obtained from the imaging parameter acquisition unit 108 are used. However, since the distortion becomes asymmetrical and non-linear depending on the driving of the image blur correction optical system, the drive amount of the image blur correction optical system cannot be directly set to the distortion shape. As a method for acquiring the distortion shape in such a case, a method of applying to a model expression representing the relationship between the amount of eccentricity and the shape of distortion, or the relationship between the driving amount of the image blur correction optical system and the focal length and the shape of distortion in advance. There is a method of using a lookup table that has been calculated and stored. Then, the shape of the rolling shutter distortion resulting from the driving of the image blur correction optical system obtained in this step is transmitted to the geometric deformation processing unit 111.

  As described above, in an imaging apparatus equipped with an image blur correction optical system, when the image blur correction optical system is largely driven to correct a large blurring, rolling shutter distortion and image blur correction optics caused by the movement of the image pickup apparatus are performed. Rolling shutter distortion occurs due to the eccentricity of the system. These two types of distortions are phenomena that occur independently of each other, and the information necessary to correct them and the calculation method of the distortion shape for correction are also different. In other words, it is difficult to calculate the shape as one distortion simply because it is the same rolling shutter distortion, and in order to perform highly accurate distortion correction, each distortion shape is individually set as shown in this embodiment. It is necessary to calculate it independently.

  In step S <b> 205, the geometric deformation processing unit 111 corrects rolling shutter distortion generated in the image using the rolling shutter distortion shape information obtained from the first distortion calculation unit 109 and the second distortion calculation unit. . Since the information on the obtained distortion shape indicates to which coordinate each pixel in the image has moved due to two rolling shutter distortions, geometric deformation processing is performed so that these pixels return to their original coordinate positions. As a result, it is possible to obtain an image in which all lines in the image are captured at the same timing.

  In step S206, the image with the rolling shutter distortion corrected is stored in a storage device (not shown) or displayed on the display device by the image output unit 112.

  As described above, the shape of the rolling shutter distortion caused by the movement of the imaging apparatus and the shape of the rolling shutter distortion caused by driving the image blur correction optical system are calculated individually. As a result, it is possible to apply information and a calculation method suitable for the calculation of each rolling shutter distortion, and it is possible to perform favorable rolling shutter distortion correction.

  FIG. 7 shows the configuration of an image pickup apparatus that is Embodiment 2 of the present invention. In this embodiment, the two rolling shutter distortion shapes generated by the movement of the image pickup apparatus and the driving of the image blur correction optical system are combined and then corrected. In the figure, the same reference numerals as those in FIG. 1 denote the same components as those shown in FIG. The imaging apparatus according to the present exemplary embodiment includes a distortion synthesis unit 701 in the configuration illustrated in FIG.

  Moreover, the flowchart in a present Example is shown in FIG. In the present embodiment, only a portion that performs processing different from that in the first embodiment in FIG. 8 will be described. Steps S201 to S204 in FIG. 8 are the same as S201 to S204 in FIG.

  In step S <b> 801, the distortion combining unit 701 combines two rolling shutter distortion shapes obtained from the first distortion calculating unit 109 and the second distortion calculating unit 110.

  Although these two rolling shutter distortions are phenomena that occur independently of each other, they occur simultaneously on the image plane, and the two rolling shutter distortions are combined. That is, it is also possible to correct by using a combination of two rolling shutter distortion shapes calculated by the first and second distortion calculation units.

Here, when the shape of the rolling shutter distortion caused by the movement of the imaging device is p3 (t) and the shape of the rolling shutter distortion caused by driving the image blur correction optical system is p4 (t), the synthesized distortion shape p5 ( t) can be calculated by the following equation.
p5 (t) = αp3 (t) + βp4 (t) (Formula 2)
In the above equation, α and β are weighting coefficients for the shape of the rolling shutter distortion, and values of 0 to 1 are set respectively. As a setting method of α and β, for example, if α = β = 1, two distortion shapes are added together. Therefore, geometric deformation for the purpose of completely correcting the rolling shutter distortion can be performed by using the combined distortion shape p5 (t) at this time. At this time, unlike the first embodiment, the geometric deformation processing for correcting the rolling shutter distortion in the geometric deformation processing unit 111 performed in S205 may be performed only once. Therefore, it is possible to prevent deterioration in image quality due to performing geometric deformation processing a plurality of times, and to reduce power consumption by reducing the amount of calculation.

  Another use of the weighting coefficient is adjustment of a margin area. When correcting the rolling shutter distortion of the image by the geometric deformation process, there is a possibility that a missing area is generated in the peripheral portion of the image by referring to the coordinate position where the image data does not exist due to the geometric deformation. In general, in order to prevent such image loss, a method is used in which a certain area further outside the output image size is imaged to secure a margin area for geometric deformation. . However, if the amount of rolling shutter distortion correction is too large and exceeds the reserved margin area, the image peripheral portion will also be missing. In such a case, by reducing the value of the weighting coefficient in Equation 2, it is possible to prevent the loss of images by allowing the remaining correction amount of the rolling shutter distortion to be within the margin. It becomes. As a method of setting the weighting coefficient at this time, there are, for example, a method of reducing the weighting with the larger distortion amount, and a method of reducing both the weightings to the same extent. The method of using the weighting coefficient is not limited to the above example. For example, when the amount of distortion is such that there is no problem without correction, the weighting coefficient may be set to zero.

  Step S206 in FIG. 8 is the same as step S206 in FIG.

  So far, the operation of the present proposal in the second embodiment has been described. In addition to independently calculating the rolling shutter distortion caused by the movement of the imaging apparatus and the rolling shutter distortion caused by driving the image blur correction optical system as described in the first embodiment, the distortion shapes are combined. Is used to correct distortion by geometric deformation processing. By performing weighting processing at the time of synthesis, it is possible to reduce image loss due to distortion correction and image quality deterioration due to geometric deformation.

  According to the present invention, the shape of the rolling shutter distortion caused by the movement of the imaging apparatus and the shape of the rolling shutter distortion caused by the driving of the image blur correction optical system are independently calculated. Accordingly, it is possible to estimate an appropriate shape according to the type of rolling shutter distortion occurring on the image, and it is possible to correct the rolling shutter distortion with high accuracy.

  An image processing apparatus and an imaging apparatus that can obtain a highly accurate rolling shutter distortion correction effect can be realized.

109 first distortion calculation unit 110 second distortion calculation unit 111 geometric deformation processing unit

Claims (10)

An image processing apparatus that corrects image shake accompanying image sensor shake,
Image acquisition means for acquiring an image generated by an imaging element included in the imaging device;
First distortion calculating means for calculating a first distortion generated in the image due to a shake of the imaging device;
Second distortion calculation means for calculating a second distortion generated in the image caused by an image blur correction optical system that corrects a shake of the imaging apparatus being decentered during an exposure time of the imaging apparatus ;
Correction means for correcting the image based on information on the first distortion obtained from the first distortion calculation means and the second distortion obtained from the second distortion calculation means;
An image processing apparatus comprising: The first distortion generated in the image due to the shake of the imaging device and the second distortion generated in the image due to the image shake correction optical system being decentered during the exposure time of the imaging device are independent. The image processing apparatus according to claim 1, wherein the image processing apparatus is calculated using   The first distortion calculation means includes shake information obtained from shake detection means for detecting shake of the imaging apparatus, drive information obtained from drive amount detection means for detecting a drive amount of the image shake correction optical system, The image processing apparatus according to claim 1, wherein a first distortion generated in the image due to a shake of the imaging apparatus is calculated based on the image. The second distortion calculating unit includes driving information obtained from a driving amount detecting unit that detects a driving amount of the image shake correcting optical system, and a photographing parameter obtained from a photographing parameter obtaining unit that obtains a photographing parameter of the imaging apparatus. 4. The second distortion generated in the image due to the image blur correction optical system being decentered during the exposure time of the imaging device based on the information is calculated. 5. An image processing apparatus according to claim 1. The image sensor is a rolling shutter type sensor from which image information is sequentially read out in units of one line.
The image processing apparatus according to claim 1, wherein the distortion is a rolling shutter distortion in which a subject image is distorted due to a shift in an exposure period for each line. Distortion combining means for combining the first distortion obtained from the first distortion calculation means and the second distortion obtained from the second distortion calculation means;
The image processing apparatus according to claim 1, wherein the correction unit corrects the image based on distortion information synthesized by the distortion synthesis unit. An imaging optical system including an image blur correction optical system;
An image sensor that photoelectrically converts a subject image formed by the imaging optical system;
An image processing apparatus comprising: the image processing apparatus according to claim 1. An image processing method for correcting shake of an image accompanying shake of an imaging device,
An image acquisition step of acquiring an image generated by an imaging element included in the imaging device;
A first distortion calculating step of calculating a first distortion generated in the image due to a shake of the imaging device;
A second distortion calculating step of calculating a second distortion generated in the image caused by an image blur correction optical system that corrects a shake of the imaging apparatus being decentered during an exposure time of the imaging apparatus ;
A correction step for correcting the image based on information on a first distortion obtained from the first distortion calculation step and a second distortion obtained from the second distortion calculation step;
An image processing method comprising:   The first distortion calculation step includes shake information obtained from shake detection means for detecting shake of the imaging apparatus, drive information obtained from drive amount detection means for detecting the drive amount of the image shake correction optical system, The image processing method according to claim 8, wherein a first distortion generated in the image due to a shake of the imaging device is calculated based on the image. The second distortion calculating step includes driving information obtained from a driving amount detection unit that detects a driving amount of the image blur correction optical system, and imaging parameters obtained from imaging parameter acquisition unit that acquires imaging parameters of the imaging apparatus. 10. The second distortion generated in the image due to the image blur correction optical system being decentered during the exposure time of the imaging device based on the information is calculated. Image processing method.