JP7137433B2 - Blur correction device, imaging device, blur correction method, and program - Google Patents

Description

The present invention relates to a blur correction device, an imaging device, a blur correction method, and a program.

2. Description of the Related Art In recent years, as the performance of imaging devices has improved, many imaging devices and imaging lenses are equipped with an image stabilization mechanism. When camera shake correction is performed in an image pickup apparatus, a camera shake detection sensor (for example, a gyro sensor) that detects acceleration, angular acceleration, angular velocity, etc. is used, and the image sensor is driven based on the output signal of this sensor. Further, when camera shake correction is performed with the photographing lens, part of the lenses constituting the photographing lens is driven based on the output signal of the similar camera shake detection sensor. In this way, it is possible to perform camera shake correction by driving the imaging device and the lens so as to cancel camera shake.

On the other hand, there is an image restoration method that restores a blurred image by performing image processing on a photographed image without installing a camera shake correction mechanism in the imaging device and the photographing lens. Among many image restoration methods, a method of restoring a blurred image using a deterioration function assumed from blur information such as a blur direction and a blur amount is generally known.

The deterioration function used in the method of restoring a blurred image will be described below. When f(x, y) is an ideal image and g(x, y) is a degraded image,
g(x,y)=∬h(x,y,x',y')f(x',y')dx'dy'+v(x,y) …(1)
We assume that there is a relationship where h(x, y, x', y') is the degradation function and v(x, y) is random noise in the output image.
If the degraded image of a point does not depend on the position of the point, except for translation, then the Point Spread Function (PSF) will be of the form h(xx', yy') , equation (1) is
g(x,y)=∬h(x-x',y-y')f(x',y')dx'dy'+v(x,y) …(2)
becomes. When there is no noise, performing the Fourier transform on both sides of equation (2) yields from the convolution theorem:
G(u,v)=H(u,v)F(u,v) …(3)
becomes. where G(u,v), H(u,v) and F(u,v) are the Fourier transforms of g(x,y), h(x,y) and f(x,y), respectively. show. Also, H(u, v) is the transfer function of the system that transforms the ideal image f(x, y) into the degraded image g(x, y).

という式が得られる。この場合、Ｔは露光時間であり、便宜的に－Ｔ／２からＴ／２までであるとした。式（４）の両辺をフーリエ変換すると、

となる。ｘ－α（ｔ）＝ξ、ｙ－β（ｔ）＝ηと置くと、式（４ｆ）は、

となる。この式（５）から、劣化は式（３）又はこれに等しい式（２）でモデル化されることがわかる。この劣化の伝達関数Ｈ（ｕ，ｖ）は、

という式で与えられる。 Here, since only degradation due to relative motion between the camera and the subject (so-called blurred image) will be considered, the degradation model will be described below. We assume that the image is temporally invariant except for motion. If the relative motion is approximately equal to that due to motion of the recording film in the plane, then the total exposure at a point on the film is the integral of the instantaneous exposure over the time the shutter is open. sought by In this case, the time required for opening and closing the shutter is negligible. Assuming that α(t) and β(t) are the x- and y-direction components of the displacement, respectively,

is obtained. In this case, T is the exposure time, which is assumed to be from -T/2 to T/2 for the sake of convenience. Fourier transforming both sides of equation (4) yields

becomes. Putting x−α(t)=ξ and y−β(t)=η, equation (4f) becomes

becomes. From this equation (5) it can be seen that the degradation is modeled by equation (3) or the equivalent equation (2). The transfer function H(u,v) of this degradation is

is given by the formula

Therefore, the point response function when the X-axis is shaken at an angle θ and a constant speed V for a time T is
H(u,v)=sinπωT/πω …(7)
is given by here,
ω=(u-u0)Vcosθ+(v-v0)Vsinθ …(8)
and u0 and v0 are the center coordinates of the image. When ω is very small, it is approximated as H(u, v)=T.

The point spread function h(x-x', y-y') is generally expressed as an image (PSF image).

Here, an image restoration method using an inverse filter, which is a basic image restoration algorithm, will be described as an example of the image restoration method using the deterioration function described above. It is assumed that the degraded image g(x, y) and the original image f(x, y) follow the model of equation (2) above.

At this time, if there is no noise, the Fourier transform of g(x, y), f(x, y) and PSF, h(x, y) satisfies the above equation (3). Here, transforming equation (3), we get
F(u,v)=G(u,v)/H(u,v) …(9)
becomes. From this equation (9), if H(u, v) is known, by inverse Fourier transforming the Fourier transform G(u, v) of the degraded image by 1/H(u, v), f It can be seen that (x, y) can be recovered.

Here, assuming that the inverse Fourier transform of 1/H in Equation (9) is the restoration filter R, the original image can be similarly restored by performing convolution processing on the real image as shown in Equation (10) below. image can be obtained.
f(x,y)=g(x,y)*R(x,y) …(10)

However, since an actual image has a noise component, if a restoration filter created by taking the complete inverse of the point response function H(u, v) of the blurred image is used as described above, the noise component will be generated along with the degraded image. It is amplified and generally does not give a good image. Regarding this point, there is known a method, such as a Wiener filter, for suppressing the recovery rate on the high frequency side of the image according to the intensity ratio between the image signal and the noise signal. As described above, since the complete reciprocal of the point response function H(u, v) cannot be used, restoration processing by image processing can reduce the effects of camera shake. can't. Therefore, in general, the correction accuracy of camera shake correction by image processing is inferior to the case of using the camera shake correction mechanism mounted on the imaging device and the photographing lens.

On the other hand, in the case of using the camera shake correction mechanism mounted on the imaging apparatus and the photographing lens described above, there is a problem that the distortion aberration of the lens used for photographing lowers the correction accuracy. The size and shape of the subject appearing in the peripheral portion of the image differ depending on the distortion aberration characteristics of the lens. Further, when the same subject appears in the central portion and the peripheral portion of the image, the size and shape of the subject on the image differ between the central portion and the peripheral portion of the image due to distortion aberration. As described above, since the size and shape of a subject in a photographed image differ depending on the distortion aberration, the size and shape of the blur seen on the image when camera shake occurs also differ depending on the distortion aberration characteristics. Due to distortion aberration, the effects of blurring in the central portion of the image and blurring in the peripheral portion of the image are different. For example, the central portion of the image is appropriately corrected, but the peripheral portion of the image is undercorrected or overcorrected, and the effects of camera shake remain.

Japanese Patent Application Laid-Open No. 2002-200001 proposes a method of reducing the effects of camera shake by performing camera shake correction through image processing on an image to which distortion aberration correction has been applied. Further, Japanese Patent Laid-Open No. 2002-200002 proposes a method of changing a reference position for distortion correction in order to suppress expansion and contraction of the peripheral portion of a photographed image caused by the above-described distortion aberration.

JP-A-2006-33759 JP 2006-129175 A

However, since the method of Patent Document 1 corrects camera shake by image processing, there is a problem that the accuracy of camera shake correction is low as described above. In addition, the method of Patent Document 2 can cope with the expansion and contraction of the peripheral portion of an image that occurs between frames of a moving image due to distortion aberration. It is not possible to reduce the residual correction caused by

SUMMARY OF THE INVENTION The present invention has been made in view of such circumstances, and an object of the present invention is to provide a technique for effectively reducing image blur caused by camera shake in an image shot using an optical system with distortion. and

In order to solve the above-mentioned problems, the present invention provides detection means for detecting blurring of an image pickup device, and reducing blurring of the image based on the blurring of the image pickup device while the image is being captured by the image pickup device. driving means for driving a correcting means; acquisition means for acquiring a residual blur component of the image based on the driving amount of the correcting means and distortion aberration of an optical system used to capture the image; and image processing means for performing image processing for reducing the residual blur component.

According to the present invention, it is possible to effectively reduce image blur caused by camera shake in an image shot using an optical system with distortion.

Other features and advantages of the present invention will become more apparent from the accompanying drawings and the description in the following detailed description.

Central cross-sectional view of the camera system. 2 is a block diagram showing the electrical configuration of the camera system; FIG. 4 is a flowchart of camera shake correction processing according to the first embodiment; FIG. 10 is a diagram showing an example of camera shake correction driving information acquired in S102; FIG. FIG. 5 is a diagram showing an example of distortion aberration information acquired in S103; The figure which showed the generation|occurrence|production principle of a shake remainder. FIG. 10 is a diagram showing blur remaining components; (a) A diagram showing the displacement of the camera shake correction drive of the image sensor 6, (b) A diagram showing the trajectory of the camera shake correction drive on the driving plane, (c) A blur residual component corresponding to the camera shake correction drive FIG. 12D shows m[t] and n[t], and (d) shows the trajectory of the remaining blur on the driving plane. 9 is a flowchart of camera shake correction processing according to the second embodiment; FIG. 10 is a view showing locations on a screen for calculating residual blur components; FIG. FIG. 8 is a block diagram showing the configuration of an image processing unit 7 according to the third embodiment; FIG. 10 is a diagram showing a trajectory of residual blur when the direction in which distortion aberration occurs in a certain area is assumed to be constant; 10 is a flowchart of camera shake correction processing according to the third embodiment; The figure which shows an example of the table which showed the restoration filter to select. (a) A diagram showing an example of the trajectory of camera shake correction driving and the direction in which distortion aberration is generated. (b) A diagram showing an example of displacement γ[t] in the direction of distortion aberration generation in camera shake correction driving. 6 is a flowchart of camera shake correction processing in a configuration that holds a pre-calculated point response function of remaining blur as correction data. 14 is a flowchart of camera shake correction processing according to the fourth embodiment;

Embodiments of the present invention will be described below with reference to the accompanying drawings. Elements with the same reference numbers represent the same or similar elements throughout the drawings. It should be noted that the technical scope of the present invention is determined by the claims and is not limited by the following individual embodiments. Also, not all combinations of features described in the embodiments are essential to the present invention. It is also possible to combine features described in separate embodiments as appropriate.

[First embodiment]
FIG. 1A is a central cross-sectional view of the camera system. In FIG. 1A, 1 is an imaging device, 2 is a lens unit attached to the imaging device 1, 3 is an imaging optical system consisting of a plurality of lenses, 4 is the optical axis of the imaging optical system 3, and 6 is an imaging element. , 9a denotes a rear display device, and 9b denotes an EVF, respectively. Reference numeral 11 denotes an electrical contact between the imaging device 1 and the lens unit 2, 12 denotes a lens control circuit provided in the lens unit 2, 14 denotes a blur correction section, 15 denotes a blur detection section, and 16 denotes a shutter mechanism. show. The imaging device 1 also functions as a blur correction device.

FIG. 1B is a block diagram showing the electrical configuration of the camera system of FIG. 1A. In FIG. 1B, 5 is a camera control circuit, 7 is an image processing section, 8 is a memory section, 9 is a display section, 10 is an operation detection section, and 13 is a lens driving section. The display unit 9 includes the rear display device 9a and the EVF 9b of FIG. 1A. The lens drive unit 13 can drive a focus lens, a diaphragm, and the like. The blur detection unit 15 can detect blur of the imaging device 1 . For example, a vibrating gyroscope or the like can be used as the shake detection unit 15 . The blur correction unit 14 is a mechanism that translates the imaging device 6 in a plane perpendicular to the optical axis 4 .

Light from an object forms an image on the imaging surface of the imaging device 6 via the imaging optical system 3 . A focus evaluation amount and an appropriate exposure amount are obtained from the imaging device 6 . By appropriately adjusting the imaging optical system 3 based on this information, the imaging apparatus 1 can expose the imaging element 6 to an appropriate amount of object light and form a subject image in the vicinity of the imaging element 6 . can. The shutter mechanism 16 also controls whether or not the subject image reaches the imaging element 6 by running the shutter curtain. The shutter mechanism 16 has at least a curtain (mechanical rear curtain) for blocking the subject image, and the shutter mechanism 16 completes the exposure.

The image processing unit 7 can perform A/D conversion, white balance adjustment, gamma correction, interpolation calculation, etc., and generate an image for recording. The other image processing section 22 of the image processing section 7 performs A/D conversion, white balance adjustment, gamma correction, interpolation calculation, etc., and generates an image for recording. A blur residual component generation unit 20 of the image processing unit 7 generates a blur residual component described later that could not be corrected by the translational control of the imaging device 6 by the blur correction unit 14 . The image restoration unit 21 of the image processing unit 7 performs image processing, which will be described later, on the image A/D-converted by the image processing unit 7 based on the generated blur residue component, thereby correcting the blur residue. .

The memory section 8 includes a recording section. The camera control circuit 5 outputs images to the recording section of the memory section 8 . The camera control circuit 5 also displays an image to be presented to the user on the display unit 9 .

The camera control circuit 5 generates and outputs a timing signal and the like for imaging. The camera control circuit 5 controls the imaging system, the image processing system, and the recording/reproducing system in accordance with external operations. For example, when the operation detection unit 10 detects that a shutter release button (not shown) has been pressed, the camera control circuit 5 controls the driving of the imaging element 6, the image processing unit 7, and the like. The camera control circuit 5 also controls the state of each segment of the information display device for displaying information included in the display unit 9 . The rear display device 9 a includes a touch panel, and the touch panel is connected to the operation detection section 10 .

An adjustment operation of the photographing optical system 3 will be described. An image processing unit 7 is connected to the camera control circuit 5 , and the camera control circuit 5 obtains an appropriate focus position and aperture value based on signals from the image sensor 6 . That is, the camera control circuit 5 performs photometry and distance measurement operations based on the signal from the image sensor 6, and determines exposure conditions (F number, shutter speed, etc.). Still images and moving images can be captured by controlling the operation of each unit of the imaging device 1 according to the user's operation to the operation detection unit 10 .

In controlling the blur correction section 14 of the imaging device 1 , the camera control circuit 5 operates the blur correction section 14 based on the signal from the blur detection section 15 . The camera control circuit 5 generates a target value based on the output of the blur detection section 15 and controls the drive of the blur correction section 14 .

FIG. 2 is a flowchart of camera shake correction processing according to the first embodiment. For example, when the operation detection unit 10 detects pressing of a shutter release button (not shown), the processing of this flowchart starts.

In S<b>101 , the camera control circuit 5 performs exposure using the imaging device 6 . During an exposure period (while an image is being captured), the camera control circuit 5 uses the blur correction unit 14 based on the blurring of the imaging device 1 detected by the blurring detection unit 15 to operate the image sensor 6 for camera shake correction. drive. A signal obtained by exposure is A/D converted and acquired as image data.

In S<b>102 , the residual blur component generation unit 20 acquires from the camera control circuit 5 drive information for the image sensor 6 by the blur correction unit 14 during the exposure period of the image sensor 6 .

FIG. 3 is a diagram showing an example of camera shake correction drive information acquired in S102. In FIG. 3A, the horizontal axis indicates time, and the vertical axis indicates displacement α[t] of the image sensor 6 in the x direction. In FIG. 3B, the horizontal axis indicates time, and the vertical axis indicates displacement β[t] of the imaging element 6 in the y direction. FIG. 3(c) is a diagram showing the trajectory of camera shake correction driving on the driving plane. As can be seen from the linear shape of the displacements α[t] and β[t], in this example, the imaging element 6 is driven linearly at a constant speed. b0 to b3 in FIG. 3(c) respectively correspond to times t0 to t3 during the exposure period shown in FIGS. 3(a) and 3(b).

In this example, the acquired camera shake correction drive information corresponds to the camera shake information detected by the shake detection unit 15, and the camera shake correction drive shown here is based on the optical axis 4 on the image sensor 6. It is assumed that this is equivalent to blur caused by camera shake of an image formed in the vicinity. Therefore, the residual blur component generator 20 may acquire the camera shake information detected by the shake detector 15 instead of acquiring the camera shake correction drive information. In this case, the residual blur component generator 20 can use the camera shake information as the camera shake correction drive information. That is, the acquisition of camera shake information by the residual blur component generation unit 20 is technically equivalent to the acquisition of camera shake correction driving information by the residual blur component generation unit 20 . Therefore, when reference is made in this specification to processing using camera shake correction driving information, the processing includes processing using camera shake information as camera shake correction driving information.

Next, in S<b>103 , the residual blur component generation unit 20 acquires distortion aberration information at the time of photographing in the photographing optical system 3 . For example, the blur residual component generation unit 20 may read distortion aberration information corresponding to shooting conditions such as the focal position at the time of shooting from the distortion aberration information stored in advance in the memory unit 8 . Alternatively, the blur residual component generation unit 20 may acquire the distortion aberration information at the time of shooting from the camera control circuit 5 via the electric contact 11 from the lens unit 2 .

FIG. 4 is a diagram showing an example of distortion aberration information acquired in S103. In FIG. 4, the horizontal axis indicates the image height indicating the distance from the optical axis 4, and the vertical axis indicates the distortion factor due to distortion aberration. In this example, the distortion factor is a negative value, and since the distortion factor takes a smaller value as the image height increases, it exhibits barrel-shaped distortion.

Next, in S104, the residual blur component generation unit 20 calculates the residual blur component based on the camera shake correction driving information (driving amount of the image sensor 6) and the distortion aberration information acquired in S102 and S103.

FIG. 5 is a diagram showing the principle of the occurrence of blur residue, and shows coordinates on a plane driven by the imaging device 6. As shown in FIG. Here, the position of interest S0 is considered. A position S0′ indicates a position where the image formed at the target position S0 is formed when no distortion aberration occurs. The position S0' can be obtained based on the distortion aberration information acquired in S103. The position S1' is a position reflecting the driving displacement of the imaging element 6 at the time t1 shown in FIG. 3 with respect to the position S0'. That is, it means that the image formed at the position S0' when no distortion aberration occurs is formed at the position S1' at the time t1 due to camera shake. A position S1 indicates a position where the image of the position S1' is formed due to distortion aberration. In other words, the image formed at the target position S0 at the start of exposure is formed at the position S1 due to camera shake at time t1. A position b1′ is a position reflecting the positional displacement of the imaging device 6 at time t1 shown in FIG. 3 with respect to the target position S0. If the image formed at the target position S0 due to camera shake is formed at the position b1′ at time t1, the camera shake correction drive of the image sensor 6 can cancel out the effects of the camera shake. , is actually imaged at position S1 instead of position b1′. That is, at the time t1, the difference between the position S1 and the position b1' cannot be corrected by the camera shake correction driving of the image sensor 6, and a blur residue occurs. Assuming that displacements in the x- and y-directions of the residual blur component with respect to time are m[t] and n[t], respectively, m[t1] and n[t1] at time t1 are calculated by the following equations.
m[t1]=x1-(x0+α[t1]) …(11)
n[t1]=y1-(y0+β[t1]) …(12)

Here, the method of obtaining the residual blur component at the time t1 has been described, but the calculations at the times t2 and t3 can be made in the same way. FIG. 6 is a diagram showing an example of m[t] and n[t] calculated in this way.

Up to this point, an example has been shown in which camera shake and camera shake correction driving are linear at a constant speed, but in many cases this is not the case with actual camera shake. FIG. 7 is a diagram showing camera shake correction driving when the camera shake that has occurred is not at a constant speed and has a curved trajectory, and the remaining components of the shake. FIG. 7A is a diagram showing the displacement of the camera shake correction driving of the imaging device 6. FIG. In FIG. 7A, the horizontal axis indicates time, and the vertical axis indicates displacement α[t] in the x direction and displacement β[t] in the y direction. FIG. 7B is a diagram showing the trajectory of camera shake correction driving on the driving plane. FIG. 7(c) is a diagram showing residual blur components m[t] and n[t] corresponding to the camera shake correction driving shown in FIGS. 7(a) and 7(b). FIG. 7D is a diagram showing the trajectory of the residual blur on the driving plane.

Note that the blur remaining component generation unit 20 may acquire the blur remaining component by another method instead of acquiring the blur remaining component by calculation. For example, the residual blur component generation unit 20 may acquire residual blur components from a lookup table (LUT) that holds camera shake correction driving information and distortion aberration information as inputs and residual blur components as outputs. This LUT is stored in the memory unit 8, for example.

Next, in S105, the image restoration unit 21 generates a point spread function h(x, y) of residual blur based on residual blur components m[t] and n[t] calculated in S104. The shape of the point spread function corresponds to the trajectory of the remaining blur. In addition, the intensity of the point spread function is set according to the change amount of the remaining blur component with respect to time, because the longer the point image moving due to the blur stays, the higher the value becomes.

In S106, the image restoration unit 21 generates a restoration filter for correcting the residual blur based on the point spread function of the residual blur. A plurality of techniques are known for generating a restoration filter from a point spread function, but the Wiener filter is used in this embodiment. The image restoration unit 21 calculates a point response function H (u, v) by Fourier transforming the point spread function h (x, y), performs an inverse Fourier transform on the Wiener filter generated from the point response function, and restores the function as a restoration filter. do.

In S107, the image restoration unit 21 convolves the generated restoration filter with the image data acquired in S101, thereby reducing the residual blur.

In this way, from the camera shake correction drive information and the distortion aberration information, the residual blur component in the image corrected for camera shake by the camera shake correction drive is calculated, and the residual blur is reduced by image processing based on the calculated residual blur component. is possible.

In the above description, the residual blurring component at one position of interest has been described. However, since the distortion aberration differs depending on the position of the image, the residual blurring component also differs depending on the position of the image. Therefore, by performing the above-described processing according to the position of the image, it is possible to correct the residual blur of the entire image.

Further, in the present embodiment, an example of driving the imaging element 6 during the exposure period has been described, but the present embodiment is similarly applied to the case of driving the blur correction lens of the photographing optical system 3 for blur correction. It is possible. Specifically, the photographic optical system 3 has a blur correction lens, and the lens control circuit 12 appropriately drives the blur correction lens via the lens driving section 13 based on the signal from the blur detection section 15 . In this case, in the above description, the driving of the imaging element 6 is replaced with the displacement of the subject by driving the blur correction lens. Further, the residual blur component generation unit 20 uses distortion aberration information based on the imaging optical system 3 in which the blur correction lens is driven. Thereby, the residual blur component generating section 20 can calculate the residual blur component in the same manner as in the case of driving the imaging element 6 .

As described above, according to the first embodiment, the image capturing apparatus 1 calculates residual blur components in an image corrected for camera shake by camera shake correction driving from camera shake correction drive information and distortion aberration information, Image processing based on the calculated blur residual component is performed to reduce blur residual. This makes it possible to effectively reduce image blur caused by camera shake in an image captured using an optical system with distortion.

[Second embodiment]
In the second embodiment, a configuration for correcting both blur residuals and aberrations will be described. Also, a configuration for reducing the load of processing for calculating the remaining blur component will be described. In this embodiment, the basic configuration of the camera system is the same as in the first embodiment (see FIG. 1). Differences from the first embodiment will be mainly described below.

In general, when an image of an object is captured by an image capturing apparatus, the obtained image is not a little degraded, especially due to the aberration of the image capturing optical system. The blur component of the degraded image is caused by spherical aberration, coma aberration, curvature of field, astigmatism, etc. of the imaging optical system. The blurring component of the image due to these aberrations is that the light beam emitted from one point on the subject should converge again at one point on the imaging surface, if there is no aberration and there is no effect of diffraction. pointing to something This is optically called a point spread function. Image blur means, for example, an out-of-focus image, but here it specifically refers to an image that is out of focus but is blurred due to the aberration of the photographing optical system. Color fringing in a color image, which is caused by longitudinal chromatic aberration, chromatic spherical aberration, and chromatic coma aberration of the imaging optical system, can also be said to be a difference in blurring for each wavelength of light.

When correcting image quality deterioration due to such aberration, if the optical transfer function obtained by Fourier transforming the point spread function corresponding to the aberration is H(u, v) in Equation (9), Equation (9) and Equation (9) and Equation (9) It is possible to correct by the method described in (10).

FIG. 8 is a flowchart of camera shake correction processing according to the second embodiment. For example, when the operation detection unit 10 detects pressing of a shutter release button (not shown), the processing of this flowchart starts.

In S204, the residual blur component generation unit 20 calculates the residual blur component based on the camera shake correction driving information and the distortion aberration information acquired in S102 and S103. In the present embodiment, the locations within the screen for which the remaining blurring components are calculated are limited from the viewpoint of reducing the processing load.

FIG. 9 is a diagram showing locations (target positions) in the screen for calculating residual blur components in the second embodiment. As indicated by the plotted points in FIG. 9, the residual blur component generation unit 20 calculates residual blur components only at 63 positions, ie, 7 vertical positions and 9 horizontal positions. The remaining blur components are calculated by calculating the remaining blur components m[t] and n[t] by a method similar to the method described in the first embodiment.

Next, in S205, the image restoration unit 21 generates a point spread function h(x, y) of 63 remaining blur components based on the remaining blur components m[t] and n[t] calculated in S204. do.

In S206, the image restoration unit 21 acquires a point spread function corresponding to the aberration of the imaging optical system 3 at the time of imaging. For example, the image restoration unit 21 may read the point spread function corresponding to the photographing state such as the focus position at the time of photographing from data of the point spread function stored in advance in the memory unit 8 . Alternatively, the image restoration unit 21 may acquire the point spread function at the time of photographing from the camera control circuit 5 via the electric contact 11 from the lens unit 2 . At this time, the image restoration unit 21 may correct the acquired point spread function as necessary. This correction involves preparing data corresponding to discrete shooting conditions in order to reduce the number of point spread function data to be prepared in advance, and adjusting the point spread function when performing actual processing. It is to correct the function. Since the aberration characteristics differ depending on the position within the screen, the image restoration unit 21 acquires the point spread functions corresponding to the 63 locations shown in FIG.

In S207, the image restoration unit 21 generates a restoration filter for correcting blur due to blur residue and aberration based on the point spread function corresponding to blur residue and the point spread function corresponding to aberration. The image restoration unit 21 calculates a point response function H(u, v) by Fourier transforming the point spread function h(x, y) corresponding to the residual blur, and calculates the point spread function H(u, v) corresponding to the aberration. , y) is Fourier-transformed to calculate a point response function H′(u, v). Then, let I(u, v) shown in the following equation be the transfer function of deterioration due to blur residue and aberration.
I(u,v)=H(u,v)H'(u,v) …(13)

As described above, it is possible to restore blur residuals and deterioration due to aberration by using the reciprocal of I. In the present embodiment, a Wiener filter generated from I is used, and the Wiener filter is inverse Fourier transformed into real space Let the upper filter be the restoration filter.

In the above description, it is assumed that the image restoration unit 21 acquires the point spread function corresponding to the aberration in S206. However, the point spread function h'(x, y) is stored in advance in the memory unit 8 or the like in the state of the point response function H'(u, v) obtained by Fourier transform, and the image restoration unit 21 restores this point response function in S206. H'(u,v) may be obtained. In this case, the Fourier transform of the point spread function h'(x, y) corresponding to the aberration is not required when generating the restoration filter in S207.

Next, in S<b>208 , the image restoration unit 21 convolves the generated restoration filter with the image data acquired in S<b>101 , thereby reducing residual blurring and blur due to aberration. Since there are 63 generated restoration filters in the screen, there are areas in the screen where there are no corresponding restoration filters. For such a region (positions other than the target position for acquiring the remaining blurring component), the image restoration unit 21 interpolates restoration filters (restoration filters for nearby target positions) at four locations around which restoration filters exist. generates and uses a restoration filter corresponding to the target pixel. As an interpolation method, a method of interpolating and generating restoration filter coefficients by an existing method such as linear interpolation can be used.

As described above, according to the second embodiment, the imaging apparatus 1 generates a restoration filter that considers aberration in addition to the residual blur component. As a result, it is possible to reduce blur remaining components as well as blurring caused by aberration by performing the process of convolving the restoration filter with the image data only once. Also, by limiting the locations in the screen where restoration filters are generated, it is possible to reduce the processing load.

[Third embodiment]
In the third embodiment, a configuration for reducing the remaining blurring component using correction data for reducing the remaining blurring component of an image will be described. In this embodiment, the basic configuration of the camera system is the same as that of the first embodiment (see FIG. 1), but the configuration of the image processing section 7 is different from that of the first embodiment. Differences from the first embodiment will be mainly described below.

FIG. 10 is a block diagram showing the configuration of the image processing section 7 according to the third embodiment. In addition, the image processing unit 22 performs A/D conversion, white balance adjustment, gamma correction, interpolation calculation, etc., and generates an image for recording. The residual blur correction data selection unit 23 selects data for correcting residual blur components that could not be corrected by the translational control of the imaging element 6 by the blur correction unit 14 . This processing will be described later. The image restoration unit 21 performs image processing, which will be described later, on the image that has been A/D converted by the image processing unit 7, thereby correcting the residual blur.

In the first embodiment, the method of generating the residual blur component has been described with reference to FIG. FIG. 7D shows the generated trajectory information of the remaining blur. In the present embodiment, it is assumed that the direction in which distortion aberration occurs is unchanged in a neighboring area on the image, and the occurrence of blur residual components is considered. That is, in FIG. 5, the direction from the position S0 of interest to the position S0' is considered to be the same as the direction from the position S1 to S1'. In this case, the direction from position b1' to position S1 is the same. As described above, the difference between the position b1' and the position S1 is the residual blur component at time t1, and similarly, the residual blur components at times t2 and t3 also point in the same direction. In this way, when the direction in which distortion aberration occurs in a certain region is assumed to be constant, the direction in which residual blur components are generated is the same at each time, and the trajectory of residual blur can be considered as a straight line as shown in FIG. is possible. In reality, the direction in which distortion aberration occurs differs depending on the position, but since the direction in which distortion aberration occurs in the vicinity of the image periphery, where distortion tends to increase, tends to be similar, it is assumed that the direction in which blurring occurs is straight. Think as

As shown in FIG. 5, the magnitude of the residual blur component (the length of the locus of the residual blur) varies depending on the drive amount of the camera shake correction mechanism (for example, the image pickup device 6) and the distortion aberration characteristics of the photographic optical system 3. It is clear from the principle of occurrence of blur residue. The larger the drive range of the camera shake correction mechanism and the larger the change in the distortion aberration distortion rate with respect to the image height, the larger the residual blur component. Also, the intensity of the point spread function of the residual blur component varies depending on the drive of the camera shake correction mechanism. Therefore, if restoration filters corresponding to a plurality of sizes of residual blur and point spread functions with representative intensities are prepared in advance, residual blur can be reduced by selecting a restoration filter.

FIG. 12 is a flowchart of camera shake correction processing according to the third embodiment. For example, when the operation detection unit 10 detects pressing of a shutter release button (not shown), the processing of this flowchart starts.

In S304, the residual blur correction data selection unit 23 selects and acquires a restoration filter (correction data for reducing the residual blur component) based on the camera shake correction driving information and the distortion aberration information acquired in S102 and S103. .

FIG. 13 is a diagram showing an example of a table showing restoration filters to be selected. In this example, the residual blur correction data selection unit 23 selects a restoration filter for each area within the screen based on the driving amount e of the camera shake correction mechanism and the gradient d of the distortion rate with respect to the image height. Here, the drive amount e of the camera shake correction mechanism is a value obtained from the displacement of the camera shake correction drive in the distortion generation direction. FIG. 14(a) is an example showing the trajectory of camera shake correction driving and the direction in which distortion aberration is generated, and FIG. This is an example. As the drive amount e of the camera shake correction mechanism, the difference between the maximum value and the minimum value of the displacement γ[t] may be used, or the standard deviation indicating the variation of the displacement γ[t] may be used. As the gradient d of the distortion rate with respect to the image height, the gradient of the distortion aberration information shown in FIG. 4 at the target image height is used.

Next, each restoration filter will be described. The restoration filter in FIG. 13 is configured to correct a larger amount of residual blur as the gradient d of the distortion rate with respect to the image height increases. That is, the magnitude of the residual blur component to be corrected is Filter 00<Filter 01<Filter 02<Filter 03. Further, the restoration filter in FIG. 13 is configured to correct a larger residual blur as the drive amount e of the camera shake correction mechanism increases. That is, the magnitude of the residual blur component to be corrected is Filter 00<Filter 10<Filter 20<Filter 30. FIG. Further, each restoration filter is assumed to have only one direction in which the remaining blurring occurs, and the trajectory is assumed to be a straight line. Here, the intensity of the point spread function, which is the basis for generating the restoration filter, is assumed to be uniform. That is, the case where the camera shake correction mechanism is driven at a constant speed is held as a representative condition.

The table shown in FIG. 13 defines the restoration filter to be used according to the drive amount e of the camera shake correction mechanism and the value of the gradient d of the distortion rate with respect to the image height. The restoration filter to use can be selected by making a comparison against . The determination thresholds K[ ] and l[ ] and the filters 00 to 33 are values set in advance based on the characteristics of the image stabilization mechanism and the photographic optical system 3 .

Next, in S305, the image restoration unit 21 rotates the restoration filter acquired in S304 according to the direction in which distortion aberration occurs. This is because the restoration filter acquired in S304 is a restoration filter in which the direction in which the residual blurring occurs is fixed to one direction. because there is

In S306, the image restoration unit 21 convolves the restoration filter rotated in S305 with the image data acquired in S101, thereby reducing residual blurring.

In this way, a plurality of restoration filters generated in advance are held (for example, in the memory unit 8), and a restoration filter is selected based on the camera shake correction drive information and the distortion aberration information, thereby reducing the residual blurring. is possible.

In the above description, a configuration has been described in which a restoration filter generated in advance is held as correction data selected by the residual blur correction data selection unit 23 . However, the correction data selected by the residual blur correction data selection unit 23 is a point spread function for generating a restoration filter or a point response function obtained by Fourier transforming the point spread function (point response function of residual blur). may

FIG. 15 is a flowchart of camera shake correction processing in a configuration in which a pre-calculated point response function of remaining blur is held as correction data. In the camera shake correction process of FIG. 15, both the remaining blur and the aberration are corrected. For example, when the operation detection unit 10 detects pressing of a shutter release button (not shown), the processing of this flowchart starts.

In S404, the residual blur correction data selection unit 23 selects a point response function for residual blur based on the camera shake correction drive information and the distortion aberration information acquired in S102 and S103. The image capturing apparatus 1 stores point response functions corresponding to the restoration filters described above with reference to FIG. 13 in a similar table format in the memory unit 8 . The remaining blur correction data selection unit 23 selects a point response function based on the driving amount e of the camera shake correction mechanism and the gradient d of the distortion rate with respect to the image height. At this time, as shown in FIG. 9, the locations within the screen where the selection is made may be limited from the viewpoint of reducing the processing load. As indicated by the plotted points in FIG. 9, the residual blur correction data selection unit 23 selects point response functions only at 63 points in total, 7 points in the vertical direction and 9 points in the horizontal direction.

In S405, the image restoration unit 21 acquires a point response function corresponding to the aberration of the imaging optical system 3 during imaging. For example, the image restoration unit 21 may read a point spread function corresponding to the photographing state such as the focal position at the time of photographing from the point response function data stored in advance in the memory unit 8 . Alternatively, the image restoration section 21 may acquire the point response function at the time of photographing from the camera control circuit 5 via the electric contact 11 from the lens unit 2 . At this time, the image restoration unit 21 may correct the acquired point response function as necessary. This correction involves preparing data corresponding to discrete imaging states in order to reduce the number of point response function data to be prepared in advance, and converting the point response function when actually executing processing. It is to correct. Since the aberration characteristics differ depending on the position within the screen, the image restoration unit 21 acquires point response functions corresponding to the 63 locations shown in FIG.

In S406, the image restoration unit 21 generates a restoration filter for correcting blur due to blur residue and aberration based on the point response function corresponding to blur residue and the point response function corresponding to aberration. The image restoration unit 21 uses the point response function H(u, v) corresponding to the residual blur and the point response function H'(u, v) corresponding to the aberration to determine the deterioration due to the residual blur and the aberration according to Equation (13). Calculate the transfer function I(u,v). Then, the image restoration unit 21 generates a restoration filter based on I(u, v) by the same processing as S207 in FIG.

In S407, the image restoration unit 21 convolves the generated restoration filter with the image data acquired in S101, thereby reducing residual blurring and blur due to aberration. Since there are 63 created restoration filters in the screen, there are areas in the screen where there are no corresponding restoration filters. For such an area, the image restoration unit 21 interpolates the four restoration filters around where the restoration filters exist to generate and use the restoration filter corresponding to the target pixel. As an interpolation method, a method of interpolating and generating restoration filter coefficients by an existing method such as linear interpolation can be used.

As described above, according to the third embodiment, the image capturing apparatus 1 holds a plurality of correction data (restoration filter, point response function, point spread function, etc.) generated in advance, and shake correction drive information. and distortion aberration information. Then, the imaging apparatus 1 reduces the residual blur component of the image based on the acquired correction data. This makes it possible to effectively reduce image blur caused by camera shake in an image captured using an optical system with distortion.

[Fourth embodiment]
In the fourth embodiment, a configuration will be described in which both the blur correction lens and the imaging device 6 are driven for blur correction. In this embodiment, the basic configuration of the camera system is the same as in the third embodiment (see FIGS. 1 and 10). Differences from the third embodiment will be mainly described below.

In this embodiment, the imaging optical system 3 has a blur correction lens, and the lens control circuit 12 drives the blur correction lens via the lens driving section 13 based on the signal from the blur detection section 15 . Further, the blur correction unit 14 translates the imaging element 6 within a plane orthogonal to the optical axis 4 based on the signal from the blur detection unit 15 .

By driving both the blur correction lens and the imaging device 6 during camera shake correction, it is possible to correct larger camera shake than when either one is driven. On the other hand, there are cases where a lens unit without a blur correction lens is attached to the imaging device 1 . In this case, the amount of camera shake that can be corrected is small, but the camera shake can be corrected by driving only the imaging device 6 .

When the blur correction lens is driven, a part of the lens (blur correction lens) constituting the photographic optical system 3 is shifted. may differ. When the optical axis 4 shifts, not only does the center of distortion aberration shift, but also the characteristics of the distortion aberration with respect to the image height, which indicates the distance from the optical axis 4, may exhibit different characteristics depending on the drive amount of the blur correction lens. This means that the residual blur component generated due to distortion differs between when the imaging element 6 is driven and when the blur correction lens is driven. Therefore, in the case of correcting the residual blurring, the correction accuracy is improved by using different correction data depending on the blurring correction method.

FIG. 16 is a flowchart of camera shake correction processing according to the fourth embodiment. For example, when the operation detection unit 10 detects pressing of a shutter release button (not shown), the processing of this flowchart starts.

In S<b>501 , the camera control circuit 5 performs exposure using the imaging device 6 . During the exposure period, the camera control circuit 5 drives the image sensor 6 for camera shake correction using the shake correction section 14 . Further, when the photographing optical system 3 has a blur correction lens, the camera control circuit 5 drives the blur correction lens via the lens driving section 13 . At this time, depending on the amount of camera shake, the camera control circuit 5 may perform control to drive only the blur correction lens without driving the image sensor 6, or control to vary the characteristics for each drive. A signal obtained by exposure is A/D converted and acquired as image data.

In S<b>502 , the remaining blur correction data selection unit 23 acquires drive information for the image sensor 6 by the blur correction unit 14 during the exposure period of the image sensor 6 from the camera control circuit 5 . It is assumed that the driving information to be acquired is the same as the information described in S102 of FIG. 2 in the first embodiment.

In S<b>503 , the residual blur correction data selection unit 23 transmits the drive information of the blur correction lens by the lens driving unit 13 during the exposure period of the image pickup device 6 from the lens unit 2 to the image pickup apparatus 1 via the electrical contact 11 . It is acquired from the camera control circuit 5 as information to be transmitted. The drive information to be acquired includes the displacement information of the blur correction lens in the x direction and the y direction with respect to time, in the same manner as the configuration of the drive information of the image sensor 6 acquired in S502.

In S504, the residual blur correction data selection unit 23 determines whether or not the blur correction lens is to be driven based on the drive information of the blur correction lens acquired in S503. If the lens unit 2 does not have a blur correction lens, it is determined here as no drive. Also, even if the lens unit 2 has a blur correction lens, if control is performed not to drive the blur correction lens, it is determined that the lens is not driven. If it is determined that the motion compensation lens is driven, the process advances to S505; otherwise, the process advances to S507.

In S505, the residual blur correction data selection unit 23 acquires distortion aberration information corresponding to the driving information of the blur correction lens. For example, the residual blur correction data selection unit 23 may read distortion aberration information corresponding to the shooting conditions such as the focus position at the time of shooting from the distortion aberration information stored in advance in the memory unit 8 . Alternatively, the residual blur correction data selection unit 23 may acquire the distortion aberration information at the time of shooting from the camera control circuit 5 via the electric contact 11 from the lens unit 2 . During the exposure period, the distortion aberration characteristics are continuously changing due to the driving of the blur correction lens. Therefore, as the distortion aberration information acquired here, an average value during the exposure period may be used, or representative distortion aberration information determined from the driving information of the blur correction lens may be used.

In S506, the remaining blur correction data selection unit 23 selects a restoration filter based on the camera shake correction drive information acquired in S502 and S503 and the distortion aberration information acquired in S505. The selection method is the same as the method of the third embodiment described with reference to FIG. 13, but it is necessary to consider that both the image sensor 6 and the blur correction lens are driven. . Therefore, the remaining blur correction data selection unit 23 calculates the distance from the optical axis center in consideration of the driving of the image sensor 6 and the blur correction lens for the image height when obtaining the tilt d of the distortion rate with respect to the image height. . Further, the remaining blur correction data selection unit 23 calculates the drive amount e of the camera shake correction mechanism by using, for example, the sum of the drive amounts of the image sensor 6 and the blur correction lens.

On the other hand, if it is determined in S504 that the blur correction lens is not driven, then in S507 the residual blur correction data selection unit 23 acquires distortion aberration information corresponding to the state in which the blur correction lens is not driven.

In S508, the remaining blur correction data selection unit 23 selects a restoration filter based on the camera shake correction drive information acquired in S502 and the distortion aberration information acquired in S507.

Next, in S509, the image restoration unit 21 rotates the restoration filter acquired in S506 or S508 according to the distortion aberration generation direction. This is because the restoration filter acquired in S506 or S508 is a restoration filter in which the direction in which blurring remains occur is fixed to one direction, so the restoration filter is rotated according to the direction in which distortion aberration occurs according to the position in the screen. because it is necessary to

In S510, the image restoration unit 21 convolves the restoration filter rotated in S509 with the image data acquired in S501, thereby reducing residual blurring.

As described above, according to the fourth embodiment, the imaging apparatus 1 acquires different residual blur correction data depending on the camera shake correction method. As a result, it is possible to effectively reduce residual blur in an imaging apparatus having a plurality of blur correction methods.

[Other embodiments]
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

DESCRIPTION OF SYMBOLS 1... Imaging device 2... Lens unit 3... Imaging optical system 5... Camera control circuit 6... Imaging element 7... Image processing part 8... Memory part 13... Lens drive part 14... Blur correction part 15... Blur detection unit 20... Blur residual component generation unit 21... Image restoration unit 23... Blur residual correction data selection unit

Claims (18)

detection means for detecting blurring of the imaging device;
driving means for driving correction means to reduce blurring of the image based on blurring of the imaging device while the imaging device is capturing an image;
acquisition means for acquiring a residual blur component of the image based on the driving amount of the correction means and the distortion aberration of an optical system used to capture the image;
image processing means for performing image processing to reduce the residual blur component of the image;
A blur correction device comprising: 2. The apparatus according to claim 1, wherein the acquiring unit acquires the blur residual component by calculating the blur residual component based on the driving amount of the correction unit and the distortion aberration of the optical system. image stabilization device. 2. The image processing means reduces the residual blur component of the image by generating a restoration filter based on the residual blur component and applying the restoration filter to the image. 3. The blur correction device according to 2. the acquiring means acquires the residual blur component for a partial target position of the image;
The image processing means generates the restoration filter for the target position based on the residual blur component, and for positions other than the target position, the restoration filter is generated by interpolating restoration filters of nearby target positions. 4. The blur correction device according to claim 3, which generates . The image processing means generates the restoration filter based on the aberration of the optical system in addition to the blur remaining component, and applies the restoration filter to the image, so that the blur remaining component of the image and the 5. The blur correction device according to claim 3, wherein blurring of the image caused by the aberration of the optical system is reduced. The blur correction device according to any one of claims 1 to 5, wherein the correction means includes an image sensor of the image pickup device or a blur correction lens included in the optical system. detection means for detecting blurring of the imaging device;
driving means for driving correction means to reduce blurring of the image based on blurring of the imaging device while the imaging device is capturing an image;
Acquisition means for acquiring correction data for reducing residual blurring components of the image based on the driving amount of the correction means and the distortion aberration of an optical system used to capture the image;
image processing means for performing image processing for reducing the residual blur component on the image based on the correction data;
A blur correction device comprising: the correction data includes a restoration filter corresponding to the residual blur component;
8. The blur correction device according to claim 7, wherein the image processing means reduces the residual blur component of the image by applying the restoration filter to the image. the acquiring means acquires the correction data including the restoration filter for a target position of a part of the image;
9. The blur correction device according to claim 8, wherein, for positions other than the target position, the image processing means acquires the restoration filter by interpolating restoration filters of nearby target positions. the correction data includes a point response function corresponding to the residual blur component or a point spread function corresponding to the residual blur component;
8. The image processing unit according to claim 7, wherein the image processing means generates a restoration filter based on the correction data, and applies the restoration filter to the image to reduce the residual blur component of the image. image stabilization device. the acquisition means acquires the correction data for a target position of a part of the image;
The image processing means generates the restoration filter for the target position based on the correction data, and generates the restoration filter for positions other than the target position by interpolating restoration filters of nearby target positions. 11. The blur correction device according to claim 10, wherein: The image processing means generates the restoration filter based on the aberration of the optical system in addition to the correction data, and applies the restoration filter to the image so that the image is processed together with the blur residual component of the image. 12. The blur correction device according to claim 10, wherein blurring of the image caused by the aberration of the system is reduced. 13. The blur correction device according to any one of claims 7 to 12, wherein the correction means includes an image sensor of the image pickup device or a blur correction lens included in the optical system. a blur correction device according to any one of claims 1 to 13;
imaging means for capturing the image;
An imaging device comprising: A blur correction method executed by a blur correction device,
a detection step of detecting blurring of the imaging device;
a driving step of driving a correction means so as to reduce blurring of the image based on blurring of the imaging device while the image is being captured by the imaging device;
an obtaining step of obtaining a residual blur component of the image based on the driving amount of the correction means and the distortion aberration of an optical system used to capture the image;
an image processing step of performing image processing to reduce the residual blur component of the image;
A blur correction method comprising: A blur correction method executed by a blur correction device,
a detection step of detecting blurring of the imaging device;
a driving step of driving a correction means so as to reduce blurring of the image based on blurring of the imaging device while the image is being captured by the imaging device;
an acquisition step of acquiring correction data for reducing a residual blur component of the image based on the driving amount of the correction means and the distortion aberration of an optical system used to capture the image;
an image processing step of performing image processing for reducing the residual blur component on the image based on the correction data;
A blur correction method comprising: A program for causing a computer to function as each means of the blur correction device according to any one of claims 1 to 6. A program for causing a computer to function as each means of the blur correction device according to any one of claims 7 to 13.

Images