JPH1124122A - Method and device for correcting camera shake image, and recording medium with recorded program for executing the same method by computer and capable of being read by computer - Google Patents

Abstract

PROBLEM TO BE SOLVED: To minimize an error in the correction of a camera shake image. SOLUTION: The fluctuation of the optical axis of a camera is detected by angular acceleration sensors 1 to 3 so as to find an angular velocity by integrating the detected data by a CPU 6. The secular change of the aperture of a shutter is detected by a shutter opening/closing sensor 4. Subsequently, a transfer function expressing a shake state in photographing is acquired from the angular velocity (bearing locus) and the secular change of the aperture of the shutter. Then, the picked-up image data is fetched from a memory 7, and the transfer function is inversely converted with respect to the image so as to correct the image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a camera shake image correction method and a camera shake image correction apparatus capable of performing a natural correction by reducing an error when correcting an image in which a camera shake has occurred. The present invention relates to a computer-readable recording medium on which a program to be executed is recorded.

[0002]

2. Description of the Related Art When taking a picture with a camera, the user normally presses a shutter while holding the camera body in his hand. At this time, the camera body often moves in conjunction with the operation of pressing the shutter, and as a result, the subject is photographed with "blurring". Usually, such image deterioration is noticed at the time of image reproduction, so that the user will regret later. The present invention provides a method capable of reproducing a high-quality image even in a situation where only such a deteriorated image remains.

Conventionally, many methods for solving such a problem have been proposed. Among them, the most practical method is as follows.
Murat Tekalp, et al., “Identification of Imag”
e and Blur Parameters for the Restoration of Nonca
usal Blurs ”, IEEE Transactions on Acoustics, Spee
ch and Signal Processing, Vol. ASSP-34, No.4, pp.9
63 (1989).

In this method, a photographed image is displayed in a frequency space, and the presence or absence, the direction, and the magnitude of the blur are determined based on the distribution shape.

[0005]

However, the above method has two problems. First, a transfer function (or a point spread function; PSF (Poin
t Spread Function) is not exactly known.
In order to restore a blurred image, it is necessary to calculate a restoration filter for performing an inverse transformation of the transfer function. For that purpose, the PSF (transfer function) must be known.

Since the PSF cannot be obtained in normal photographing, the PSF is estimated. The most commonly used estimation method considers "blur" due to camera shake to be equivalent to simple averaging of a fixed number of pixels. The PSF in this case can be represented by a flat box function. The restoration filter at this time is a function sin
It is represented by (x) / x. However, since the estimation itself often does not hold, the original image restored by such a restoration filter contains a considerable error.

Second, camera shake occurs only when the shutter of the camera is open. During the period when the shutter is open, whether camera shake occurs in only one direction or in multiple directions. In addition, there is no means for knowing what the trajectory was. For this reason, conventionally, the direction of camera shake is estimated to be only one direction, and such a direction has been obtained. However, since the movement of the optical axis of the photographing lens system due to camera shake is not necessarily a straight line, an estimation error is included as described above.

In view of the above, the present invention has been made in view of the above, and has a method and apparatus for correcting a camera shake image, which can accurately determine a PSF and a camera shake direction and can reduce an image error. It is an object of the present invention to provide a computer-readable recording medium on which a program for causing a computer to execute the above is recorded.

[0009]

According to a first aspect of the present invention, there is provided a method for correcting a camera shake image, comprising the steps of: obtaining a change in a camera optical axis as an angular acceleration; A procedure for acquiring an angular velocity that becomes an azimuth locus of a camera optical axis by integrating with an axis, a procedure for acquiring a change over time of an aperture of a shutter of a camera, and , A procedure for obtaining a transfer function representing a blurred state at the time of shooting, and a procedure for performing an inverse conversion of the transfer function on a shot image.

According to a second aspect of the present invention, there is provided a camera shake image correcting method for obtaining a change in a camera optical axis as an angular velocity which is an azimuth trajectory of the camera optical axis and a procedure for obtaining a temporal change in the aperture of a shutter of the camera. From the azimuth trajectory of the optical axis and the change over time of the opening of the shutter, a step of obtaining a transfer function representing a blurred state at the time of shooting, and a step of performing an inverse transform of the transfer function on the shot image, Is included.

According to a third aspect of the present invention, in the above-mentioned camera shake image correcting method, the fluctuation of the camera optical axis is obtained from a predetermined time or more before the shutter of the camera is pressed, and the obtained information is obtained. Based on the fluctuation of the camera optical axis, the temporal origin of the fluctuation is determined.

According to a fourth aspect of the present invention, there is provided a camera shake image correcting apparatus, comprising: an optical axis variation acquiring unit for acquiring a variation of a camera optical axis; and a shutter opening temporal change acquiring unit for acquiring a temporal change of a shutter of a camera. And a data storage unit for temporarily storing data obtained by the optical axis fluctuation obtaining unit and the shutter opening temporal change obtaining unit, and a transfer function representing a blur state at the time of shooting from the data stored in the data storage unit. Image correction means for performing an inverse conversion of the transfer function on the acquired and photographed image.

According to a fifth aspect of the present invention, there is provided a computer-readable recording medium storing a program for causing a computer to execute the above-mentioned camera shake image correction method.

[0014]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below in detail with reference to the drawings. The present invention is not limited by the embodiment.

FIG. 1 is a schematic configuration diagram showing the configuration of a camera shake image correction apparatus according to the present invention. The camera shake image correction apparatus 100 includes a Z-axis angular acceleration sensor 1 for detecting a change in the optical axis direction of a camera lens when the CCD surface with respect to the optical axis (Z axis) is an XY plane, and a X-axis direction change. An X-angle acceleration sensor 2 for detecting a situation, a Y-angle acceleration sensor 3 for detecting a variation state of a Y-axis azimuth, a shutter opening / closing sensor 4 for detecting opening / closing of a shutter, and a buffer memory 5 for storing a sensor output for a certain period of time. The CPU 6 is configured to calculate a PSF and a camera-shake direction based on detection signals from the respective sensors to correct an image, and a memory 7 for storing a captured image.

As described above, this camera shake image correction apparatus 1
00 has at least three components of angular acceleration sensors (1 to 3) for detecting the variation of the optical axis azimuth of the camera lens K, and two of the three components (angular acceleration sensors 2 and 3).
3) corresponds to an orthogonal vector component in the plane of the CCD 101. These three angular acceleration components are represented by variables of θ, Φ, and ε, as shown in FIG.

FIG. 3 is a flowchart showing the procedure for implementing the camera shake correction method according to the present invention. [Steps S301 and S302] In step S301, a captured image is read from the memory 7. Step S
At 302, the camera shake amount is determined. If the amount of camera shake does not matter, the image is not corrected.

[Step S303] In step S303, data of related sensors such as the X angular acceleration sensor 2 and the Y angular acceleration sensor 3 are read, and the PSF is calculated. First, if the time origin of the function representing the movement of the optical axis of the camera is defined, the angular velocity can be obtained by integrating the output signals of the respective angular acceleration sensors on the time axis. That is, the azimuth locus of the optical axis can be obtained.

FIG. 4 is a graph showing the position of the optical axis that changes with time. In this way, the position of the optical axis changes to positions p1, p2, p3, p4,... Due to camera shake. In the drawing, the vector before the vector p1 (left side of the broken line) represents the position of the optical axis before the time when the shutter is pressed.

As shown in FIG. 5, aperture information from the start of opening the shutter to the closing thereof is known at the time of design as a characteristic unique to the camera. The shutter pressed at time t0 opens in proportion to time, and at time t0
1 opens fully. Then, a preset time (t2
The opening is continued for -t1). Thereafter, the shutter starts closing at time t2, and is completely closed at time t3.

FIG. 6 is a graph showing changes in the aperture ratio of the shutter. After pressing the shutter, the vector p1
The aperture ratio reaches 100% at time t1 corresponding to
It is completely closed at time tp after 2, t3.

Next, based on the azimuth locus of the optical axis and the change with time of the aperture of the shutter, the "point spread function;
F "can be calculated. This PSF is represented by the formula [shutter opening time series pulse] * [shutter opening function], and the two-dimensional PSF expressed along the locus of the optical axis azimuth when the shutter is opened is “camera shake”. It becomes an expression.
In the above equation, the convolution operation is indicated by the symbol *.
Actually, since it is necessary to consider the angular acceleration around the optical axis, the three-dimensional expression is used (symbol ε in FIG. 1). This corresponds to a rotational movement about the center of the image.

Next, how to determine the origin of the trajectory of the optical axis direction will be described. Since the original purpose is to detect camera shake, it is desirable to set the origin when the camera is stationary. However, the output of the angular acceleration sensor is "0" at the time of constant speed movement. Therefore, other mechanisms or assumptions are needed to estimate the stationary state.

As an example, there is a method of using information of a moving image continuously shot. That is, if the image information of each frame of the moving image is constant (if there is no change in time), it is determined that the moving image is stationary, and this is set as the origin.

As another example, there is a method using a general operation characteristic of the shutter operation. That is, when photographing with a camera, it is a general human action to stop for a while after aiming at the subject. Then, the user presses the shutter after aiming at the subject, and the camera shake is induced by the operation of pressing the shutter at this time. Therefore,
A few seconds (usually 1-2 seconds) just before pressing the shutter,
You can imagine it's time to aim. In general, camera shake when aiming is small. The time at which the shutter is pressed can be easily detected by a shutter open / close sensor. And from the time when the shutter is pressed, 1
Going back about 2 seconds, the output average value of each angular acceleration sensor at that time is defined as the origin.

For example, the origin of FIG. 4 is determined not only by the signal after the time t0 (broken line) when the shutter is pressed, but also by the time t0.
(Period tp). Such a signal is temporarily stored in the buffer memory 5.

[Step S304] In step S304, "blur" of the image is corrected. The correction is performed by a conventionally known method for calculating the inverse transform filter of the PSF equation or the inverse transform process.

The trajectory in FIG. 4 can be regarded as a set of a plurality of linear vectors. Therefore, the “blur” due to the camera shake can be corrected by sequentially applying the conventional correction method of the camera shake image to each of the constituent vectors. That is, the angular acceleration around the (X, Y) axis is
It is corrected in time series as an element vector on the (X, Y) plane.

Further, since the rotational movement about the optical axis corresponds to the rotational movement (ε) around the center of the image, a blur amount proportional to the distance from the center of the pixel of interest may be assumed. Therefore, the rotational movement about the Z axis is expressed in polar coordinates with the origin at the center of the image, and the same correction may be made only for the rotational axis. Normally, the correction of the blur accompanying the rotational motion is performed after the blur correction of the other two components is completed.

In the above description, the angular velocity is obtained by calculating the angular acceleration by the angular acceleration sensor and integrating the obtained angular acceleration on the time axis. However, the angular velocity is directly measured by using a gyro sensor or the like. You may do so.

[0031]

As described above, according to the camera shake image correction method of the present invention (claim 1), the fluctuation of the camera optical axis is obtained as angular acceleration, and the obtained angular acceleration is integrated on the time axis. Obtain the angular velocity that is the azimuth locus of the camera optical axis. In addition, the change with time of the opening of the shutter of the camera is acquired. Then, a transfer function representing a blurred state at the time of shooting is acquired from the azimuth locus of the optical axis and the change with time of the opening of the shutter, and the transfer function is inversely transformed for the shot image. For this reason, the fluctuation of the camera optical axis can be obtained in detail, and the error of the corrected image can be reduced.

According to a second aspect of the present invention, there is provided a camera-shake image correction method in which a change in a camera optical axis is obtained as an angular velocity corresponding to an azimuth locus of the camera optical axis, and a temporal change in the aperture of a shutter of the camera is obtained. Then, a transfer function representing a blurred state at the time of shooting is acquired from the azimuth locus of the optical axis and the change with time of the opening of the shutter, and the transfer function is inversely transformed for the shot image. For this reason, the fluctuation of the camera optical axis can be obtained in detail, and the error of the corrected image can be reduced. Further, since the angular velocity is directly obtained, the structure can be simplified.

In a third aspect of the present invention, there is provided a camera shake image correction method in which a fluctuation of a camera optical axis is acquired from a predetermined time or later before a shutter of a camera is pressed, and the acquired fluctuation of the camera optical axis is obtained. , The temporal origin of the fluctuation is determined, so that the state in which the camera is stationary can be estimated with high accuracy. As a result, the error of the image correction can be further reduced.

The camera shake image correcting apparatus according to the next invention is an optical axis fluctuation acquiring means for acquiring each variation of the optical axis of a camera, and a shutter opening temporal change for acquiring a temporal change of a shutter of the camera. An acquisition unit, a data storage unit for temporarily storing data acquired by the optical axis variation acquisition unit and a shutter opening temporal change acquisition unit, and a transmission indicating a blur state at the time of shooting from the data stored in the data storage unit. Image correction means for acquiring a function and performing the inverse conversion of the transfer function on the captured image, so that the fluctuation of the camera optical axis can be obtained in detail, and the error of the corrected image can be reduced. Become like

A computer-readable recording medium according to the present invention (claim 5) stores a program for causing a computer to execute the above-described camera shake image correction method, so that the camera shake image can be corrected on the computer. it can.

[Brief description of the drawings]

FIG. 1 is a configuration diagram showing a camera shake correction apparatus according to the present invention.

FIG. 2 is a schematic diagram showing an angular acceleration component.

FIG. 3 is a flowchart showing a procedure for implementing a camera shake correction method according to the present invention.

FIG. 4 is a graph showing the position of the optical axis that changes with time.

FIG. 5 is a graph showing an opening state of a shutter.

FIG. 6 is a graph showing a change in shutter aperture ratio.

[Description of Signs] 100 Camera shake image correction device 1 Z axis angular acceleration sensor 2 X angular acceleration sensor 3 Y angular acceleration sensor 4 Shutter open / close sensor 5 Buffer memory 6 CPU 7 Memory

Claims (5)

[Claims] A step of acquiring a variation of a camera optical axis as an angular acceleration; a step of integrating the acquired angular acceleration on a time axis to acquire an angular velocity which is an azimuth locus of the camera optical axis; Acquiring a change over time in the aperture of the shutter; acquiring a transfer function representing a blur state at the time of shooting from the trajectory of the azimuth of the optical axis and the change over time of the opening of the shutter; A camera shake image correction method, comprising: performing a function inverse transformation. A step of acquiring a change in the optical axis of the camera as an angular velocity serving as an azimuth locus of the camera optical axis; a procedure of acquiring a change over time of an opening of a shutter of the camera; an azimuth locus of the optical axis and the shutter; A camera shake image correction method, comprising: obtaining a transfer function representing a blurred state at the time of shooting from the change in aperture with time; and performing an inverse transform of the transfer function on the shot image. . 3. A camera optical axis variation is acquired from a predetermined time or later before the shutter of the camera is pressed, and a temporal origin of the variation is determined based on the acquired camera optical axis variation. 3. The method according to claim 1, wherein the image is corrected. 4. An optical axis fluctuation obtaining means for obtaining each fluctuation of a camera optical axis, a shutter opening time change obtaining means for obtaining a time change of a shutter opening of the camera, an optical axis fluctuation obtaining means, and a shutter opening time. A data storage unit for temporarily storing data obtained by the change obtaining unit; and a transfer function representing a blur state at the time of shooting from the data stored in the data storage unit, and transmitting the transfer function to the shot image. An image stabilizing apparatus comprising: an image correcting unit that performs an inverse conversion of a function. 5. A computer-readable recording medium storing a program for causing a computer to execute the method according to claim 1. Description: