JP4345940B2  Camera shake image correction method, recording medium, and imaging apparatus  Google Patents
Camera shake image correction method, recording medium, and imaging apparatus Download PDFInfo
 Publication number
 JP4345940B2 JP4345940B2 JP10503399A JP10503399A JP4345940B2 JP 4345940 B2 JP4345940 B2 JP 4345940B2 JP 10503399 A JP10503399 A JP 10503399A JP 10503399 A JP10503399 A JP 10503399A JP 4345940 B2 JP4345940 B2 JP 4345940B2
 Authority
 JP
 Japan
 Prior art keywords
 camera shake
 image
 blur correction
 point spread
 function
 Prior art date
 Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
 Expired  Fee Related
Links
 238000003702 image correction Methods 0.000 title claims description 25
 238000003384 imaging method Methods 0.000 title claims description 25
 230000001131 transforming Effects 0.000 claims description 45
 239000011159 matrix materials Substances 0.000 claims description 44
 230000000875 corresponding Effects 0.000 claims description 27
 238000004364 calculation methods Methods 0.000 claims description 4
 230000003287 optical Effects 0.000 description 12
 238000000034 methods Methods 0.000 description 11
 230000001133 acceleration Effects 0.000 description 9
 238000010586 diagrams Methods 0.000 description 6
 230000015556 catabolic process Effects 0.000 description 4
 230000004059 degradation Effects 0.000 description 4
 238000006731 degradation reactions Methods 0.000 description 4
 238000007781 preprocessing Methods 0.000 description 4
 238000005311 autocorrelation function Methods 0.000 description 2
 238000007430 reference method Methods 0.000 description 2
 281000027677 Western Digital companies 0.000 description 1
 230000001276 controlling effects Effects 0.000 description 1
 230000000694 effects Effects 0.000 description 1
 239000004973 liquid crystal related substances Substances 0.000 description 1
 238000005457 optimization Methods 0.000 description 1
 230000004044 response Effects 0.000 description 1
 230000001340 slower Effects 0.000 description 1
Images
Description
[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a technique for correcting image degradation due to camera shake.
[0002]
[Prior art]
When a subject is photographed with a digital camera or the like, camera movement during exposure, that is, blur due to camera shake occurs in the photographed image. Various techniques have been developed in the past to avoid image degradation due to camera shake.
[0003]
For example, the camera is equipped with an acceleration sensor or the like for detecting the movement of camera shake, the detection signal of the sensor is converted into an image movement, and the camera optical system (prism, A method of mechanically moving a lens or an image pickup device is known (for example, Japanese Patent No. 2637464).
[0004]
The movement of a virtual pointlike subject on the image due to camera shake corresponds to blurring, and is often called a point spread function. Also known is a method for correcting camera shake by estimating the point spread function from the change over time in the aperture ratio of the shutter at the time of shooting and the movement of the camera optical axis detected by an acceleration sensor, etc., and applying the inverse correction to the entire image. (JPA7226905).
[0005]
Furthermore, research for estimating camera shake from an image without using an acceleration sensor or the like has been made for a long time. For example, in the document “A. About Alib and L. M. Silverman; Rotation of Motion Degraded Images; IEEE Trans. Circuits and Systems, p. Are listed. Also, in the direction of camera shake, if the camera shake can be approximated to be uniform, the density should be uniform (brightness). Therefore, taking the autocorrelation function of the image in the camera shake direction and differentiating this will yield the amount of camera shake. The differential value shows a minimum at a corresponding interval and shows a maximum at the zero point in the direction without blurring, so that it is possible to estimate the magnitude of the blur, “Y. Yitzhaky and NS Kopeika; Identification of Blur Parameters from Motion Blurred Images; Graphical Models and Image Processing Vol.59, pp.310320, 1997 ". Regarding estimation of the blur direction, for example, the document “R. Fabian and D. Malah; Robust Identification and OutOfFocus Blur Parameters from Blurred and Noisy Images; Graphical Models and Image Processing, Vol. 53, pp. 403412, 1991 ”describes a method using a spatial frequency distribution.
[0006]
[Problems to be solved by the invention]
When the camera rotates around the optical axis at the time of shooting, camera shake on the image becomes nonuniform. It is an object of the present invention to provide a camera shake that can restore a sharp image without mechanical operation as in the abovementioned Japanese Patent No. 2637464 even when such a camera shake including uneven rotation and translation occurs. An image correction method and an imaging apparatus are provided. Another object of the present invention is a camera shake image correction method capable of accurately correcting nonuniform camera shake even with respect to an image taken with an ordinary camera that does not have an acceleration sensor or the like for detecting shake of an optical axis. Is to provide. Another object of the present invention is to provide an imaging apparatus capable of obtaining an image in which nonuniform camera shake is accurately corrected without providing an acceleration sensor or the like for detecting optical axis shake.
[0007]
The speed of camera shake is not always uniform from the beginning to the end of camera shake. As the speed of camera shake changes with time, the shape of the point spread function also changes. Another object of the present invention is to provide a camera shake image correction method and an imaging apparatus capable of appropriate camera shake correction even when the camera shake speed fluctuates with time.
[0008]
In some cases, such as Japanese Patent Application LaidOpen No. 7226905, which presupposes that image data that directly reflects a change in the amount of light due to a change in the aperture ratio of the shutter as it is, good camera shake correction may not be expected. This problem is particularly noticeable in digital cameras that have become widespread in recent years. In a general digital camera, an optical image formed by a lens system is converted into an electric signal by an image pickup device such as a CCD. This electric signal is converted to an external brightness or color deviation (red in a clear sky evening) Because it is corrected according to the shooting environment conditions such as strong blue and strong blue under fluorescent lamps), it is not possible to obtain image data that directly reflects the change in light quantity due to the change in shutter aperture over time. is there. Another object of the present invention is to provide a camera shake image correction method and an imaging apparatus capable of performing appropriate camera shake correction even when correction according to the shooting environment conditions described above is performed. .
[0009]
[Means for Solving the Problems]
According to the first to seventh aspects of the present invention, an improved camera shake image correction method is provided. Its characteristics are as follows.
[0010]
A camera shake image correction method according to the first aspect of the present invention includes:
A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated in the second step at the position;
Among the plurality of blur correction functions corresponding to the point spread function of a plurality of typical camera shake patterns including a camera shake pattern with a nonuniform camera shake speed prepared in advance, the blur correction function is used in the third step. Including a fourth step in which a user selects a blur correction function.
[0011]
A camera shake image correction method according to a second aspect of the present invention includes:
A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated in the second step at the position;
A fourth step of estimating a point spread function due to camera shake of the target image;
A fifth step of selecting a point spread function most similar to the point spread function estimated in the fourth step from a plurality of typical point spread functions prepared in advance;
A sixth step of selecting a blur correction function corresponding to the typical point spread function selected in the fifth step from a plurality of blur correction functions prepared in advance.
Including
The blur function selected in the sixth step is used for blur correction in the third step.
[0012]
According to a third aspect of the present invention, there is provided a camera shake image correction method.
A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated by the second step at the position;
A fourth step of displaying a graph of a point spread function prepared in advance on a display screen;
A fifth step of editing the point spread function graph displayed on the screen according to an operation of a pointing device;
Corresponding to the point spread function graph after editing in the fifth step, a sixth step of correcting a blur correction function used for blur correction in the third step is included.
[0013]
According to a fourth aspect of the present invention, there is provided a camera shake image correction method.
A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated in the second step at the position;
A fourth step of displaying a graph of a point spread function due to camera shake of the target image on a display screen;
A fifth step of editing the point spread function graph displayed on the screen according to an operation of a pointing device;
A sixth step of generating a blur correction function corresponding to the point spread function after editing in the fifth step;
The blur correction function generated in the sixth step is used for blur correction in the third step.
[0014]
According to a fifth aspect of the present invention, in the camera shake image correction method according to any one of the first to fourth aspects, the first step includes a direction of camera shake at a plurality of positions on the target image, and Including a step of estimating the size based on the target image, and based on the direction and size of the camera shake estimated by the step, a coordinate transformation matrix representing rotation and translation of the pixel on the target image due to the camera shake Is to decide.
[0015]
According to a sixth aspect of the present invention, in the camera shake image correction method according to any one of the first to fifth aspects, blur correction is performed in the third step by an interpolation operation along a camera shake vector. It is to be.
[0016]
According to the eighth to thirteenth aspects of the present invention, there is provided an improved image pickup apparatus that picks up a subject with an image pickup device and obtains a digital image. Its characteristics are as follows.
[0017]
An image pickup apparatus according to the invention described in claim 8 provides:
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital target image according to the hand movement vector estimated by the second means at the position;
Among the plurality of blur correction functions prepared in advance and corresponding to the point spread function of a plurality of typical camera shake patterns including a camera shake pattern having a nonuniform camera shake speed, the third means uses the blur correction function in the third means. Fourth means for the user to select a blur correction function for
It is characterized by including.
[0018]
An image pickup apparatus according to a ninth aspect of the present invention provides:
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital image according to the hand movement vector estimated by the second means at the position;
A fourth means for estimating a point spread function due to camera shake in a digital image;
A fifth means for selecting a point spread function most similar to the point spread function estimated by the fourth means from a plurality of typical point spread functions prepared in advance;
Sixth means for selecting a blur correction function corresponding to the typical point spread function selected by the fifth means from a plurality of blur correction functions prepared in advance.
Including
The blur function selected by the sixth means is used for blur correction in the third means.
[0019]
An imaging apparatus according to the invention of claim 10 is provided.
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital image according to the hand movement vector estimated by the second means at the position;
A fourth means for displaying a graph of a point spread function prepared in advance on a display screen;
A fifth means for editing the graph of the point spread function displayed on the screen according to an operation of a pointing device;
Corresponding to the point spread function graph after editing by the fifth means, sixth means for correcting the blur correction function used for blur correction in the third means
It is characterized by including.
[0020]
An imaging device according to the invention of claim 11 is provided.
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital image according to the hand movement vector estimated by the second means at the position;
A fourth means for displaying a graph of a point spread function due to camera shake of a digital image on a display screen;
A fifth means for editing the graph of the point spread function displayed on the screen according to an operation of a pointing device;
Sixth means for generating a blur correction function corresponding to the point spread function after editing by the fifth means
Including
The blur correction function generated by the sixth means is used for blur correction in the third means.
[0021]
According to a twelfth aspect of the present invention, in the imaging apparatus according to any one of the eighth to eleventh aspects, the first means includes a direction and a size of camera shake at a plurality of positions on the digital image. Is determined based on the digital image, and a coordinate transformation matrix representing rotation and translation of the pixel on the digital image due to the camera shake is determined based on the direction and size of the camera shake estimated by the means. That is.
[0022]
According to a thirteenth aspect of the present invention, in the imaging apparatus according to any one of the eighth to twelfth aspects, blur correction is performed by an interpolation operation along a camera shake vector in the third means. .
[0023]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 shows an example of a processing procedure for camera shake image correction according to the present invention. In FIG. 1, 100 is a step in which an image in which blur is caused by camera shake is input, 101 is a step in which a coordinate transformation matrix representing rotation and translation of pixels on the input image due to camera shake is determined, and 102 is the coordinates A step of estimating a camera shake vector (direction and magnitude of “blurring” due to camera shake) at each pixel position on the input image using the transformation matrix, and 103 for each pixel value on the input image A step in which blur correction according to the estimated camera shake vector is performed and an image in which blur due to camera shake is corrected is generated, and 104 is a step in which a corrected image is output. Note that the camera shake image correction processing procedure according to the present invention can be variously changed as will be described later.
[0024]
<< Determination of coordinate transformation matrix: Step101》
Step101The specific processing content of will be described.
Pixels on the image move from their original positions due to camera shake when taking an image. Since such minute blur can be expressed by affine transformation, the following equation is obtained between the original pixel coordinate value (x, y) and the coordinate value (X, Y) changed by the camera shake. It holds.
[0025]
[Expression 1]
Where θ is the rotation angle and σ_{x}, Σ_{y}Represents the amount of parallel movement in the x and y directions. Since the rotation angle θ due to camera shake is generally sufficiently small, the equation (1) may be approximated as in the equation (2) or may be further approximated as in the equation (3).
[0026]
[Expression 2]
[Equation 3]
[0027]
Step101In the three parameters (θ, σ) of the coordinate transformation matrix of the formula (1), (2) or (3)_{x}, Σ_{y}). When taking an input image with a camera and using an acceleration sensor, a gyro sensor or the like to detect the shake amount of the optical axis of the camera and the rotation angle around the optical axis, the detection result is, for example, a step. 100 is input as additional information of the image, and based on the additional information, three parameters (θ, σ_{x}, Σ_{y}) Can be determined. According to one embodiment of the present invention, such a method is employed.
[0028]
However, when a general camera not equipped with such a sensor is used, the abovedescribed method cannot be used. Therefore, according to another embodiment of the present invention, a set of (x, y) and (X, Y) at two or more places on the input image is estimated, in other words, in FIG. By estimating the direction and magnitude (shake vector) of the camera shake as shown by B3 and substituting it into the formula (1), (2) or (3), the parameters (θ, σ) of the coordinate transformation matrix are calculated._{x}, Σ_{y}). An example of a specific processing procedure will be described below.
[0029]
First, select at least two appropriate points on the input image. This may be automatically performed by a processing program, or may be designated by a user using a pointing device on an appropriate display screen on which an image is displayed.
[0030]
Next, the direction of camera shake at each selected point is estimated. For example, the spatial frequency distribution is measured in the vicinity of each selected point, and the direction in which the reduction rate of the high frequency distribution is maximum is estimated as the direction of camera shake. Such a method is described in detail in Japanese Patent Application No. 9180846 (Japanese Patent LaidOpen No. 11027574) or the abovementioned R. Fabian et al.
[0031]
Next, the magnitude of camera shake at each selected point is estimated. For example, taking the autocorrelation function of the image in the direction of the estimated camera shake for the region near each selected point and differentiating it in the direction of the camera shake, the differential value is minimal at intervals corresponding to the size of the camera shake. Therefore, the size of the shake is estimated from the interval between the local minimum points. This method is detailed in the abovementioned literature by Y. Yitzhaky et al.
[0032]
When such estimation is finished, the x and y coordinate values (x, y) of each selected point and the x and y coordinate values of the point moved by the estimated camera shake direction in the estimated camera shake direction. By substituting the pair with (X, Y) into the above formula (1), (2) or (3), the parameters (θ, σ_{x}, Σ_{y}) Thus, the coordinate transformation matrix of the formula (1), the formula (2) or the formula (3) is determined.
[0033]
The input image is enlarged and displayed on an appropriate display screen, and the user determines whether the image is shaken due to camera shake on the screen, and uses a pointing device to find the appropriate two or more points and the points moved by the shake. Directly instructing and using the indicated coordinate values, parameters (θ, σ_{x}, Σ_{y}) Is also possible. Even with this simple method, a skilled user can determine parameters with reasonable accuracy.
[0034]
<< Estimation of Shake Vector at Each Pixel Position: Step 102 >>
Next, in step 102, the hand movement vector B (x, y) (motion vector due to hand movement) at each pixel position (x, y) on the input image is estimated using the coordinate transformation matrix determined in step 101. That is, (X, Y) is calculated by substituting (x, y) into the equation (1), (2) or (3), and B (x, y) is obtained by the following equation.
[0035]
[Expression 4]
[0036]
<< Bokeh correction: Step 103 >>
In step 103, each pixel value of the input image is subjected to blur correction according to a camera shake vector whose position is estimated, and a corrected image in which blur due to camera shake is compensated is generated.
[0037]
When camera shake is introduced by uniform movement, blur due to camera shake is expressed by a onedimensional point spread function as shown in Equation (5), and blur due to such camera shake is as shown in Equation (6). It is known that correction can be performed with a correction function. In these two formulas, t is the distance of camera shake. a is the length of the blur, which corresponds to the length of the hand movement vector. However, if the camera shake speed is not uniform, the point spread function and the blur correction function will be slightly different. For more accurate camera shake correction, select or correct the blur correction function according to the camera shake pattern. It is desirable to be able to do this, but this will be described later.
[0038]
[Equation 5]
[Formula 6]
[0039]
According to one embodiment of the present invention, each pixel (i_{c}, J_{c}) For the corrected pixel value V (i) by performing a onedimensional interpolation operation along the hand movement vector B at that position._{c}, J_{c}). In particular
[Expression 7]
In the equation, θ is an angle formed with respect to the x axis of the shake vector B. In addition, the following relationship holds.
[Equation 8]
S is a pixel in a blurred range, and a_{x}, A_{y}Is the length of the x and y components of the hand movement vector B, the equation (7) can be written as follows.
[Equation 9]
[0040]
P () in the equations (7) and (9) is an interpolated pixel value on the camera shake vector B or an extension line thereof calculated from the previous and next pixel values along the camera shake vector B, and is expressed as (9) By substituting into the equation, the corrected pixel value V (i_{c}, J_{c}) Is obtained. Interpolation function
I (x) = h^{1}(X)
Is assumed to hold.
[0041]
If only the pixel adjacent to the target pixel is used, the interpolation function I (x) is calculated from the pixels on both sides along the camera shake vector B as indicated by the symbol “*” in FIG. Here, the interpolated pixel value p () is the value of the virtual pixel indicated by the symbol “=” in FIG. This pixel value can also be obtained by the interpolation function I (x) (in this case, a = 1). The values of x and y may generally be arbitrary. However, in order to perform efficient calculation, the pixel pitch is set to 1,
[Expression 10]
(FIG. 3 shows such a case). Therefore, equation (9) can be rewritten as follows.
## EQU11 ##
If only pixels on both sides of the hand movement vector B are used, the conditional expression i, j A
Can be expressed as:
[Expression 12]
In the formula, [x] represents an integer not exceeding x.
[0042]
The abovedescribed camera shake image correction method according to the present invention can perform highly accurate camera shake correction by real space processing even when there is uneven camera shake (rotation and translation) in an image. Since preprocessing and postprocessing such as image rotation are not included, image quality degradation associated with such processing does not occur. In addition, since such preprocessing and postprocessing are not included, the interpolation pixel value along the camera shake vector can be calculated at high speed by the table reference method, so that highspeed processing as a whole is possible.
[0043]
Now, the point spread function of equation (5) is represented as a graph as shown in FIG.ThisYou can. In FIG. 4, the horizontal axis represents the blur distance t, and the vertical axis represents the response magnitude. Actual camera shake often has a nonuniform velocity, and the point spread function is slightly different from the graph shown in FIG. However, camera shake patterns can be classified into several types. Graphs of point spread functions corresponding to some typical camera shake patterns are shown in FIGS. 4 (C), (D) and (E). The graph in (C) represents a point spread function corresponding to a blurring pattern that is slow and gradually fast at first, and the graph in (D) is a point spread function corresponding to a blurring pattern that is fast at first and gradually slows down. To express. The graph of (E) represents a point spread function corresponding to a camera shake pattern in which the beginning and the end are slow and the motion is fast in the middle. Once you get used to it, you can figure out what kind of handshake pattern it is, with considerable accuracy, by observing the image.
[0044]
According to one embodiment of the present invention, a blur correction function corresponding to a point spread function of a plurality of typical camera shake patterns such as (A), (C), (D), and (E) in FIG. A blur correction function that is prepared in the form of a table and is used for blur correction is selected by the user. In the context of the processing flow shown in FIG. 1, the step for selecting such a blur correction function is placed inside or before the blur correction processing step 103.
[0045]
According to another embodiment of the present invention, such a blur correction function is automatically selected. An example of the procedure for this is shown in FIG.
[0046]
In step 200 of FIG. 5, a point spread function due to camera shake of the input image is estimated. When a camera equipped with an acceleration sensor to detect optical axis shake is used to capture an input image, the sensor detects a change in the optical axis over time and superimposes it with the shutter release timing of the camera. Thus, the point spread function of the input image can be estimated directly. When such a camera is not used, a point spread function is estimated from the input image. Specifically, it is only necessary to observe a density (luminance) distribution along the tail image formed by the blurring of one point on the subject. For more accurate estimation, for example, the method described in the abovementioned Y. Yitzhaky et al. Paper is used for a highcontrast portion (usually called an edge) adjacent to a smooth background portion in an image. do it.
[0047]
In the next step 201, the one most similar to the point spread function estimated in the previous step is selected from the point spread functions corresponding to a plurality of typical hand movement patterns prepared in advance. Distance can be used as a criterion for this selection. For example, the following distance D_{j}Has the smallest point spread function h_{j}(T) is selected.
[Formula 13]
Where h (t) is the estimated point spread function, h_{j}(T) is a point spread function prepared in advance.
[0048]
A plurality of blur correction functions corresponding to respective point spread functions of typical hand movement patterns prepared in advance are also prepared in advance. In step 202, a blur correction function corresponding to the point spread function selected in the previous step is selected from the prepared blur correction functions. The blur correction function selected in this way is used in the blur correction processing step 103 in FIG.
[0049]
According to an embodiment of the present invention, a user can correct a blur correction function for use in blur correction processing by checking a point spread function on a display and performing necessary editing. The procedure is shown in FIG.
[0050]
In step 300 of FIG. 6, a point spread function prepared in advance is displayed on an appropriate display screen. In step 301, the user makes necessary edits to the point spread function on the screen using an appropriate pointing device. For example, when a point spread function graph as shown in FIG. 4A is displayed, the graph can be edited as shown in FIG. 4B using a pointing device, for example. . Note that the point spread function displayed in step 300 may be selected by the user from a plurality of point spread functions.
[0051]
In step 302, the blur correction function prepared in advance is corrected so as to reflect the edited content of the point spread function in the previous step. Thus, the corrected blur correction function is used in the blur edit processing step 103 in FIG.
[0052]
Another embodiment of the present invention will be described with reference to FIG. In step 300, a point spread function due to camera shake of the input image is displayed graphically. Specifically, for example, by projecting the luminance in a direction orthogonal to the blur direction for a uniform background portion in the input image, a graph as shown in FIG. 4 can be directly obtained, so that it is displayed on an appropriate display. To do. Since the point spread function measured in this way often involves measurement errors, in step 301, the user makes necessary corrections to the point spread function on the screen of the display using a pointing device. In step 302, a blur correction function corresponding to the corrected point spread function is generated. The generated blur correction function is used for the blur correction process.
[0053]
The present invention includes a CPU 400 as shown in FIG. 7, a memory 401, a display 402, a pointing device 403 such as a mouse, a drive 405 for a recording medium 404 such as a CDROM, an auxiliary storage device 406 such as a hard disk, and an external interface. It can be implemented by software using a computer configured such that 407 and the like are interconnected by a system bus 409. The program 410 for the processing described in relation to FIG. 1, FIG. 5 and FIG. 6, and the table 411 of the point spread function and blur correction function as described in relation to FIG. 5 or FIG. The data is read into the memory 401 by the drive 405 from the recorded recording medium 404. Alternatively, the program 410 and the table 411 are temporarily stored in the auxiliary storage device 406 and are read from the auxiliary storage device 406 into the memory 401 at the time of processing execution. For example, the image data to be processed is read into the memory 401 through the external interface 407 from the digital camera 408 that captured the image data. The display 402 and the pointing device 403 are used for the graphic display and editing of the point spread function described with reference to FIG.
[0054]
According to another embodiment of the present invention, the processing functions described in relation to FIGS. 1, 5, and 6 are implemented in an imaging device such as a digital camera.FIG.FIG. 2 is a schematic block diagram of an example of such a digital camera. In FIG. 8, 500 is a CCD image sensor on which an optical image is formed through an optical system (not shown), and 502 is a signal for amplifying an analog image signal output from the CCD image sensor 500, converting it to a digital signal, or the like. A circuit unit 502 is a data memory for storing image data and the like, 503 is a digital signal processor for compressing / decompressing digital image data and converting it into a video signal, and 504 is a monitor display using a liquid crystal display panel or the like. is there. Reference numeral 506 denotes a microcomputer for controlling the inside of the camera, and includes a CPU 510, a ROM 511, a RAM 512, and other external interfaces (not shown). An operation unit 507 is used by the user to input various instructions to the microcomputer 506.
[0055]
In a state where the shooting button of the operation unit 507 is not pressed, the digital signal processor 503 converts the digital image data continuously output from the signal circuit unit 501 into a video signal and outputs the video signal to the monitor display 504. The video inside can be monitored on the monitor display 504. When the user presses the shooting button on the operation unit 507, digital image data for one image output from the signal circuit unit 501 is temporarily stored in the data memory 502 under the control of the microcomputer 506. Next, the image data is transferred to the digital signal processor 503 and compressed after necessary gradation correction and the like. The obtained compressed image data is transferred to the data memory 502. When the operation unit 507 instructs to output image data to an external personal computer or the like, the compressed image data in the data memory 502 is transferred to the digital signal processor 503 and decompressed under the control of the microcomputer 506. The decompressed image data is written into the data memory 502 again. The image data is transferred to an external personal computer or the like under the control of the microprocessor 506. Such general control inside the digital camera is performed according to a program stored in the ROM 511.
[0056]
In the digital camera shown here, the means for processing steps described in relation to FIGS. 1, 5 and 6 are implemented by software. That is, a program 520 for causing the microcomputer 506 to execute each processing step described with reference to FIG. 1, FIG. 5, and FIG. 6, a point spread function and a blur as described with reference to FIG. The correction function table 521 is also stored in the ROM 511 (or RAM 512). When the user instructs camera shake correction by operating the operation unit 507, the microcomputer 506 transfers the compressed image data on the data memory 502 to the digital signal processor 503 according to the program 520 and decompresses the decompressed image data. The data is stored in the data memory 502. Since the video signal of the decompressed image data is also output from the digital signal processor 503, the user can confirm the image of the image data on the monitor display 504. Then, the microprocessor 506 executes the camera shake image correction processing according to the present invention as described above on the decompressed image data on the data memory 502. The corrected image data obtained on the data memory 502 is transferred to the digital signal processor 503, the video signal is given to the monitor display 504, and the user can check the image after the camera shake correction. The user can select the blur correction function or edit the point spread function as described above by giving an instruction using the operation unit 507. In this case, the microcomputer 506 displays an image for selecting the blur correction function and a point spread function on the monitor display 504 via the digital signal processor 503. The user can instruct selection of a blur correction function or editing of a point spread function on the screen of the monitor display 504 by operating a pointing device (not shown) provided in the operation unit 507.
[0057]
【The invention's effect】
As is apparent from the above description, the claims1 to 6, 8 to 13According to the inventions described in the above items, even when a camera shake including nonuniform rotation and translation occurs, an appropriate camera shake correction is possible. In addition, preprocessing and postprocessing such as image rotation are not required for camera shake correction, and image quality degradation associated with such processing does not occur. Therefore, it is possible to restore a highquality image in which camera shake is corrected from a nonuniform camera shake image. Claim6 or 13According to the described invention, calculation for camera shake correction can be performed at high speed by a table reference method and the like, and preprocessing and postprocessing such as image rotation are not necessary, so that highspeed processing as a whole is possible. It is. Claim5According to the described invention, since the direction and size of camera shake at a plurality of positions of the image are estimated from the image itself, an appropriate camera shake correction can be performed even for an image taken with a camera having no acceleration sensor or the like. Is possible. Claim12According to the described invention, camera shake correction can be appropriately performed on a captured image without providing an acceleration sensor or the like in the imaging device. Claim1 or 8According to the described invention, the user can observe the image, select the blur correction function determined to be appropriate, perform camera shake correction, and reselect the blur correction function when determined to be necessary. Therefore, it is possible to cope with various hand movement patterns. Claims2 or 9According to this, it is possible to perform appropriate camera shake correction using a blur correction function corresponding to a camera shake pattern at the time of image capturing without involving the user. Claims3, 4, 10 or 11According to the described invention, the user can arbitrarily edit the point spread function according to the difference in the correction according to the shooting environment condition and the difference in the shake pattern as described in JPA7226905. The blur correction function can be optimized. Claim7According to the described invention, it is possible to perform the abovedescribed camera shake image correction, blur correction function optimization, and the like using a computer.
[Brief description of the drawings]
FIG. 1 is a flowchart illustrating an example of a camera shake image correction processing procedure according to the present invention.
FIG. 2 is a diagram showing camera shake vectors at a plurality of positions on an image related to determination of a coordinate transformation matrix.
FIG. 3 is a diagram for explaining a onedimensional interpolation operation along a camera shake vector;
FIG. 4 is a diagram illustrating an example of a point spread function corresponding to a typical camera shake pattern and an example of editing the point spread function.
FIG. 5 is a flowchart for explaining an example of a procedure for automatically selecting a blur correction function.
FIG. 6 is a flowchart for explaining a procedure for correcting a blur correction function by editing a point spread function;
FIG. 7 is a block diagram illustrating an example of a computer for implementation by software of the present invention.
FIG. 8 is a block diagram showing an example of a digital camera according to the present invention.
[Explanation of symbols]
400 CPU
401 memory
402 Display device
403 pointing device
410 Program for image stabilization
411 Point spread function and blur correction function table
500 CCD image sensor
501 Signal circuit
502 data memory
503 Digital signal processor
504 Monitor display
506 Microcomputer
507 Operation unit including pointing device
520 Program for image stabilization
521 Table of point spread function and blur correction function
Claims (13)
 A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated in the second step at the position;
Among the plurality of blur correction functions corresponding to the point spread function of a plurality of typical camera shake patterns including a camera shake pattern with a nonuniform camera shake speed prepared in advance, the blur correction function is used in the third step. A camera shake image correction method comprising: a fourth step in which a user selects a blur correction function for the purpose.  A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated in the second step at the position;
A fourth step of estimating a point spread function due to camera shake of the target image;
A fifth step of selecting a point spread function most similar to the point spread function estimated in the fourth step from a plurality of typical point spread functions prepared in advance;
Includes a sixth step of selecting from among said fifth blur correction function corresponding to the typical point spread function selected in step, a plurality of blur correction function prepared in advance,
A blurring image correction method, wherein the blur function selected in the sixth step is used for blur correction in the third step .  A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated by the second step at the position;
A fourth step of displaying a graph of a point spread function prepared in advance on a display screen;
A fifth step of editing the point spread function graph displayed on the screen according to an operation of a pointing device;
A camera shake image correction comprising: a sixth step of correcting a blur correction function used for blur correction in the third step in correspondence with the point spread function graph after editing in the fifth step Method.  A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated in the second step at the position;
A fourth step of displaying a graph of a point spread function due to camera shake of the target image on a display screen;
A fifth step of editing the point spread function graph displayed on the screen according to an operation of a pointing device;
A sixth step of generating a blur correction function corresponding to the point spread function after editing in the fifth step ;
The blur image correction method , wherein the blur correction function generated in the sixth step is used for blur correction in the third step .  The first step includes the step of estimating the direction and size of camera shake at a plurality of positions on the target image based on the target image, and based on the direction and size of the camera shake estimated by the step, 5. The camera shake image correction method according to claim 1, wherein a coordinate transformation matrix representing rotation and translation of a pixel on a target image due to camera shake is determined.
 6. The camera shake image correction method according to claim 1 , wherein in the third step, blur correction is performed by an interpolation calculation along a camera shake vector.
 A computerreadable recording medium on which a program for causing a computer to execute each step of the camera shake image correction method according to any one of claims 1 to 6 is recorded.
 An imaging apparatus that obtains a digital image by imaging a subject with an imaging device,
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital target image according to the hand movement vector estimated by the second means at the position;
Among the plurality of blur correction functions prepared in advance and corresponding to the point spread function of a plurality of typical camera shake patterns including a camera shake pattern having a nonuniform camera shake speed, the third means uses the blur correction function in the third means. An image pickup apparatus comprising: a fourth means for a user to select a blur correction function for use.  An imaging apparatus that obtains a digital image by imaging a subject with an imaging device,
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital image according to the hand movement vector estimated by the second means at the position;
A fourth means for estimating a point spread function due to camera shake in a digital image;
A fifth means for selecting a point spread function most similar to the point spread function estimated by the fourth means from a plurality of typical point spread functions prepared in advance;
A sixth means for selecting a blur correction function corresponding to the typical point spread function selected by the fifth means from a plurality of blur correction functions prepared in advance;
An image pickup apparatus, wherein the blur function selected by the sixth means is used for blur correction by the third means.  An imaging apparatus that obtains a digital image by imaging a subject with an imaging device,
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital image according to the hand movement vector estimated by the second means at the position;
A fourth means for displaying a graph of a point spread function prepared in advance on a display screen;
A fifth means for editing the graph of the point spread function displayed on the screen according to an operation of a pointing device;
An image pickup apparatus comprising: sixth means for correcting a blur correction function used for blur correction in the third means in correspondence with the point spread function graph after editing by the fifth means.  An imaging apparatus that obtains a digital image by imaging a subject with an imaging device,
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital image according to the hand movement vector estimated by the second means at the position;
A fourth means for displaying a graph of a point spread function due to camera shake of a digital image on a display screen;
A fifth means for editing the graph of the point spread function displayed on the screen according to an operation of a pointing device;
A sixth means for generating a blur correction function corresponding to the point spread function after editing by the fifth means;
An image pickup apparatus, wherein the blur correction function generated by the sixth means is used for blur correction in the third means.  The first means includes means for estimating the direction and size of camera shake at a plurality of positions on the digital image based on the digital image, and based on the direction and size of camera shake estimated by the means, The imaging apparatus according to claim 8, wherein a coordinate transformation matrix representing rotation and translation of a pixel on a digital image due to camera shake is determined.
 13. The imaging apparatus according to claim 8, wherein in the third means, blur correction is performed by an interpolation calculation along a camera shake vector.
Priority Applications (1)
Application Number  Priority Date  Filing Date  Title 

JP10503399A JP4345940B2 (en)  19990413  19990413  Camera shake image correction method, recording medium, and imaging apparatus 
Applications Claiming Priority (1)
Application Number  Priority Date  Filing Date  Title 

JP10503399A JP4345940B2 (en)  19990413  19990413  Camera shake image correction method, recording medium, and imaging apparatus 
Publications (2)
Publication Number  Publication Date 

JP2000298300A JP2000298300A (en)  20001024 
JP4345940B2 true JP4345940B2 (en)  20091014 
Family
ID=14396714
Family Applications (1)
Application Number  Title  Priority Date  Filing Date 

JP10503399A Expired  Fee Related JP4345940B2 (en)  19990413  19990413  Camera shake image correction method, recording medium, and imaging apparatus 
Country Status (1)
Country  Link 

JP (1)  JP4345940B2 (en) 
Families Citing this family (23)
Publication number  Priority date  Publication date  Assignee  Title 

JP2005003719A (en)  20030609  20050106  Olympus Corp  Photographing device 
JP4685966B2 (en) *  20030926  20110518  富士フイルム株式会社  Image processing method, apparatus, and program 
JP2005122688A (en) *  20030926  20050512  Fuji Photo Film Co Ltd  Image processing method, device, and program 
JP4306457B2 (en)  20040108  20090805  株式会社デンソーウェーブ  Optical information reader 
JP4566591B2 (en) *  20040319  20101020  キヤノン株式会社  Image deformation estimation method and image deformation estimation apparatus 
JP4527439B2 (en) *  20040325  20100818  富士フイルム株式会社  Image processing method, apparatus, and program 
JP4345651B2 (en)  20041129  20091014  セイコーエプソン株式会社  Image information evaluation method, image information evaluation program, and image information evaluation apparatus 
JP4342493B2 (en)  20050922  20091014  三洋電機株式会社  Image stabilizer 
US8243148B2 (en)  20051228  20120814  Nittoh Kogaku K.K.  Image processing device 
WO2007141863A1 (en) *  20060608  20071213  Nippon Computer System Co., Ltd  Image processing program and computer readable recording medium with the program recorded therein and image processing device 
JP4825748B2 (en) *  20070713  20111130  株式会社モルフォ  Image data processing method and imaging apparatus 
JP2009033479A (en) *  20070727  20090212  Sharp Corp  Camera shake correction device 
JP4661922B2 (en)  20080903  20110330  ソニー株式会社  Image processing apparatus, imaging apparatus, solidstate imaging device, image processing method, and program 
JP4591603B2 (en) *  20090126  20101201  セイコーエプソン株式会社  Image information evaluation method, image information evaluation program, and image information evaluation apparatus 
JP5396964B2 (en) *  20090330  20140122  株式会社ニコン  Imaging apparatus and image processing apparatus 
JP5379572B2 (en) *  20090619  20131225  キヤノン株式会社  Image processing apparatus, image processing method, program, storage medium, and imaging apparatus 
JP2011008322A (en) *  20090623  20110113  Univ Of Tokyo  Imaging apparatus and image processing method 
JP5627256B2 (en) *  20100316  20141119  キヤノン株式会社  Image processing apparatus, imaging apparatus, and image processing program 
KR101839617B1 (en) *  20110711  20180319  포항공과대학교 산학협력단  Method and apparatus for removing nonuniform motion blur using multiframe 
US9338354B2 (en)  20111003  20160510  Nikon Corporation  Motion blur estimation and restoration using light trails 
KR101804215B1 (en) *  20120306  20171205  삼성전자주식회사  A method and an apparatus for estimating a nonuniform motion blur robustly 
KR101810876B1 (en) *  20120313  20180126  삼성전자주식회사  A method and an apparatus for debluring nonuniform motion blur of a large scale input image based on a tile unit 
KR101844332B1 (en) *  20120313  20180403  삼성전자주식회사  A method and an apparatus for debluring nonuniform motion blur usign multiframe comprises of a blur image and a noise image 

1999
 19990413 JP JP10503399A patent/JP4345940B2/en not_active Expired  Fee Related
Also Published As
Publication number  Publication date 

JP2000298300A (en)  20001024 
Similar Documents
Publication  Publication Date  Title 

US8885061B2 (en)  Image processing apparatus, image processing method and program  
KR101624450B1 (en)  Image processing device, image processing method, and storage medium  
US10116879B2 (en)  Method and apparatus for obtaining an image with motion blur  
US10080006B2 (en)  Stereoscopic (3D) panorama creation on handheld device  
JP5574423B2 (en)  Imaging apparatus, display control method, and program  
US8593533B2 (en)  Image processing apparatus, imagepickup apparatus, and image processing method  
JP4429241B2 (en)  Image processing apparatus and method  
JP3770271B2 (en)  Image processing device  
JP4042563B2 (en)  Image noise reduction  
US9407814B2 (en)  Approach for camera control  
US5596346A (en)  Method and apparatus for applying a function to a localized area of a digital image using a window  
US8208017B2 (en)  Imaging device, product package, and semiconductor integrated circuit  
US8494300B2 (en)  Method of notifying users regarding motion artifacts based on image analysis  
US7379621B2 (en)  Method and system for composing universally focused image from multiple images  
US8494299B2 (en)  Method of determining PSF using multiple instances of a nominally similar scene  
KR101442610B1 (en)  Digital photographing apparatus, method for controlling the same, and recording medium storing program to implement the method  
JP4720859B2 (en)  Image processing apparatus, image processing method, and program  
JP4539729B2 (en)  Image processing apparatus, camera apparatus, image processing method, and program  
JP4760973B2 (en)  Imaging apparatus and image processing method  
Karpenko et al.  Digital video stabilization and rolling shutter correction using gyroscopes  
US8289405B2 (en)  Image processing apparatus and method thereof  
JP5596138B2 (en)  Imaging apparatus, image processing apparatus, image processing method, and image processing program  
US8244053B2 (en)  Method and apparatus for initiating subsequent exposures based on determination of motion blurring artifacts  
US8860848B2 (en)  Image processing apparatus and method  
US7535497B2 (en)  Generation of static image data from multiple image data 
Legal Events
Date  Code  Title  Description 

A621  Written request for application examination 
Free format text: JAPANESE INTERMEDIATE CODE: A621 Effective date: 20060309 

A131  Notification of reasons for refusal 
Free format text: JAPANESE INTERMEDIATE CODE: A131 Effective date: 20090408 

A521  Written amendment 
Free format text: JAPANESE INTERMEDIATE CODE: A523 Effective date: 20090608 

TRDD  Decision of grant or rejection written  
A01  Written decision to grant a patent or to grant a registration (utility model) 
Free format text: JAPANESE INTERMEDIATE CODE: A01 Effective date: 20090708 

A01  Written decision to grant a patent or to grant a registration (utility model) 
Free format text: JAPANESE INTERMEDIATE CODE: A01 

A61  First payment of annual fees (during grant procedure) 
Free format text: JAPANESE INTERMEDIATE CODE: A61 Effective date: 20090709 

FPAY  Renewal fee payment (event date is renewal date of database) 
Free format text: PAYMENT UNTIL: 20120724 Year of fee payment: 3 

FPAY  Renewal fee payment (event date is renewal date of database) 
Free format text: PAYMENT UNTIL: 20120724 Year of fee payment: 3 

FPAY  Renewal fee payment (event date is renewal date of database) 
Free format text: PAYMENT UNTIL: 20130724 Year of fee payment: 4 

LAPS  Cancellation because of no payment of annual fees 