JP4345940B2 - Camera shake image correction method, recording medium, and imaging apparatus - Google Patents

Camera shake image correction method, recording medium, and imaging apparatus Download PDF

Info

Publication number
JP4345940B2
JP4345940B2 JP10503399A JP10503399A JP4345940B2 JP 4345940 B2 JP4345940 B2 JP 4345940B2 JP 10503399 A JP10503399 A JP 10503399A JP 10503399 A JP10503399 A JP 10503399A JP 4345940 B2 JP4345940 B2 JP 4345940B2
Authority
JP
Japan
Prior art keywords
camera shake
image
blur correction
point spread
function
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
JP10503399A
Other languages
Japanese (ja)
Other versions
JP2000298300A (en
Inventor
拓之 坂本
公一 江尻
海克 関
伸 青木
Original Assignee
株式会社リコー
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株式会社リコー filed Critical 株式会社リコー
Priority to JP10503399A priority Critical patent/JP4345940B2/en
Publication of JP2000298300A publication Critical patent/JP2000298300A/en
Application granted granted Critical
Publication of JP4345940B2 publication Critical patent/JP4345940B2/en
Anticipated expiration legal-status Critical
Expired - Fee Related legal-status Critical Current

Links

Images

Description

[0001]
BACKGROUND OF THE INVENTION
  The present invention relates to a technique for correcting image degradation due to camera shake.
[0002]
[Prior art]
  When a subject is photographed with a digital camera or the like, camera movement during exposure, that is, blur due to camera shake occurs in the photographed image. Various techniques have been developed in the past to avoid image degradation due to camera shake.
[0003]
  For example, the camera is equipped with an acceleration sensor or the like for detecting the movement of camera shake, the detection signal of the sensor is converted into an image movement, and the camera optical system (prism, A method of mechanically moving a lens or an image pickup device is known (for example, Japanese Patent No. 2637464).
[0004]
  The movement of a virtual point-like subject on the image due to camera shake corresponds to blurring, and is often called a point spread function. Also known is a method for correcting camera shake by estimating the point spread function from the change over time in the aperture ratio of the shutter at the time of shooting and the movement of the camera optical axis detected by an acceleration sensor, etc., and applying the inverse correction to the entire image. (JP-A-7-226905).
[0005]
  Furthermore, research for estimating camera shake from an image without using an acceleration sensor or the like has been made for a long time. For example, in the document “A. About Alib and L. M. Silverman; Rotation of Motion Degraded Images; IEEE Trans. Circuits and Systems, p. Are listed. Also, in the direction of camera shake, if the camera shake can be approximated to be uniform, the density should be uniform (brightness). Therefore, taking the autocorrelation function of the image in the camera shake direction and differentiating this will yield the amount of camera shake. The differential value shows a minimum at a corresponding interval and shows a maximum at the zero point in the direction without blurring, so that it is possible to estimate the magnitude of the blur, “Y. Yitzhaky and NS Kopeika; Identification of Blur Parameters from Motion Blurred Images; Graphical Models and Image Processing Vol.59, pp.310-320, 1997 ". Regarding estimation of the blur direction, for example, the document “R. Fabian and D. Malah; Robust Identification and Out-Of-Focus Blur Parameters from Blurred and Noisy Images; Graphical Models and Image Processing, Vol. 53, pp. 403-412, 1991 ”describes a method using a spatial frequency distribution.
[0006]
[Problems to be solved by the invention]
  When the camera rotates around the optical axis at the time of shooting, camera shake on the image becomes non-uniform. It is an object of the present invention to provide a camera shake that can restore a sharp image without mechanical operation as in the above-mentioned Japanese Patent No. 2637464 even when such a camera shake including uneven rotation and translation occurs. An image correction method and an imaging apparatus are provided. Another object of the present invention is a camera shake image correction method capable of accurately correcting non-uniform camera shake even with respect to an image taken with an ordinary camera that does not have an acceleration sensor or the like for detecting shake of an optical axis. Is to provide. Another object of the present invention is to provide an imaging apparatus capable of obtaining an image in which non-uniform camera shake is accurately corrected without providing an acceleration sensor or the like for detecting optical axis shake.
[0007]
  The speed of camera shake is not always uniform from the beginning to the end of camera shake. As the speed of camera shake changes with time, the shape of the point spread function also changes. Another object of the present invention is to provide a camera shake image correction method and an imaging apparatus capable of appropriate camera shake correction even when the camera shake speed fluctuates with time.
[0008]
  In some cases, such as Japanese Patent Application Laid-Open No. 7-226905, which presupposes that image data that directly reflects a change in the amount of light due to a change in the aperture ratio of the shutter as it is, good camera shake correction may not be expected. This problem is particularly noticeable in digital cameras that have become widespread in recent years. In a general digital camera, an optical image formed by a lens system is converted into an electric signal by an image pickup device such as a CCD. This electric signal is converted to an external brightness or color deviation (red in a clear sky evening) Because it is corrected according to the shooting environment conditions such as strong blue and strong blue under fluorescent lamps), it is not possible to obtain image data that directly reflects the change in light quantity due to the change in shutter aperture over time. is there. Another object of the present invention is to provide a camera shake image correction method and an imaging apparatus capable of performing appropriate camera shake correction even when correction according to the shooting environment conditions described above is performed. .
[0009]
[Means for Solving the Problems]
  According to the first to seventh aspects of the present invention, an improved camera shake image correction method is provided. Its characteristics are as follows.
[0010]
  A camera shake image correction method according to the first aspect of the present invention includes:
A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated in the second step at the position;
Among the plurality of blur correction functions corresponding to the point spread function of a plurality of typical camera shake patterns including a camera shake pattern with a non-uniform camera shake speed prepared in advance, the blur correction function is used in the third step. Including a fourth step in which a user selects a blur correction function.
[0011]
  A camera shake image correction method according to a second aspect of the present invention includes:
A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated in the second step at the position;
A fourth step of estimating a point spread function due to camera shake of the target image;
A fifth step of selecting a point spread function most similar to the point spread function estimated in the fourth step from a plurality of typical point spread functions prepared in advance;
A sixth step of selecting a blur correction function corresponding to the typical point spread function selected in the fifth step from a plurality of blur correction functions prepared in advance.
Including
The blur function selected in the sixth step is used for blur correction in the third step.
[0012]
  According to a third aspect of the present invention, there is provided a camera shake image correction method.
A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated by the second step at the position;
A fourth step of displaying a graph of a point spread function prepared in advance on a display screen;
A fifth step of editing the point spread function graph displayed on the screen according to an operation of a pointing device;
Corresponding to the point spread function graph after editing in the fifth step, a sixth step of correcting a blur correction function used for blur correction in the third step is included.
[0013]
  According to a fourth aspect of the present invention, there is provided a camera shake image correction method.
A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated in the second step at the position;
A fourth step of displaying a graph of a point spread function due to camera shake of the target image on a display screen;
A fifth step of editing the point spread function graph displayed on the screen according to an operation of a pointing device;
A sixth step of generating a blur correction function corresponding to the point spread function after editing in the fifth step;
The blur correction function generated in the sixth step is used for blur correction in the third step.
[0014]
  According to a fifth aspect of the present invention, in the camera shake image correction method according to any one of the first to fourth aspects, the first step includes a direction of camera shake at a plurality of positions on the target image, and Including a step of estimating the size based on the target image, and based on the direction and size of the camera shake estimated by the step, a coordinate transformation matrix representing rotation and translation of the pixel on the target image due to the camera shake Is to decide.
[0015]
  According to a sixth aspect of the present invention, in the camera shake image correction method according to any one of the first to fifth aspects, blur correction is performed in the third step by an interpolation operation along a camera shake vector. It is to be.
[0016]
  According to the eighth to thirteenth aspects of the present invention, there is provided an improved image pickup apparatus that picks up a subject with an image pickup device and obtains a digital image. Its characteristics are as follows.
[0017]
  An image pickup apparatus according to the invention described in claim 8 provides:
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital target image according to the hand movement vector estimated by the second means at the position;
Among the plurality of blur correction functions prepared in advance and corresponding to the point spread function of a plurality of typical camera shake patterns including a camera shake pattern having a non-uniform camera shake speed, the third means uses the blur correction function in the third means. Fourth means for the user to select a blur correction function for
It is characterized by including.
[0018]
  An image pickup apparatus according to a ninth aspect of the present invention provides:
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital image according to the hand movement vector estimated by the second means at the position;
A fourth means for estimating a point spread function due to camera shake in a digital image;
A fifth means for selecting a point spread function most similar to the point spread function estimated by the fourth means from a plurality of typical point spread functions prepared in advance;
Sixth means for selecting a blur correction function corresponding to the typical point spread function selected by the fifth means from a plurality of blur correction functions prepared in advance.
Including
The blur function selected by the sixth means is used for blur correction in the third means.
[0019]
  An imaging apparatus according to the invention of claim 10 is provided.
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital image according to the hand movement vector estimated by the second means at the position;
A fourth means for displaying a graph of a point spread function prepared in advance on a display screen;
A fifth means for editing the graph of the point spread function displayed on the screen according to an operation of a pointing device;
Corresponding to the point spread function graph after editing by the fifth means, sixth means for correcting the blur correction function used for blur correction in the third means
It is characterized by including.
[0020]
  An imaging device according to the invention of claim 11 is provided.
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital image according to the hand movement vector estimated by the second means at the position;
A fourth means for displaying a graph of a point spread function due to camera shake of a digital image on a display screen;
A fifth means for editing the graph of the point spread function displayed on the screen according to an operation of a pointing device;
Sixth means for generating a blur correction function corresponding to the point spread function after editing by the fifth means
Including
The blur correction function generated by the sixth means is used for blur correction in the third means.
[0021]
  According to a twelfth aspect of the present invention, in the imaging apparatus according to any one of the eighth to eleventh aspects, the first means includes a direction and a size of camera shake at a plurality of positions on the digital image. Is determined based on the digital image, and a coordinate transformation matrix representing rotation and translation of the pixel on the digital image due to the camera shake is determined based on the direction and size of the camera shake estimated by the means. That is.
[0022]
  According to a thirteenth aspect of the present invention, in the imaging apparatus according to any one of the eighth to twelfth aspects, blur correction is performed by an interpolation operation along a camera shake vector in the third means. .
[0023]
DETAILED DESCRIPTION OF THE INVENTION
  FIG. 1 shows an example of a processing procedure for camera shake image correction according to the present invention. In FIG. 1, 100 is a step in which an image in which blur is caused by camera shake is input, 101 is a step in which a coordinate transformation matrix representing rotation and translation of pixels on the input image due to camera shake is determined, and 102 is the coordinates A step of estimating a camera shake vector (direction and magnitude of “blurring” due to camera shake) at each pixel position on the input image using the transformation matrix, and 103 for each pixel value on the input image A step in which blur correction according to the estimated camera shake vector is performed and an image in which blur due to camera shake is corrected is generated, and 104 is a step in which a corrected image is output. Note that the camera shake image correction processing procedure according to the present invention can be variously changed as will be described later.
[0024]
  << Determination of coordinate transformation matrix: Step101
  Step101The specific processing content of will be described.
  Pixels on the image move from their original positions due to camera shake when taking an image. Since such minute blur can be expressed by affine transformation, the following equation is obtained between the original pixel coordinate value (x, y) and the coordinate value (X, Y) changed by the camera shake. It holds.
[0025]
[Expression 1]
Where θ is the rotation angle and σx, ΣyRepresents the amount of parallel movement in the x and y directions. Since the rotation angle θ due to camera shake is generally sufficiently small, the equation (1) may be approximated as in the equation (2) or may be further approximated as in the equation (3).
[0026]
[Expression 2]
[Equation 3]
[0027]
  Step101In the three parameters (θ, σ) of the coordinate transformation matrix of the formula (1), (2) or (3)x, Σy). When taking an input image with a camera and using an acceleration sensor, a gyro sensor or the like to detect the shake amount of the optical axis of the camera and the rotation angle around the optical axis, the detection result is, for example, a step. 100 is input as additional information of the image, and based on the additional information, three parameters (θ, σx, Σy) Can be determined. According to one embodiment of the present invention, such a method is employed.
[0028]
  However, when a general camera not equipped with such a sensor is used, the above-described method cannot be used. Therefore, according to another embodiment of the present invention, a set of (x, y) and (X, Y) at two or more places on the input image is estimated, in other words, in FIG. By estimating the direction and magnitude (shake vector) of the camera shake as shown by B3 and substituting it into the formula (1), (2) or (3), the parameters (θ, σ) of the coordinate transformation matrix are calculated.x, Σy). An example of a specific processing procedure will be described below.
[0029]
  First, select at least two appropriate points on the input image. This may be automatically performed by a processing program, or may be designated by a user using a pointing device on an appropriate display screen on which an image is displayed.
[0030]
  Next, the direction of camera shake at each selected point is estimated. For example, the spatial frequency distribution is measured in the vicinity of each selected point, and the direction in which the reduction rate of the high frequency distribution is maximum is estimated as the direction of camera shake. Such a method is described in detail in Japanese Patent Application No. 9-180846 (Japanese Patent Laid-Open No. 11-027574) or the above-mentioned R. Fabian et al.
[0031]
  Next, the magnitude of camera shake at each selected point is estimated. For example, taking the autocorrelation function of the image in the direction of the estimated camera shake for the region near each selected point and differentiating it in the direction of the camera shake, the differential value is minimal at intervals corresponding to the size of the camera shake. Therefore, the size of the shake is estimated from the interval between the local minimum points. This method is detailed in the above-mentioned literature by Y. Yitzhaky et al.
[0032]
  When such estimation is finished, the x and y coordinate values (x, y) of each selected point and the x and y coordinate values of the point moved by the estimated camera shake direction in the estimated camera shake direction. By substituting the pair with (X, Y) into the above formula (1), (2) or (3), the parameters (θ, σx, Σy) Thus, the coordinate transformation matrix of the formula (1), the formula (2) or the formula (3) is determined.
[0033]
  The input image is enlarged and displayed on an appropriate display screen, and the user determines whether the image is shaken due to camera shake on the screen, and uses a pointing device to find the appropriate two or more points and the points moved by the shake. Directly instructing and using the indicated coordinate values, parameters (θ, σx, Σy) Is also possible. Even with this simple method, a skilled user can determine parameters with reasonable accuracy.
[0034]
  << Estimation of Shake Vector at Each Pixel Position: Step 102 >>
  Next, in step 102, the hand movement vector B (x, y) (motion vector due to hand movement) at each pixel position (x, y) on the input image is estimated using the coordinate transformation matrix determined in step 101. That is, (X, Y) is calculated by substituting (x, y) into the equation (1), (2) or (3), and B (x, y) is obtained by the following equation.
[0035]
[Expression 4]
[0036]
  << Bokeh correction: Step 103 >>
  In step 103, each pixel value of the input image is subjected to blur correction according to a camera shake vector whose position is estimated, and a corrected image in which blur due to camera shake is compensated is generated.
[0037]
  When camera shake is introduced by uniform movement, blur due to camera shake is expressed by a one-dimensional point spread function as shown in Equation (5), and blur due to such camera shake is as shown in Equation (6). It is known that correction can be performed with a correction function. In these two formulas, t is the distance of camera shake. a is the length of the blur, which corresponds to the length of the hand movement vector. However, if the camera shake speed is not uniform, the point spread function and the blur correction function will be slightly different. For more accurate camera shake correction, select or correct the blur correction function according to the camera shake pattern. It is desirable to be able to do this, but this will be described later.
[0038]
[Equation 5]
[Formula 6]
[0039]
  According to one embodiment of the present invention, each pixel (ic, Jc) For the corrected pixel value V (i) by performing a one-dimensional interpolation operation along the hand movement vector B at that position.c, Jc). In particular
[Expression 7]
In the equation, θ is an angle formed with respect to the x axis of the shake vector B. In addition, the following relationship holds.
[Equation 8]
S is a pixel in a blurred range, and ax, AyIs the length of the x and y components of the hand movement vector B, the equation (7) can be written as follows.
[Equation 9]
[0040]
  P () in the equations (7) and (9) is an interpolated pixel value on the camera shake vector B or an extension line thereof calculated from the previous and next pixel values along the camera shake vector B, and is expressed as (9) By substituting into the equation, the corrected pixel value V (ic, Jc) Is obtained. Interpolation function
  I (x) = h-1(X)
Is assumed to hold.
[0041]
  If only the pixel adjacent to the target pixel is used, the interpolation function I (x) is calculated from the pixels on both sides along the camera shake vector B as indicated by the symbol “*” in FIG. Here, the interpolated pixel value p () is the value of the virtual pixel indicated by the symbol “=” in FIG. This pixel value can also be obtained by the interpolation function I (x) (in this case, a = 1). The values of x and y may generally be arbitrary. However, in order to perform efficient calculation, the pixel pitch is set to 1,
[Expression 10]
(FIG. 3 shows such a case). Therefore, equation (9) can be rewritten as follows.
## EQU11 ##
  If only pixels on both sides of the hand movement vector B are used, the conditional expression i, j A
Can be expressed as:
[Expression 12]
In the formula, [x] represents an integer not exceeding x.
[0042]
  The above-described camera shake image correction method according to the present invention can perform highly accurate camera shake correction by real space processing even when there is uneven camera shake (rotation and translation) in an image. Since pre-processing and post-processing such as image rotation are not included, image quality degradation associated with such processing does not occur. In addition, since such pre-processing and post-processing are not included, the interpolation pixel value along the camera shake vector can be calculated at high speed by the table reference method, so that high-speed processing as a whole is possible.
[0043]
  Now, the point spread function of equation (5) is represented as a graph as shown in FIG.ThisYou can. In FIG. 4, the horizontal axis represents the blur distance t, and the vertical axis represents the response magnitude. Actual camera shake often has a non-uniform velocity, and the point spread function is slightly different from the graph shown in FIG. However, camera shake patterns can be classified into several types. Graphs of point spread functions corresponding to some typical camera shake patterns are shown in FIGS. 4 (C), (D) and (E). The graph in (C) represents a point spread function corresponding to a blurring pattern that is slow and gradually fast at first, and the graph in (D) is a point spread function corresponding to a blurring pattern that is fast at first and gradually slows down. To express. The graph of (E) represents a point spread function corresponding to a camera shake pattern in which the beginning and the end are slow and the motion is fast in the middle. Once you get used to it, you can figure out what kind of hand-shake pattern it is, with considerable accuracy, by observing the image.
[0044]
  According to one embodiment of the present invention, a blur correction function corresponding to a point spread function of a plurality of typical camera shake patterns such as (A), (C), (D), and (E) in FIG. A blur correction function that is prepared in the form of a table and is used for blur correction is selected by the user. In the context of the processing flow shown in FIG. 1, the step for selecting such a blur correction function is placed inside or before the blur correction processing step 103.
[0045]
  According to another embodiment of the present invention, such a blur correction function is automatically selected. An example of the procedure for this is shown in FIG.
[0046]
  In step 200 of FIG. 5, a point spread function due to camera shake of the input image is estimated. When a camera equipped with an acceleration sensor to detect optical axis shake is used to capture an input image, the sensor detects a change in the optical axis over time and superimposes it with the shutter release timing of the camera. Thus, the point spread function of the input image can be estimated directly. When such a camera is not used, a point spread function is estimated from the input image. Specifically, it is only necessary to observe a density (luminance) distribution along the tail image formed by the blurring of one point on the subject. For more accurate estimation, for example, the method described in the above-mentioned Y. Yitzhaky et al. Paper is used for a high-contrast portion (usually called an edge) adjacent to a smooth background portion in an image. do it.
[0047]
  In the next step 201, the one most similar to the point spread function estimated in the previous step is selected from the point spread functions corresponding to a plurality of typical hand movement patterns prepared in advance. Distance can be used as a criterion for this selection. For example, the following distance DjHas the smallest point spread function hj(T) is selected.
[Formula 13]
Where h (t) is the estimated point spread function, hj(T) is a point spread function prepared in advance.
[0048]
  A plurality of blur correction functions corresponding to respective point spread functions of typical hand movement patterns prepared in advance are also prepared in advance. In step 202, a blur correction function corresponding to the point spread function selected in the previous step is selected from the prepared blur correction functions. The blur correction function selected in this way is used in the blur correction processing step 103 in FIG.
[0049]
  According to an embodiment of the present invention, a user can correct a blur correction function for use in blur correction processing by checking a point spread function on a display and performing necessary editing. The procedure is shown in FIG.
[0050]
  In step 300 of FIG. 6, a point spread function prepared in advance is displayed on an appropriate display screen. In step 301, the user makes necessary edits to the point spread function on the screen using an appropriate pointing device. For example, when a point spread function graph as shown in FIG. 4A is displayed, the graph can be edited as shown in FIG. 4B using a pointing device, for example. . Note that the point spread function displayed in step 300 may be selected by the user from a plurality of point spread functions.
[0051]
  In step 302, the blur correction function prepared in advance is corrected so as to reflect the edited content of the point spread function in the previous step. Thus, the corrected blur correction function is used in the blur edit processing step 103 in FIG.
[0052]
  Another embodiment of the present invention will be described with reference to FIG. In step 300, a point spread function due to camera shake of the input image is displayed graphically. Specifically, for example, by projecting the luminance in a direction orthogonal to the blur direction for a uniform background portion in the input image, a graph as shown in FIG. 4 can be directly obtained, so that it is displayed on an appropriate display. To do. Since the point spread function measured in this way often involves measurement errors, in step 301, the user makes necessary corrections to the point spread function on the screen of the display using a pointing device. In step 302, a blur correction function corresponding to the corrected point spread function is generated. The generated blur correction function is used for the blur correction process.
[0053]
  The present invention includes a CPU 400 as shown in FIG. 7, a memory 401, a display 402, a pointing device 403 such as a mouse, a drive 405 for a recording medium 404 such as a CD-ROM, an auxiliary storage device 406 such as a hard disk, and an external interface. It can be implemented by software using a computer configured such that 407 and the like are interconnected by a system bus 409. The program 410 for the processing described in relation to FIG. 1, FIG. 5 and FIG. 6, and the table 411 of the point spread function and blur correction function as described in relation to FIG. 5 or FIG. The data is read into the memory 401 by the drive 405 from the recorded recording medium 404. Alternatively, the program 410 and the table 411 are temporarily stored in the auxiliary storage device 406 and are read from the auxiliary storage device 406 into the memory 401 at the time of processing execution. For example, the image data to be processed is read into the memory 401 through the external interface 407 from the digital camera 408 that captured the image data. The display 402 and the pointing device 403 are used for the graphic display and editing of the point spread function described with reference to FIG.
[0054]
  According to another embodiment of the present invention, the processing functions described in relation to FIGS. 1, 5, and 6 are implemented in an imaging device such as a digital camera.FIG.FIG. 2 is a schematic block diagram of an example of such a digital camera. In FIG. 8, 500 is a CCD image sensor on which an optical image is formed through an optical system (not shown), and 502 is a signal for amplifying an analog image signal output from the CCD image sensor 500, converting it to a digital signal, or the like. A circuit unit 502 is a data memory for storing image data and the like, 503 is a digital signal processor for compressing / decompressing digital image data and converting it into a video signal, and 504 is a monitor display using a liquid crystal display panel or the like. is there. Reference numeral 506 denotes a microcomputer for controlling the inside of the camera, and includes a CPU 510, a ROM 511, a RAM 512, and other external interfaces (not shown). An operation unit 507 is used by the user to input various instructions to the microcomputer 506.
[0055]
  In a state where the shooting button of the operation unit 507 is not pressed, the digital signal processor 503 converts the digital image data continuously output from the signal circuit unit 501 into a video signal and outputs the video signal to the monitor display 504. The video inside can be monitored on the monitor display 504. When the user presses the shooting button on the operation unit 507, digital image data for one image output from the signal circuit unit 501 is temporarily stored in the data memory 502 under the control of the microcomputer 506. Next, the image data is transferred to the digital signal processor 503 and compressed after necessary gradation correction and the like. The obtained compressed image data is transferred to the data memory 502. When the operation unit 507 instructs to output image data to an external personal computer or the like, the compressed image data in the data memory 502 is transferred to the digital signal processor 503 and decompressed under the control of the microcomputer 506. The decompressed image data is written into the data memory 502 again. The image data is transferred to an external personal computer or the like under the control of the microprocessor 506. Such general control inside the digital camera is performed according to a program stored in the ROM 511.
[0056]
  In the digital camera shown here, the means for processing steps described in relation to FIGS. 1, 5 and 6 are implemented by software. That is, a program 520 for causing the microcomputer 506 to execute each processing step described with reference to FIG. 1, FIG. 5, and FIG. 6, a point spread function and a blur as described with reference to FIG. The correction function table 521 is also stored in the ROM 511 (or RAM 512). When the user instructs camera shake correction by operating the operation unit 507, the microcomputer 506 transfers the compressed image data on the data memory 502 to the digital signal processor 503 according to the program 520 and decompresses the decompressed image data. The data is stored in the data memory 502. Since the video signal of the decompressed image data is also output from the digital signal processor 503, the user can confirm the image of the image data on the monitor display 504. Then, the microprocessor 506 executes the camera shake image correction processing according to the present invention as described above on the decompressed image data on the data memory 502. The corrected image data obtained on the data memory 502 is transferred to the digital signal processor 503, the video signal is given to the monitor display 504, and the user can check the image after the camera shake correction. The user can select the blur correction function or edit the point spread function as described above by giving an instruction using the operation unit 507. In this case, the microcomputer 506 displays an image for selecting the blur correction function and a point spread function on the monitor display 504 via the digital signal processor 503. The user can instruct selection of a blur correction function or editing of a point spread function on the screen of the monitor display 504 by operating a pointing device (not shown) provided in the operation unit 507.
[0057]
【The invention's effect】
  As is apparent from the above description, the claims1 to 6, 8 to 13According to the inventions described in the above items, even when a camera shake including non-uniform rotation and translation occurs, an appropriate camera shake correction is possible. In addition, pre-processing and post-processing such as image rotation are not required for camera shake correction, and image quality degradation associated with such processing does not occur. Therefore, it is possible to restore a high-quality image in which camera shake is corrected from a non-uniform camera shake image. Claim6 or 13According to the described invention, calculation for camera shake correction can be performed at high speed by a table reference method and the like, and pre-processing and post-processing such as image rotation are not necessary, so that high-speed processing as a whole is possible. It is. Claim5According to the described invention, since the direction and size of camera shake at a plurality of positions of the image are estimated from the image itself, an appropriate camera shake correction can be performed even for an image taken with a camera having no acceleration sensor or the like. Is possible. Claim12According to the described invention, camera shake correction can be appropriately performed on a captured image without providing an acceleration sensor or the like in the imaging device. Claim1 or 8According to the described invention, the user can observe the image, select the blur correction function determined to be appropriate, perform camera shake correction, and reselect the blur correction function when determined to be necessary. Therefore, it is possible to cope with various hand movement patterns. Claims2 or 9According to this, it is possible to perform appropriate camera shake correction using a blur correction function corresponding to a camera shake pattern at the time of image capturing without involving the user. Claims3, 4, 10 or 11According to the described invention, the user can arbitrarily edit the point spread function according to the difference in the correction according to the shooting environment condition and the difference in the shake pattern as described in JP-A-7-226905. The blur correction function can be optimized. Claim7According to the described invention, it is possible to perform the above-described camera shake image correction, blur correction function optimization, and the like using a computer.
[Brief description of the drawings]
FIG. 1 is a flowchart illustrating an example of a camera shake image correction processing procedure according to the present invention.
FIG. 2 is a diagram showing camera shake vectors at a plurality of positions on an image related to determination of a coordinate transformation matrix.
FIG. 3 is a diagram for explaining a one-dimensional interpolation operation along a camera shake vector;
FIG. 4 is a diagram illustrating an example of a point spread function corresponding to a typical camera shake pattern and an example of editing the point spread function.
FIG. 5 is a flowchart for explaining an example of a procedure for automatically selecting a blur correction function.
FIG. 6 is a flowchart for explaining a procedure for correcting a blur correction function by editing a point spread function;
FIG. 7 is a block diagram illustrating an example of a computer for implementation by software of the present invention.
FIG. 8 is a block diagram showing an example of a digital camera according to the present invention.
[Explanation of symbols]
  400 CPU
  401 memory
  402 Display device
  403 pointing device
  410 Program for image stabilization
  411 Point spread function and blur correction function table
  500 CCD image sensor
  501 Signal circuit
  502 data memory
  503 Digital signal processor
  504 Monitor display
  506 Microcomputer
  507 Operation unit including pointing device
  520 Program for image stabilization
  521 Table of point spread function and blur correction function

Claims (13)

  1. A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
    A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
    A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated in the second step at the position;
    Among the plurality of blur correction functions corresponding to the point spread function of a plurality of typical camera shake patterns including a camera shake pattern with a non-uniform camera shake speed prepared in advance, the blur correction function is used in the third step. A camera shake image correction method comprising: a fourth step in which a user selects a blur correction function for the purpose.
  2. A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
    A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
    A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated in the second step at the position;
    A fourth step of estimating a point spread function due to camera shake of the target image;
    A fifth step of selecting a point spread function most similar to the point spread function estimated in the fourth step from a plurality of typical point spread functions prepared in advance;
    Includes a sixth step of selecting from among said fifth blur correction function corresponding to the typical point spread function selected in step, a plurality of blur correction function prepared in advance,
    A blurring image correction method, wherein the blur function selected in the sixth step is used for blur correction in the third step .
  3. A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
    A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
    A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated by the second step at the position;
    A fourth step of displaying a graph of a point spread function prepared in advance on a display screen;
    A fifth step of editing the point spread function graph displayed on the screen according to an operation of a pointing device;
    A camera shake image correction comprising: a sixth step of correcting a blur correction function used for blur correction in the third step in correspondence with the point spread function graph after editing in the fifth step Method.
  4. A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
    A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
    A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated in the second step at the position;
    A fourth step of displaying a graph of a point spread function due to camera shake of the target image on a display screen;
    A fifth step of editing the point spread function graph displayed on the screen according to an operation of a pointing device;
    A sixth step of generating a blur correction function corresponding to the point spread function after editing in the fifth step ;
    The blur image correction method , wherein the blur correction function generated in the sixth step is used for blur correction in the third step .
  5. The first step includes the step of estimating the direction and size of camera shake at a plurality of positions on the target image based on the target image, and based on the direction and size of the camera shake estimated by the step, 5. The camera shake image correction method according to claim 1, wherein a coordinate transformation matrix representing rotation and translation of a pixel on a target image due to camera shake is determined.
  6. 6. The camera shake image correction method according to claim 1 , wherein in the third step, blur correction is performed by an interpolation calculation along a camera shake vector.
  7.   A computer-readable recording medium on which a program for causing a computer to execute each step of the camera shake image correction method according to any one of claims 1 to 6 is recorded.
  8. An imaging apparatus that obtains a digital image by imaging a subject with an imaging device,
    A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
    Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
    Third means for performing blur correction on each pixel value on the digital target image according to the hand movement vector estimated by the second means at the position;
    Among the plurality of blur correction functions prepared in advance and corresponding to the point spread function of a plurality of typical camera shake patterns including a camera shake pattern having a non-uniform camera shake speed, the third means uses the blur correction function in the third means. An image pickup apparatus comprising: a fourth means for a user to select a blur correction function for use.
  9. An imaging apparatus that obtains a digital image by imaging a subject with an imaging device,
    A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
    Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
    Third means for performing blur correction on each pixel value on the digital image according to the hand movement vector estimated by the second means at the position;
    A fourth means for estimating a point spread function due to camera shake in a digital image;
    A fifth means for selecting a point spread function most similar to the point spread function estimated by the fourth means from a plurality of typical point spread functions prepared in advance;
    A sixth means for selecting a blur correction function corresponding to the typical point spread function selected by the fifth means from a plurality of blur correction functions prepared in advance;
    An image pickup apparatus, wherein the blur function selected by the sixth means is used for blur correction by the third means.
  10. An imaging apparatus that obtains a digital image by imaging a subject with an imaging device,
    A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
    Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
    Third means for performing blur correction on each pixel value on the digital image according to the hand movement vector estimated by the second means at the position;
    A fourth means for displaying a graph of a point spread function prepared in advance on a display screen;
    A fifth means for editing the graph of the point spread function displayed on the screen according to an operation of a pointing device;
    An image pickup apparatus comprising: sixth means for correcting a blur correction function used for blur correction in the third means in correspondence with the point spread function graph after editing by the fifth means.
  11. An imaging apparatus that obtains a digital image by imaging a subject with an imaging device,
    A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
    Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
    Third means for performing blur correction on each pixel value on the digital image according to the hand movement vector estimated by the second means at the position;
    A fourth means for displaying a graph of a point spread function due to camera shake of a digital image on a display screen;
    A fifth means for editing the graph of the point spread function displayed on the screen according to an operation of a pointing device;
    A sixth means for generating a blur correction function corresponding to the point spread function after editing by the fifth means;
    An image pickup apparatus, wherein the blur correction function generated by the sixth means is used for blur correction in the third means.
  12.   The first means includes means for estimating the direction and size of camera shake at a plurality of positions on the digital image based on the digital image, and based on the direction and size of camera shake estimated by the means, The imaging apparatus according to claim 8, wherein a coordinate transformation matrix representing rotation and translation of a pixel on a digital image due to camera shake is determined.
  13.   13. The imaging apparatus according to claim 8, wherein in the third means, blur correction is performed by an interpolation calculation along a camera shake vector.
JP10503399A 1999-04-13 1999-04-13 Camera shake image correction method, recording medium, and imaging apparatus Expired - Fee Related JP4345940B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP10503399A JP4345940B2 (en) 1999-04-13 1999-04-13 Camera shake image correction method, recording medium, and imaging apparatus

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP10503399A JP4345940B2 (en) 1999-04-13 1999-04-13 Camera shake image correction method, recording medium, and imaging apparatus

Publications (2)

Publication Number Publication Date
JP2000298300A JP2000298300A (en) 2000-10-24
JP4345940B2 true JP4345940B2 (en) 2009-10-14

Family

ID=14396714

Family Applications (1)

Application Number Title Priority Date Filing Date
JP10503399A Expired - Fee Related JP4345940B2 (en) 1999-04-13 1999-04-13 Camera shake image correction method, recording medium, and imaging apparatus

Country Status (1)

Country Link
JP (1) JP4345940B2 (en)

Families Citing this family (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005003719A (en) 2003-06-09 2005-01-06 Olympus Corp Photographing device
JP4685966B2 (en) * 2003-09-26 2011-05-18 富士フイルム株式会社 Image processing method, apparatus, and program
JP2005122688A (en) * 2003-09-26 2005-05-12 Fuji Photo Film Co Ltd Image processing method, device, and program
JP4306457B2 (en) 2004-01-08 2009-08-05 株式会社デンソーウェーブ Optical information reader
JP4566591B2 (en) * 2004-03-19 2010-10-20 キヤノン株式会社 Image deformation estimation method and image deformation estimation apparatus
JP4527439B2 (en) * 2004-03-25 2010-08-18 富士フイルム株式会社 Image processing method, apparatus, and program
JP4345651B2 (en) 2004-11-29 2009-10-14 セイコーエプソン株式会社 Image information evaluation method, image information evaluation program, and image information evaluation apparatus
JP4342493B2 (en) 2005-09-22 2009-10-14 三洋電機株式会社 Image stabilizer
US8243148B2 (en) 2005-12-28 2012-08-14 Nittoh Kogaku K.K. Image processing device
WO2007141863A1 (en) * 2006-06-08 2007-12-13 Nippon Computer System Co., Ltd Image processing program and computer readable recording medium with the program recorded therein and image processing device
JP4825748B2 (en) * 2007-07-13 2011-11-30 株式会社モルフォ Image data processing method and imaging apparatus
JP2009033479A (en) * 2007-07-27 2009-02-12 Sharp Corp Camera shake correction device
JP4661922B2 (en) 2008-09-03 2011-03-30 ソニー株式会社 Image processing apparatus, imaging apparatus, solid-state imaging device, image processing method, and program
JP4591603B2 (en) * 2009-01-26 2010-12-01 セイコーエプソン株式会社 Image information evaluation method, image information evaluation program, and image information evaluation apparatus
JP5396964B2 (en) * 2009-03-30 2014-01-22 株式会社ニコン Imaging apparatus and image processing apparatus
JP5379572B2 (en) * 2009-06-19 2013-12-25 キヤノン株式会社 Image processing apparatus, image processing method, program, storage medium, and imaging apparatus
JP2011008322A (en) * 2009-06-23 2011-01-13 Univ Of Tokyo Imaging apparatus and image processing method
JP5627256B2 (en) * 2010-03-16 2014-11-19 キヤノン株式会社 Image processing apparatus, imaging apparatus, and image processing program
KR101839617B1 (en) * 2011-07-11 2018-03-19 포항공과대학교 산학협력단 Method and apparatus for removing non-uniform motion blur using multiframe
US9338354B2 (en) 2011-10-03 2016-05-10 Nikon Corporation Motion blur estimation and restoration using light trails
KR101804215B1 (en) * 2012-03-06 2017-12-05 삼성전자주식회사 A method and an apparatus for estimating a non-uniform motion blur robustly
KR101810876B1 (en) * 2012-03-13 2018-01-26 삼성전자주식회사 A method and an apparatus for debluring non-uniform motion blur of a large scale input image based on a tile unit
KR101844332B1 (en) * 2012-03-13 2018-04-03 삼성전자주식회사 A method and an apparatus for debluring non-uniform motion blur usign multiframe comprises of a blur image and a noise image

Also Published As

Publication number Publication date
JP2000298300A (en) 2000-10-24

Similar Documents

Publication Publication Date Title
US8885061B2 (en) Image processing apparatus, image processing method and program
KR101624450B1 (en) Image processing device, image processing method, and storage medium
US10116879B2 (en) Method and apparatus for obtaining an image with motion blur
US10080006B2 (en) Stereoscopic (3D) panorama creation on handheld device
JP5574423B2 (en) Imaging apparatus, display control method, and program
US8593533B2 (en) Image processing apparatus, image-pickup apparatus, and image processing method
JP4429241B2 (en) Image processing apparatus and method
JP3770271B2 (en) Image processing device
JP4042563B2 (en) Image noise reduction
US9407814B2 (en) Approach for camera control
US5596346A (en) Method and apparatus for applying a function to a localized area of a digital image using a window
US8208017B2 (en) Imaging device, product package, and semiconductor integrated circuit
US8494300B2 (en) Method of notifying users regarding motion artifacts based on image analysis
US7379621B2 (en) Method and system for composing universally focused image from multiple images
US8494299B2 (en) Method of determining PSF using multiple instances of a nominally similar scene
KR101442610B1 (en) Digital photographing apparatus, method for controlling the same, and recording medium storing program to implement the method
JP4720859B2 (en) Image processing apparatus, image processing method, and program
JP4539729B2 (en) Image processing apparatus, camera apparatus, image processing method, and program
JP4760973B2 (en) Imaging apparatus and image processing method
Karpenko et al. Digital video stabilization and rolling shutter correction using gyroscopes
US8289405B2 (en) Image processing apparatus and method thereof
JP5596138B2 (en) Imaging apparatus, image processing apparatus, image processing method, and image processing program
US8244053B2 (en) Method and apparatus for initiating subsequent exposures based on determination of motion blurring artifacts
US8860848B2 (en) Image processing apparatus and method
US7535497B2 (en) Generation of static image data from multiple image data

Legal Events

Date Code Title Description
A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20060309

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20090408

A521 Written amendment

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20090608

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20090708

A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20090709

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20120724

Year of fee payment: 3

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20120724

Year of fee payment: 3

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20130724

Year of fee payment: 4

LAPS Cancellation because of no payment of annual fees