JP2015109547A - Shake correction method, shake correction device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To correct a shake of an image even when a three-dimensional position of a camera is moving.SOLUTION: A shake correction device comprises: an image acquisition unit which acquires an image; an image division unit which divides the image into tile shape small images; a camera movement estimation unit which estimates a camera movement based on the image and the small images, which is movement of the camera during exposure time while photographing the image, and is represented by six degrees of freedom; a shake correction kernel calculation unit which calculates a shake correction kernel based on the estimated camera movement; a shake correction unit which performs shake correction for each small image by use of the calculated shake correction kernel; and an image integration unit integrating the small images on which the shake correction is applied to generate an output image that is the same size as the image.

Description

  The present invention relates to a shake correction method, a shake correction apparatus, and a program for correcting shake in an image.

  In the field of computer vision, many studies have been made to correct a blur and restore a clear image when the image blurs due to underexposure during shooting. For example, Non-Patent Document 1 defines a blurred image as an image obtained by adding together clear images taken from different camera postures during the exposure time, and simultaneously with the camera posture and the clear image. A method of restoration has been proposed.

Oliver Whyte, Josef Sivic, Andrew Zisserman, and Jean Ponce, "Non-uniform Deblurring for Shaken Images", International Journal of Computer Vision, Volume 98, Issue 2, June 2012, p.168-186

  As described above, the method of simultaneously restoring the camera posture and the clear image enables high-speed computation by considering the premise that the camera does not move greatly during the exposure time. In the method described in Non-Patent Document 1, the three-dimensional position of the camera during the exposure time is unchanged, and the image is restored under the assumption that the camera posture (orientation) is a factor of blurring. .

  However, although the above assumptions may hold under general conditions such as shooting by the photographer, in the case of shooting with a camera attached to the human body, such as a wearable camera or action camera, Since the camera itself may move greatly in accordance with the movement of the body, a situation occurs where the above assumption is not satisfied. In such a situation, there is a problem that the method described in Non-Patent Document 1 may not be able to correct blurring.

  The present invention has been made in order to solve the above-described problem, and is a blur correction method capable of correcting blur caused in an image even when the three-dimensional position of the camera is moving. An object of the present invention is to provide a shake correction apparatus and a program.

  One aspect of the present invention is a shake correction method performed by a shake correction apparatus, an image acquisition step of acquiring an image, an image division step of dividing the image into tile-shaped small images, the image and the small image A camera motion estimation step for estimating the camera motion of the camera during the exposure time in capturing the image based on the camera motion and expressed using six degrees of freedom; and a blur correction kernel based on the estimated camera motion A first blur correction kernel calculating step for calculating the first blur correction step, a first blur correction step for performing blur correction on each of the small images using the calculated blur correction kernel, and a small image subjected to the blur correction. An image integration step of integrating and generating an output image having the same size as the image.

  According to another aspect of the present invention, in the shake correction method, the camera motion estimation step includes: a camera motion hypothesis generation step that generates a plurality of camera motion hypotheses; and a camera shake correction kernel for each camera motion hypothesis. A second blur correction kernel calculation step to calculate; a second blur correction step to correct blur of the small image using the blur correction kernel calculated in the second blur correction kernel calculation step; and the second A hypothesis image generation step for generating a hypothesis image corresponding to each of the camera motion hypotheses from the small image that has undergone the blur correction in the blur correction step, and calculating a sharpness of the hypothesis image corresponding to each of the camera motion hypotheses And a camera motion determination step for determining the camera motion based on the sharpness of each hypothesis image. Characterized in that it has Tsu and up, the.

  According to another aspect of the present invention, an image acquisition unit that acquires an image, an image division unit that divides the image into tile-shaped small images, and an exposure time in shooting the image based on the image and the small image A camera motion estimator that estimates camera motion represented by 6 degrees of freedom, a shake correction kernel calculator that calculates a shake correction kernel based on the estimated camera motion, A blur correction unit that performs blur correction on each of the small images using the calculated blur correction kernel, and image integration that generates an output image having the same size as the image by integrating the small images subjected to the blur correction A motion compensation device comprising: a motion compensation unit.

  One embodiment of the present invention is a program for causing a computer to execute the above-described blur correction method.

  According to the present invention, the camera motion during the exposure time when the image is captured based on the image is estimated, and the blur correction is performed based on the estimation result, so that the three-dimensional position of the camera is moving. Even in this case, the blurring that occurs in the image can be corrected.

It is a block diagram which shows the structural example of the blurring correction apparatus 1 in embodiment which concerns on this invention. It is a figure which shows an example of the image division by the image division part 12 in the embodiment. It is a flowchart which shows the blurring correction process which the blurring correction apparatus 1 in the embodiment performs. It is a flowchart which shows the process of step S303 in which the camera motion estimation part 13 estimates a camera motion. It is a figure which shows an example of the table which the camera motion estimation part 13 uses in the case of voting.

  Hereinafter, a shake correction method, a shake correction apparatus, and a program according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration example of a shake correction apparatus 1 according to an embodiment of the present invention. As shown in FIG. 1, the shake correction apparatus 1 includes an image acquisition unit 11, an image division unit 12, a camera motion estimation unit 13, a shake correction kernel calculation unit 14, a shake correction unit 15, an image integration unit 16, and a hypothesis evaluation. A storage unit 21 is provided.

  The image acquisition unit 11 acquires image data obtained by shooting or the like. The image acquisition unit 11 may be, for example, an image sensor (imaging device) in a digital camera, a video camera, or the like, or an interface that inputs an image obtained by an external device such as a digital camera.

  The image dividing unit 12 divides the image acquired by the image acquiring unit 11 into small tile-shaped images having a predetermined size. At this time, the image dividing unit 12 divides the images so that adjacent small tile-shaped images overlap. FIG. 2 is a diagram illustrating an example of image division by the image division unit 12 in the present embodiment. As shown in FIG. 2, the image is divided into tile-shaped areas of a certain size and divided into small images including a plurality of tile-shaped areas. At this time, adjacent small images are determined so as to include overlapping tile-shaped regions. For example, in the example illustrated in FIG. 2, the small image A and the small image B in the image are determined so as to include the region a and the region b in an overlapping manner.

  The camera motion estimation unit 13 predicts a plurality of hypotheses about the camera motion of the camera that captured the image acquired by the image acquisition unit 11. For each predicted hypothesis, the camera motion estimation unit 13 calculates a shake correction kernel for each small image using the camera motion when the hypothesis is correct, and corrects the shake of the small image using the calculated shake correction kernel. . The camera motion estimation unit 13 performs evaluation on an image restored by integrating each small image whose blur is corrected. The camera motion estimation unit 13 associates the evaluation result with each hypothesis about the camera motion and stores it in the hypothesis evaluation storage unit 21. The camera motion estimation unit 13 estimates a likely camera motion from the hypothesis based on the evaluation result calculated for each hypothesis.

The blur correction kernel calculation unit 14 calculates a blur correction kernel for each small image in the image based on the camera motion estimated by the camera motion estimation unit 13.
The blur correction unit 15 performs blur correction on each small image input from the image dividing unit 12 using the blur correction kernel calculated by the blur correction kernel calculation unit 14 for each small image.
The image integration unit 16 integrates the small images that have undergone the shake correction by the shake correction unit 15 to generate a clear image. The image integration unit 16 outputs the generated image to the outside as an output image.

  FIG. 3 is a flowchart showing a shake correction process performed by the shake correction apparatus 1 according to the present embodiment. When the shake correction process is started in the shake correction device 1, the image acquisition unit 11 acquires image data Im photographed by a camera or the like (step S301).

  The image dividing unit 12 divides the image indicated by the image data Im acquired by the image acquiring unit 11 into a plurality of small images (step S301). At this time, as shown in FIG. 2, the image dividing unit 12 divides the image so that adjacent small images are overlapped with each other. The number of divisions (number of small images), the degree of overlap, and the like are determined in advance. In order to perform a frequency domain operation on each small image, the image dividing unit 12 applies a window function to the small image. For example, if the degree of overlap is 50% with respect to the area of the small image and a Bartlett-Han window function is used as the window function, the overlap can be integrated with the original image.

  The camera motion estimation unit 13 estimates the camera motion during the exposure time based on the small image obtained by the image dividing unit 12 (step S303).

  FIG. 4 is a flowchart showing the process of step S303 in which the camera motion estimation unit 13 estimates the camera motion. When the camera motion estimation unit 13 starts estimating the camera motion, the camera motion estimation unit 13 predicts a hypothesis of the camera motion (step S401). The camera motion is represented by parameters (Tx, Ty, Tz, θx, θy, θz) of six degrees of freedom between the three-dimensional position and the camera posture. Note that (Tx, Ty, Tz) is a predetermined three-dimensional position in the world coordinate system, and (θx, θy, θz) is a rotation amount with respect to the X axis, the Y axis, and the Z axis in the world coordinate system.

  Let the camera motion at the exposure start time be (0, 0, 0, 0, 0, 0). In addition, the sampling number k within the exposure time is determined in advance. In general, the camera moves freely during the exposure time, but the hypothesis of the camera motion predicts the hypothesis assuming that the camera moves at a constant speed from the initial position to the end position. That is, when the camera motion at the end point (kth sampling) is Mk = (Tx, Ty, Tz, θx, θy, θz), the time between the exposure start time and the exposure end time (kth sampling). The camera motion Mi of (i-th sampling) can be expressed as Mi = (i−1) Mk / (k−1).

  Hypothesis prediction is performed by random sampling. A range of values that each of the camera motion elements (Tx, Ty, Tz, θx, θy, θz) can take is determined in advance, and a combination of values obtained by random sampling from a uniform distribution is assumed as a hypothesis. . The camera motion estimation unit 13 finally selects N predetermined combinations of six parameters, and sets a combination of the selected parameter values as a hypothesis.

  The camera motion estimation unit 13 calculates a blur correction kernel for each small image based on the established hypothesis (a combination of randomly sampled parameter values) (step S402). The blur correction kernel performs an approximation that is uniform for each small image, thereby calculating a non-uniform blur correction kernel associated with free motion as a whole.

In addition, camera calibration is performed in advance by capturing a known pattern with a camera that captures an image to be processed, and an internal matrix indicating internal parameters of the camera is acquired in advance. The camera motion estimation unit 13 calculates a shake correction kernel a using a known technique based on the established hypothesis and the internal matrix. As a known technique, for example, the technique described in Reference Document 1 below can be used. Note that the camera motion estimation unit 13 takes an arbitrary value as in the technique described in Reference Document 1.
[Reference 1] Neel Joshi, Sing Bing Kang, C. Lawrence Zitnick, and Richard Szeliski, "Image Deblurring using Inertial Measurement Sensors", ACM Transactions on Graphics, Volume 29, Issue. 4, July 2010, Article No.30

  The camera motion estimation unit 13 corrects each small image by using the shake correction kernel a calculated for each small image (step S403). When the data of the small image is g, the small image f after blur correction can be calculated by the calculation in the frequency space by the following equation (1).

In Expression (1), F (•) represents a function for performing conversion to a frequency space by Fourier transform. F ⁻¹ (•) represents a function that performs inverse transformation, that is, transformation into time space by inverse Fourier transformation. In addition, ◯ is an operation symbol indicating a product operation between corresponding vector elements. F (•) ^* represents a complex conjugate of F (•).

  The camera motion estimation unit 13 generates a single corrected image using each small image f obtained by the correction according to the equation (1) (step S404). The image generated by the camera motion estimation unit 13 from each small image f is an image having the same size as the image acquired by the image acquisition unit 11. An image is generated from each small image f by, for example, a simple superimposition operation when the degree of overlap is 50% when an image is divided into small images and a Bartlett Hann window function is used as a window function. It can be carried out. That is, the processing is to add the small image f obtained by the correction to the area of the small image g in the original image.

The camera motion estimation unit 13 evaluates an image generated from the small image f (step S405). Evaluation of an image generated from the small image f, that is, an image after correction, can be performed using an evaluation index of sharpness metric with respect to the image described in Reference Document 2, for example. In addition to the evaluation index described in Reference Document 2, any evaluation function can be used as long as an evaluation value inversely proportional to the degree of blurring in the image or the sharpness of the edge of the subject in the image can be calculated.
[Reference 2] Rony Ferzli and Lina J. Karam, "A No-Reference Objective Image Sharpness Metric Based on the Notion of Just Noticeable Blur (JNB)", IEEE Transaction on Image Processing, Vol.18, No.4, April 2009, p.717-727

The camera motion estimation unit 13 stores the hypothesis and the score S indicating the evaluation result of the image obtained by the correction based on the hypothesis in the hypothesis evaluation storage unit 21 in association with each other (step S406).
The camera motion estimation unit 13 determines whether or not the number of sets of hypotheses and scores S stored in the hypothesis evaluation storage unit 21 has reached a predetermined number N (step S407).

If the number of pairs of stored hypotheses and scores S has not reached N (step S407: NO), the camera motion estimation unit 13 returns the processing to step S401 to establish a new hypothesis.
On the other hand, when the number of stored hypotheses and scores S has reached N (step S407: YES), the camera motion estimator 13 includes the N hypotheses stored in the hypothesis evaluation storage 21. A camera motion within the exposure time is determined using the set with the score S (step S408).

  In the camera motion estimation, the camera motion estimation unit 13 normalizes the score S for each hypothesis so that “(the sum of the scores S for N hypotheses) × (sampling number k within the exposure time)” is 1. Turn into. Also, a ballot box in which the camera motion of 6 degrees of freedom is quantized is prepared, and the score S ′ normalized for the camera motion of all hypotheses is voted on the corresponding ballot box.

  The vote by the camera motion estimation unit 13 will be specifically described. FIG. 5 is a diagram illustrating an example of a table used by the camera motion estimation unit 13 for voting. The table used for voting is created in advance by quantizing the range of values that each parameter indicating the camera motion can take. As shown in FIG. 5, the table includes items for identifying a combination of quantized values, parameters indicating camera motion (Tx, Ty, Tz, θx, θy, θz), and the number of votes. And a row for each quantized value combination (voting box).

  Each parameter indicating the camera motion can be limited to a possible value such as 0 ≦ Tx, Ty, Tz <10, 0 ≦ θx, θy, θz <360. In the example shown in FIG. 5, the parameter Tx is quantized to 0.1 units. The other parameters are similarly quantized. For example, the table includes a ballot box having a number “2”, a camera motion (0.1, 0, 0, 0, 0, 0), and a vote number “0.03”.

  The camera motion estimation unit 13 performs voting using N hypotheses. At this time, the camera motion estimation unit 13 performs voting for each k camera motions constituting each hypothesis. The score S 'corresponding to each camera motion is voted on the ballot box of the parameter combination closest to the camera motion parameter. Here, the closest parameter combination is, for example, the one having the smallest distance between the parameter value of the ballot box and the camera motion (the square root of the square of the difference for each parameter) or the smallest sum of the differences for each parameter. .

  As a result of the above-described voting, the camera motion indicated by the parameter corresponding to the ballot box with a high number of votes indicates that the time during which the camera was in the position and posture indicated by the camera motion is long. . The camera motion estimation unit 13 outputs the camera motion having the highest vote count to the blur correction kernel calculation unit 14 as a camera motion estimation result during the exposure time.

  In addition, this vote does not apply the hypothetical condition that the camera motion is constant speed, and the quantized three-dimensional position and posture are weighted according to the score. Can express camera movement.

  Thereby, it is possible to represent a camera motion of sampling number k × six dimensions while keeping the dimension of the camera motion estimation space as low as six dimensions. In addition, since the processing for each of the N hypotheses (each processing from step S401 to step S407) can be performed in parallel, the time required for camera motion estimation is reduced by performing the parallel operation. It is possible.

  In the ballot box in the table shown in FIG. 5, a combination of parameters in which all but six parameters are zero is exemplified, but the combinations of the values of the six parameters are possible ranges in the camera motion. Any combination or all combinations of values obtained by quantizing the values may be used.

Returning to FIG. 3, the description of the blur correction process will be continued.
When the camera motion estimation unit 13 estimates the camera motion, the blur correction kernel calculation unit 14 calculates a blur correction kernel for each small image using the camera motion estimated by the camera motion estimation unit 13 (step S304). Note that the calculation of the blur correction kernel by the blur correction kernel calculation unit 14 is the same as the processing performed by the camera motion estimation unit 13 in step S402.

  The blur correction unit 15 performs blur correction on each small image using the blur correction kernel calculated by the blur correction kernel calculation unit 14 (step S305). The blur correction in each small image by the blur correction unit 15 is the same as the process performed by the camera motion estimation unit 13 in step S403.

  The image integration unit 16 integrates the small images that have undergone the blur correction by the blur correction unit 15 to generate an output image, outputs the generated image (step S306), and ends the blur correction process. The integration of the small images by the image integration unit 16 is the same as the process performed by the camera motion estimation unit 13 in step S404.

  The blur correction device 1 according to the present embodiment estimates the camera motion, which is the movement of the camera, from an image in which blur occurs because the camera has moved during the exposure time when shooting an image in the above-described blur correction processing. Blur correction is performed for each small image by the blur correction kernel based on the estimation result, and an image subjected to blur correction is generated from the small image subjected to blur correction. By this blur correction process, a clear blur-free image can be calculated from the blurring image. Further, as described above, since the process for estimating the camera motion can be easily performed in parallel, it is possible to reduce the time required for the calculation and obtain a clear and blur-free image in a short time.

  In addition, according to the shake correction apparatus 1 of the present embodiment, a blur-free image can be obtained from an image that has been shaken because the camera has moved during the exposure time of shooting. Even when the camera moves greatly, it is possible to obtain an image in which blurring is corrected.

  Further, in the shake correction apparatus 1 of the present embodiment, the camera motion estimation unit 13 estimates the camera motion based on N hypotheses determined by random sampling, so that an image and an image are used as prior information used for the estimation. Since it is sufficient if there is an internal parameter (internal matrix) of the camera to be acquired, it is easy to apply to various usage scenes.

  You may make it implement | achieve the blurring correction apparatus 1 in embodiment mentioned above with a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” is a program that dynamically holds a program for a short time, like a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

DESCRIPTION OF SYMBOLS 1 ... Shake correction apparatus 11 ... Image acquisition part 12 ... Image division part 13 ... Camera motion estimation part 14 ... Shake correction kernel calculation part 15 ... Shake correction part 16 ... Image integration part 21 ... Hypothesis evaluation storage part

Claims (4)

A blur correction method performed by the blur correction device,
An image acquisition step of acquiring an image;
An image dividing step of dividing the image into tile-shaped small images;
A camera motion estimation step for estimating a camera motion of the camera during an exposure time in capturing the image based on the image and the small image and represented using 6 degrees of freedom;
A first blur correction kernel calculating step for calculating a blur correction kernel based on the estimated camera motion;
A first blur correction step for performing blur correction on each of the small images using the calculated blur correction kernel;
An image integration step of integrating a small image subjected to blur correction to generate an output image having the same size as the image;
A blur correction method comprising: The blur correction method according to claim 1,
The camera motion estimation step includes
A camera motion hypothesis generation step for generating a plurality of camera motion hypotheses;
A second blur correction kernel calculating step for calculating a blur correction kernel for each camera motion hypothesis;
A second blur correction step of correcting blur of the small image using the blur correction kernel calculated in the second blur correction kernel calculation step;
A hypothesis image generation step of generating a hypothesis image corresponding to each of the hypotheses of the camera motion from the small image subjected to the blur correction in the second blur correction step;
An evaluation step for calculating a sharpness of a hypothesis image corresponding to each of the hypotheses of the camera motion;
A camera motion determination step for determining the camera motion based on the sharpness of each of the hypothesis images;
A blur correction method comprising: An image acquisition unit for acquiring images;
An image dividing unit for dividing the image into tile-shaped small images;
A camera motion estimator for estimating a camera motion represented by using 6 degrees of freedom during the exposure time in capturing the image based on the image and the small image;
A blur correction kernel calculation unit that calculates a blur correction kernel based on the estimated camera motion;
A blur correction unit that performs blur correction on each of the small images using the calculated blur correction kernel;
An image integration unit that generates an output image having the same size as the image by integrating the small images subjected to blur correction;
A shake correction apparatus comprising:   A program for causing a computer to execute the shake correction method according to claim 1.

Images