JP2007267114A - Shaking correction apparatus, photographic apparatus using the same, shaking correction method, shaking correction program, and recording medium for shaking correction program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology for precisely correcting shaking, for only a slow motion image with high accuracy for detecting motion vector. <P>SOLUTION: A shaking correction apparatus includes a frame image acquisition means for acquiring a frame image from a moving picture or still pictures that are consecutive in time series; a motion vector calculation means for calculating a motion vector, denoting the motion of the frame images as a whole; a stable image discrimination means for discriminating the frame image to be a stable image with a slow motion, when the motion vector is smaller than a prescribed threshold; and an image correction means for generating correction frame image, resulting from correcting a camera shake of the frame image, when the discrimination means determines that the frame image is the stable image. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

The present invention relates to a camera shake correction apparatus that acquires a frame image from a moving image or a still image continuous in time series, and corrects camera shake included in the acquired frame image, a photographing apparatus using the same, a camera shake correction method, and a camera shake correction program. And a recording medium on which a camera shake correction program is recorded.

In recent years, a desired image such as an athletic meet has been taken as a moving image by using a video camera, a digital camera, and a photographing device such as a mobile phone having an image photographing function. At this time, if the photographing apparatus is photographed by hand, camera shake may occur in the photographed image. For this reason, conventionally, an image processing technique for correcting camera shake during shooting has been proposed (for example,
Patent Document 1).

The image processing apparatus of Patent Document 1 calculates a motion vector of a frame image in a captured image,
Camera shake is corrected by shifting the frame image by an integral vector calculated from the calculated motion vector.

Japanese Patent No. 2957851

However, if the motion of the entire frame image becomes too fast to follow with the eyes, the moving image becomes a non-clear image and the motion vector detection accuracy is lowered, so that there is a problem that the accuracy of camera shake correction is also lowered. .

The present invention has been made paying attention to such a conventional problem, and an object of the present invention is to provide a camera shake correction device that corrects camera shake accurately only for a slow-moving image with good motion vector detection accuracy. It is an object of the present invention to provide an image capturing apparatus, an image stabilization method, an image stabilization program, and a recording medium for an image stabilization program.

In order to solve the above problems, an image stabilization apparatus according to the present invention includes a frame image acquisition unit that acquires a frame image from a moving image or a time-series continuous still image, and a motion vector that indicates the motion of the entire image of the frame image. A motion vector calculation means for calculating the image, a stable image determination means for determining that the frame image is a stable image with a slow image movement when the motion vector is smaller than a predetermined threshold, and the frame image is the stable image. The gist of the invention is that the image correction means includes a correction frame image that generates a corrected frame image in which the camera shake of the frame image is corrected when it is determined to be an image.

According to the camera shake correction device of the present invention, the stable image determination unit determines the stable image when the motion vector calculated by the motion vector calculation unit from the frame image acquired by the frame image acquisition unit is smaller than a predetermined threshold. The frame image is determined to be a stable image by the means. Then, the image correcting unit generates a corrected frame image in which the camera shake of the frame image is corrected when the stable image determining unit determines that the frame image is a stable image.
In this way, camera shake correction can be performed on a frame image in which the motion of the entire frame image is slow, that is, the motion vector detection accuracy is high.

In the camera shake correction apparatus according to the present invention described above, the image correction unit sets the previous frame image as the correction frame image in time series when it is determined that the frame image is not the stable image.

When the frame image is not a stable image, the movement of the image in the frame image is fast, and it is difficult for a person who views the displayed image to follow with the eyes. By using the previous image in the time series order as the correction frame image, the same effect as frame dropping can be obtained, and the correction frame image can be easily followed by the eyes.

In the camera shake correction apparatus according to the present invention, the predetermined threshold value is a value determined based on a frame rate and a screen size when the corrected frame image is displayed.

In this way, the predetermined threshold can be determined from the viewpoint of whether the image moves fast or slow when the correction frame image is actually displayed.

In the above-described camera shake correction device according to the present invention, the predetermined threshold value is that the movement of the image on the display screen when viewing the display screen on which the correction frame image is viewed is 30 deg /
The value is determined based on the frame rate and the screen size so as to be smaller than sec.

The standard value of the follow-up movement of the eyeball, which is a movement for keeping the object moving at low speed in the center of the visual field, is 30 deg / sec. Based on the frame rate and screen size, if a predetermined threshold value is determined so that the motion of the image when viewing the display screen displaying the corrected frame image is smaller than 30 deg / sec, only the slowly moving image that can be followed by the eyes Camera shake can be corrected.

The above-described camera shake correction apparatus according to the present invention further includes warning message output means for outputting a warning message when it is determined that the frame image is not the stable image.

In this way, when there is a fast-moving frame image that is difficult to follow with the eyes, the user of the image stabilizer can be notified by outputting a message, so the user can create a fast-moving moving image. It is possible to take measures such as correction.

In addition, a camera shake correction method corresponding to the above-described camera shake correction apparatus of the present invention shows a frame image acquisition step of acquiring a frame image from a moving image or a time-series continuous still image, and movement of the entire frame image. A motion vector calculation step of calculating a motion vector, a stable image determination step of determining that the frame image is a stable image with slow motion of the image when the motion vector is smaller than a predetermined threshold, and the frame image The present invention includes an image correction step of generating a corrected frame image in which camera shake of the frame image is corrected when it is determined that the image is a stable image.

In addition, a camera shake correction program corresponding to the above-described camera shake correction apparatus of the present invention shows a frame image acquisition function for acquiring a frame image from a moving image or a time-series continuous still image, and movement of the entire frame image. A motion vector calculation function for calculating a motion vector, a stable image determination function for determining that the frame image is a stable image with slow motion of the image when the motion vector is smaller than a predetermined threshold, and the frame image The gist is to cause a computer to realize an image correction function for generating a corrected frame image obtained by correcting camera shake of the frame image when it is determined that the image is a stable image.

The present invention can also be understood as a recording medium on which the above-described camera shake correction program is recorded. Recording media include flexible disks, CD-ROMs, DVD-ROMs,
Various media readable by a computer such as an IC card and a punch card can be used.

According to the above-described camera shake correction method, camera shake correction program, and recording medium, effects equivalent to those of the camera shake correction device of the present invention can be obtained.

The present invention can also be understood as a photographing apparatus including the above-described camera shake correction device. In this way, it is possible to correct camera shake of a captured moving image without using another device, and to accurately correct camera shake only for a slow moving image with good motion vector detection accuracy. be able to.

  Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is an explanatory diagram illustrating a schematic configuration of a camera shake correction apparatus. The camera shake correction apparatus 100 according to the first embodiment includes a frame image acquisition unit 110, a motion vector calculation unit 120, a stable image determination unit 130, and an image correction unit 140. The camera shake correction apparatus 100 acquires a moving image from the input device 200 and moves the camera shake of the moving image. Are corrected and output to the output device 210.

As the input device 200, a device capable of outputting a moving image such as a video, a digital video camera (DVC), a broadcast, and a moving image database (DB) for storing a moving image can be used. Further, a storage medium such as a hard disk or a memory card in which moving images are stored may be used as the input device 200.

The frame image acquisition unit 110 sequentially acquires frame images F (n) (n is an integer indicating a time-series order) from the moving image output from the input device 200.

The motion vector calculation unit 120 calculates the entire image of the frame image F (n) from the frame image F (n) acquired by the frame image acquisition unit 110 and the previous frame image F (n−1) in time series order. A motion vector V (n) indicating motion is calculated.

The stable image determination means 130 is the motion vector V (
The magnitude of n) is compared with a predetermined threshold, and when the magnitude of the motion vector V (n) is smaller than the predetermined threshold, the frame image F (n) is determined to be a stable image.

The image correcting unit 140 determines that the frame image F (n) is a stable image,
A corrected frame image HFG (n) in which the camera shake of the frame image F (n) is corrected is generated and output to the output device 210. When it is determined that the frame image F (n) is not a stable image, the previous frame image F (n−1) is generated as the corrected frame image HFG (n),
Output to the output device 210.

As the output device 210, a CRT, a projector, a television, or the like can be used.
Further, the corrected frame image HFG (n) may be distributed via a network. Further, the output frame 210 may be a storage medium such as a hard disk or a DVD-ROM, and the corrected frame image HFG (n) may be recorded.

The camera shake correction apparatus 100 having the above configuration is configured by a computer that implements the processing in each of the above-described units by a software program and installs the camera shake correction program as the processing program. This computer (not shown) includes a CPU, ROM, RAM
, A general computer equipped with a hard disk, an interface, etc., connected to a keyboard, a display, etc. The computer's ROM stores a camera shake correction program, and the CPU functions as the camera shake correction device 100 when the camera shake correction program is executed. In the following, the processing of each unit executed by the CPU in this computer will be described with reference to the flowchart of FIG.

When the process is started, the frame image acquisition unit 110 acquires the frame image F (n) from the input device 200 (step S100).

Next, the motion vector calculation unit 120 calculates a motion vector indicating the motion of the entire frame image F (n) from the frame image F (n) and the previously acquired frame image F (n−1). V (n) is calculated (step S110).

Here, the motion vector V (n) will be described with reference to an explanatory diagram for explaining the frame image F (n) and the motion vector V (n) in FIG.

FIG. 3 (upper side) shows an example of the first frame image F (1) in time-series order acquired by the frame image acquisition unit 110, and the second and third frame images F (2) and F (3). Show. FIG. 3 is an example in which the house shown in the image is moving in the lower left direction.

The camera shake correction apparatus 100 cuts out an image of a predetermined size from the frame image F (n) at a position that reduces image shake due to camera shake, thereby correcting the camera shake correction frame image HF.
G (n) is generated. The default cut-out positions of the corrected frame image HFG (n) are frame images FK (1) to FK (3) indicated by broken lines in FIG. Here, as shown in FIG. 3, the frame image FK (n) is smaller in screen size than the frame image F (n) by a predetermined correction limit value Smax (Sxmax in the X direction and Symax in the Y direction). Become an image. In the first embodiment, a predetermined correction limit value Smax (X component is Sxmax, Y component is Symax) is set to a predetermined value in advance in the camera shake correction program.

Similarly, in FIG. 3 (lower side), frame images F (1) to F (3) and FK (1) are arranged so that the objects shown in the frame images F (1) to F (3) are at the same position. -FK (3) was shown. A difference in relative position of the frame image FK (n) when arranged in this way is a motion vector. That is, the motion vector V (2) is a difference between the relative positions of the frame images FK (2) and FK (1), and the motion vector V (3) is the frame image FK (3) and the frame image FK (2).
) Relative position difference. In the example of FIG. 3, there is no frame preceding the frame image F (1), so the motion vector V (1) is zero.

The relative position between the frame images can be obtained by a known image processing method such as image pattern matching or feature point tracking. Since the specific relative position calculation processing method is not the essence of the present invention, the description thereof will be omitted.

Next, when the magnitude of the motion vector V (n) is smaller than the predetermined threshold T, the stable image determination unit 130 determines that the frame image F (n) is a stable image with slow motion of the entire image. (Step S120 (FIG. 2)).

The camera shake correction apparatus 100 according to the first embodiment defines the frame image F (n) as a stable image when the movement of the entire image of the frame image F (n) is slow and can be followed with eyes. Then, when the frame image F (n) is a stable image, camera shake is corrected, and when the motion is so fast that the eye cannot follow, that is, when the frame image F (n) is not a stable image, it is not a camera shake correction but is so fast that the eye cannot follow. A correction process is performed to make it easy to see the motion image.

In this way, for the frame image F (n) whose motion is slow and the detection accuracy of the vector V (n) is high, high-precision camera shake correction using the high-precision motion vector V (n) can be performed. it can. On the other hand, for the frame image F (n) that has a fast motion and a low detection accuracy of the vector V (n), it is possible to perform a correction process that makes it easy to see the fast motion instead of correcting the camera shake.
In addition, since it is only necessary to determine which correction process is performed once, the determination process is simple.

Further, the criterion for determining whether or not the image is a stable image may be determined based on the performance of the optical system that captured the moving image, such as the flow of the frame image F (n), rather than whether the image can be tracked with the eyes. In this way, the motion vector detection accuracy is further improved, and the camera shake can be corrected with high accuracy.

FIG. 4 is an explanatory diagram for explaining a method of calculating the predetermined threshold T. The display screen 300 is
This is a screen for displaying the corrected frame image HFG (n) on the output device 210 such as a CRT, a projector, or a TV. The viewer 310 who views the corrected frame image HFG (n) displayed on the display screen 300 has a distance that is three times the screen size that is said to be the optimal viewing position for viewing the display screen 300. It is at a position away from the display screen 300 by 3H.

The standard value of the follow-up movement of the eyeball, which is a movement for keeping the object moving at a low speed at the center of the visual field, is 30 deg / sec. Therefore, if the motion of the entire image of the frame image F (n) is smaller than 30 deg / sec, it can be said that the frame image F (n) is a stable image with a slow motion that can be followed by the eyes. When the motion of the entire image is 30 deg / sec, T representing the magnitude of motion in the frame image F (n) in terms of the number of pixels is expressed by the following equation (1).
It can be calculated by

T = k × W × (30 / F) / A
= K × W × 30 / (2 × F × arctan (W / 6H) (1)
H: Height of display screen 300 W: Length of horizontal width of display screen 300 k: Number of horizontal pixels / W of corrected frame image HFG (n)
F: Frame rate (Frame / sec)
A: Viewing angle when viewing screen 300 from viewer 310
A = 2 × arctan ((W / 2) / 3H)

Thus, the threshold value T is calculated so that the motion of the frame image F (n) becomes 30 deg / sec using the frame rate F when actually displayed and the display screen sizes H and W. Therefore, it is possible to accurately determine whether the image is so fast that it cannot be followed by the eyes.

The X component and Y component of the motion vector V (n) calculated by the motion vector calculation means 120 are represented by the number of pixels as described with reference to FIG. Therefore, the magnitude of the motion vector V (n) is T
If it is smaller, it can be determined that the motion of the frame image F (n) is a slow motion stable image that can be followed by the eyes.

Next, it is determined whether or not the frame image F (n) is determined to be a stable image by the camera shake determination unit 130 (step S130). When it is determined that the frame image F (n) is a stable image (step S130: YES), the image correction unit 140 uses the corrected frame image F (n) obtained by correcting the camera shake of the frame image F (n). Generate (Step S140
).

FIG. 5 is an explanatory diagram for explaining correction of camera shake by the image correction unit 140. FIG.
Will be used to describe a method of correcting camera shake in the frame images F (2) and F (3) described with reference to FIG. The correction position vectors S (2) and S (3) are vectors indicating positions where the correction frame images HFG (2) and HFG (3) are cut out from the frame images F (2) and F (3). It is represented by a relative position from the frame images FK (2) and FK (3) at the same position as the frame image.

Here, if the correction position vectors S (2) and S (3) are vectors obtained by changing the directions of the motion vectors V (2) and V (3) in the opposite directions, the corrected frame images HFG (2),
The position of the house shown in HFG (3) is not changed, and camera shake can be corrected.

On the other hand, when it is determined that the frame image F (n) is not a stable image (step S130:
NO) sets an image cut out from the previous frame image F (n−1) as a corrected frame image HFG (n) in order to make it easy to see a fast-moving image (step S150). The previous frame image F (n−1) is cut out by the same method as that for the camera shake correction when the correction position vector S (n) is 0.

In this way, by generating the corrected frame image HFG (n) from the previous frame image F (n−1), it is possible to produce an effect such as frame dropping, and to make it easy to see a fast moving image. Can do. Further, when the frame image F (n) that is not a stable image continues for several frames, the same corrected frame image HFG (n) may be continuously output like a still image.

Next, the corrected frame image HFG (n) is output to the output device 210 (step S160).
). When the camera shake correction of the frame image F (n) is thus completed, it is next checked whether or not the frame image F (n) is the end of the moving image (step S170). Frame image F (
If n) is the end of the moving image (step S170: YES), the camera shake correction program is terminated.

If the frame image F (n) is not the end of the moving image (step S170: NO), 1 is added to n to correct camera shake of the frame image F (n + 1), and step S100 is performed.
Return to. In this way, correction of camera shake of the frame image F (n) proceeds in time-series order, one frame at a time.

As described above, the camera shake correction apparatus 100 according to the first embodiment can accurately correct camera shake according to the motion of the frame image.

Hereinafter, modified examples related to the camera shake correction apparatus according to the first embodiment will be described.
(Modification 1)
An image having the same size as the correction frame image of one black color is prepared in advance. Step S150
Then, when it is determined that the frame image F (n) is not a stable image, the image correcting unit 140 generates a black one-color image as the corrected frame image HFG (n). In this way, when the corrected frame image (n) is reproduced by the output device 210 such as a television or a projector, it can be immediately understood that the image is so fast that the movement cannot be observed with the eyes. In addition, the output device 210 is
If nothing is displayed when there is no corrected frame image HFG (n), the corrected frame image HFG (n) may not be generated when the image is not a stable image. In this way, since the output device 210 does not display anything, it has the same effect as displaying a black one-color image.

(Modification 2)
In step S150, if the frame image F (n) is not a stable image, the message output means uses a warning message output means connected to the computer as a warning message output means connected to the computer. Output from the speakers. In this way, since it is immediately known that the image is so fast that it cannot be followed by the eyes, the creator of the moving image can immediately take measures such as recreating the moving image. The warning message output by the message output means may be a beep sound output from a speaker, a melody, or a light using LED or the like.

(Modification 3)
A photographing device such as a digital video camera equipped with a CPU, ROM, and RAM may function as a camera shake correction device. The camera shake correction program according to the first embodiment is stored in the ROM of the image capturing apparatus, and the camera shake correction program is executed by the CPU of the image capturing apparatus, thereby causing the image capturing apparatus to function as a camera shake correction apparatus. In this way, it is possible to perform camera shake correction of a moving image shot by the shooting apparatus at the same time as shooting. In addition, when there is an image that moves so fast that the moving image cannot be followed by the eyes, the photographer can be immediately notified by the method of the first modification or the second modification.

Explanatory drawing explaining the structure outline of a camera-shake correction apparatus. The flowchart explaining the process sequence of a camera shake correction apparatus. Explanatory drawing for demonstrating frame image F (n) and motion vector V (n). Explanatory drawing for demonstrating the calculation method of a predetermined threshold value. Explanatory drawing for demonstrating the correction of camera shake by an image correction means.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... Camera shake correction apparatus, 110 ... Frame image acquisition means, 120 ... Motion vector calculation means, 130 ... Stable image determination means, 140 ... Image correction means, 200 ... Input apparatus, 210 ...
Output device, 300 ... display screen, 310 ... viewer, F (n) ... frame image, FK (n) ...
Frame image, V (n) ... motion vector, S (n) ... corrected position vector, HFG (n) ...
Corrected frame image.

Claims (9)

Frame image acquisition means for acquiring a frame image from a moving image or a still image continuous in time series;
Motion vector calculating means for calculating a motion vector indicating the motion of the entire frame image;
Stable image determination means for determining that the frame image is a stable image with a slow image movement when the motion vector is smaller than a predetermined threshold;
An image stabilization apparatus comprising: an image correction unit configured to generate a corrected frame image obtained by correcting camera shake of the frame image when it is determined that the frame image is the stable image. The camera shake correction device according to claim 1,
The camera shake correction apparatus according to claim 1, wherein when the frame image is determined not to be the stable image, the image correction unit sets the previous frame image in time series order as the correction frame image. The camera shake correction device according to claim 1,
The camera shake correction apparatus according to claim 1, wherein the predetermined threshold value is a value determined based on a frame rate and a screen size when the correction frame image is displayed. The camera shake correction device according to claim 3,
The predetermined threshold is determined based on the frame rate and the screen size so that the movement of the image on the display screen when viewing the display screen on which the corrected frame image is displayed is smaller than 30 deg / sec. A camera shake correction device characterized by being a value. The camera shake correction device according to claim 1,
A camera shake correction apparatus, further comprising warning message output means for outputting a warning message when it is determined that the frame image is not the stable image. A frame image acquisition step of acquiring a frame image from a moving image or a still image continuous in time series;
A motion vector calculation step of calculating a motion vector indicating the motion of the entire image of the frame image;
A stable image determination step of determining that the frame image is a stable image with slow image movement when the motion vector is smaller than a predetermined threshold;
An image correction method comprising: an image correction step of generating a corrected frame image obtained by correcting camera shake of the frame image when it is determined that the frame image is the stable image. A frame image acquisition function for acquiring a frame image from a moving image or a still image continuous in time series, and
A motion vector calculation function for calculating a motion vector indicating the motion of the entire frame image;
A stable image determination function for determining that the frame image is a stable image with a slow image movement when the motion vector is smaller than a predetermined threshold;
When the frame image is determined to be the stable image, a camera shake program that causes a computer to realize an image correction function that generates a corrected frame image in which camera shake of the frame image is corrected. A computer-readable recording medium on which the camera shake correction program according to claim 7 is recorded. The imaging device provided with the camera-shake correction apparatus of any one of Claims 1-5.

Images