JP3325823B2 - Video still image display method and apparatus, and video still image display program storage recording medium - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and an apparatus for displaying a continuously shot image as a still image when constructing a user interface such as an image database system and a digital image editing system, and a recording storing the program. Regarding the medium.

[0002]

2. Description of the Related Art In order to search for a specific scene on a video tape using a video deck, there is usually no other way but to repeat fast-turning and rewinding, and there is a problem that it takes time and effort.

[0003] As one method for solving this,
There is known a method of detecting a transition (cut) in a scene of a video, extracting one still image for each scene as an image (referred to as a representative image) representing the scene, and displaying a list of the images. According to this method, there is an effect that a desired scene can be visually searched in a short time only by listing the representative images without watching the video in chronological order. However, this method has the following problems.

(1) In a scene including camera operations such as pan (operation for rotating the camera in the horizontal direction) and tilt (operation for rotating the camera in the vertical direction), a representative image representing the entire image of the scene is extracted. It is difficult. As an example, consider an image of a person's foot to face while moving the camera from bottom to top. Since the first image of this scene shows only the feet, it is inappropriate as a representative image in the sense that only a part of the scene is shown, and the last image shows only the face, so Are also inappropriate as representative images. (2) The type, direction, and speed of the camera operation are not represented in the representative image extracted from the video. In addition to the pan and tilt, the types of camera operation include zoom (operation for continuously changing the size of the subject by changing the angle of view of the zoom lens), truck (operation for moving the camera in the left and right direction), and crane operation. (Operation to move the camera up and down). (3) The motion of the subject cannot be expressed only by the representative image extracted from the video.

As one method for solving the problem (1), there is an approach in which a plurality of continuous images are combined to generate an image having a wide field of view, and the image is used as a representative image (for example, see Japanese Patent Application Laid-Open No. H05-205,1992). -304675). this is,
By calculating the movement of the camera in the video by image processing, and combining the images while translating and enlarging / reducing the amount of the shift, a single wide-field image is generated. . This method solves the problem (1) but does not solve the problems (2) and (3).

As one method for solving the problem (2), there is a method in which the type and direction of a camera operation are represented by arrows, and are superimposed on a representative image extracted from a video and displayed. ("Proposal of an interactive video editing method applying recognition technology" (Transactions of the Institute of Electronics, Information and Communication Engineers, D-II, Vol. J75-D-1)
1, No. 2, pp. 216-225)). FIG. 2 shows an example of the expression by this method. 21, 22, 2 in FIG.
Numerals 3 indicate pan, tilt, and zoom, respectively, by arrows.

However, this conventional technique cannot express the speed of camera operation, and does not solve the problems (1) and (3). Furthermore, since the arrow is not associated with the image content, there is a problem that it is difficult to know how the image content changes due to the camera operation. That is, the image before the camera operation is displayed as the representative image, but the image after the camera operation is not displayed. Further, it does not describe a method of expressing motion information (movement information of a subject) other than a camera.

[0008]

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems of the prior art, and generates a representative image representing the entire scene even if the camera moves, and operates the camera. The type, direction, speed, etc. are displayed in association with the representative image, and further displayed in association with the representative image so that the motion of the subject can be intuitively understood. It is an object to provide a program storage medium.

[0009]

According to a first aspect of the present invention, there is provided a video still image display method for extracting motion information included in a video and synthesizing a plurality of images constituting the video in accordance with the motion information. , Generate one representative image and include it in the video
And classify the movements and visualize the movement information according to the results
Select a method and use the selected visualization method to generate motion information.
It is characterized by being visualized in association with the representative image .

According to a second aspect of the present invention, in the video still image display method of the first aspect, the operation speed of the camera or the subject is extracted as motion information, and the operation speed is visualized.

According to a third aspect of the present invention, there is provided a video still image display device,
A video input unit for inputting a continuous video, a motion information extracting unit for extracting motion information included in the video, and a plurality of images constituting the video are synthesized according to the motion information; an image synthesizing means for generating an image, included in the video
And classify the movements and visualize the movement information according to the results
A motion information visualizing means for selecting a method, visualizing the motion information in association with the representative image by the selected visualizing method, and an image output means for outputting an image or information obtained from the image synthesizing means and the motion information visualizing means. It is characterized by having.

According to a fourth aspect of the present invention, there is provided a recording medium for storing a still image display program for a video, wherein a continuous image is inputted, and a motion process included in the image is extracted in an algorithm for displaying a still image representing the scene. When a processing process of creating a representative image by combining a plurality of images constituting the image according to the motion information, Ru is included in the video
Classify motions and visualize motion information according to the results
A method of selecting a method and visualizing the information in association with the representative image by the selected visualization method .

[0013]

DESCRIPTION OF THE PREFERRED EMBODIMENTS One embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a processing flowchart of an embodiment of a video still image display method according to the present invention. Each step of the processing flow in FIG. 1 includes a video input device for inputting an image sequence, a hard disk or a device similar thereto, a memory of an image buffer necessary for calculating motion information, or image synthesis, or the like. A computer (CP) that includes a display device such as a device and a display for displaying a still image, and controls the hard disk, the buffer memory, and the display device based on a predetermined procedure.
U) or a device equivalent thereto can be appropriately executed. Since this type of computer system is well known, it is not shown.

In FIG. 1, a step 11 inputs N images I1, I2,..., IN taken continuously. If the input video is composed of a plurality of scenes, a scene change is detected in advance, divided into shot units, and in step 11, an image sequence is input for each shot. Step 12 is to set a variable i indicating an image number to 1
Initialize to Step 13 is an image buffer (Com
p) is initialized and the first image I1 is stored in the image buffer. In step 14, camera parameters are extracted as motion information from two adjacent images Ii and Ii + 1. Camera parameters quantitatively describe camera movement,
Details of the extraction method will be described later. Step 15
Transforms the image Ii + 1 according to the camera parameters, and step 16 combines and stores the transformed image with the image already stored in the image buffer Comp. In the example of the image sequence shown in FIG. 11A, since the camera is panned from left to right, in step 15, the image Ii + 1 is transformed in parallel to the left, and in step 16, the transformed image Ii + 1 is transformed. Are sequentially combined with the previous images to obtain a horizontally long combined image shown in 16A. A step 17 adds 1 to a variable i, and a step 18 checks whether all image sequences have been processed. If not, the process returns to the step 14; Step 19 is Step 14
Is calculated based on the camera parameters obtained in step (1) by a method described later, and the path is overwritten on the image in the image buffer to draw the final still image 19A.
Is displayed.

Here, the method of extracting camera parameters in step 14 will be described. In order to extract camera parameters, first, a camera model is defined. The camera model is a mathematical model that indicates to which point in the image at the next time a point in the two-dimensional image moves. In the present embodiment, the following (1) considering the pan, tilt, and zoom of the camera is used. ) The model of the formula is used. (X ', y') = (ax + b, ay + c) (1) The model of equation (1) represents the correspondence between a certain image point (x, y) and the point (x ', y') of the immediately preceding image. . Where a,
b and c are called camera parameters, which correspond to zoom, pan, and tilt of the camera, respectively. For example, if the camera operation is panning right without zooming, a = 1, b
> 0, c = 0. If the camera is zoomed in, a <1. As a camera model, a translation model, an affine model, a Bilinear model, a projection model, or the like can be used other than the one defined by the equation (1).

From the two images Ii and Ii + 1, camera parameters ai, bi and ci representing the movement of the camera between them are estimated as follows. In order to estimate an error amount between the two images Ii and Ii + 1, a mean square error MSE (Mean Square) is used.
Err) is defined as in the following equation (2).

[0017]

(Equation 1)

Where the sum (Σx, y) is calculated for all corresponding pixels, and N is the number of corresponding pixels. As an evaluation function of the amount of error, there is an M estimation method that introduces a concept of robust statistics in addition to MSE.

It is assumed that a, b, and c that minimize the error MSE are camera parameters ai, bi, and ci to be obtained. That is,

[0020]

(Equation 2)

As a solution of the optimization problem defined by the equation (3), MSEi is simply changed while a, b and c are changed.
A simple method of calculating (ai, bi, ci) and outputting a parameter that achieves the minimum value may be used, but the problem is that the processing cost is enormous.

As a solution to this, an image (called a pyramid image) in which the resolution is reduced stepwise is created,
First, camera parameters are estimated using the image with the lowest resolution, and the estimated camera parameters are used as initial values.
There is a method of efficiently and accurately estimating parameters by repeating the step of obtaining an optimum solution for an image with a higher resolution while increasing the resolution stepwise (HS Sawhneyand S. Ayer: Compact Repre).
sentation of Videos Through Dominantand Multip
le Motion Estimation, IEEE Transactions on
Pattern Analysis and Machine Intelligence, Vo
l. 18, No. 8, pp. 814-830,199
6).

The image transformation operation in step 15 is performed by calculating the camera parameters ai, bi,
This is performed using an expression in which ci is substituted.

The method described above is for estimating camera parameters by image processing. However, if the movement is measured by a sensor attached to the camera and the camera parameters can be obtained, the method may be used. Good.

Next, the procedure for calculating and drawing the camera operation trajectory in step 19 will be described. From equation (1), k =
, N−1, the correspondence between the point (x, y) on the image Ik + 1 and the point (x ′, y ′) on the image I1 is Wk (x, y) = (x ′, y) ') = (A ₁ (a ₂ (... ( _ak x + b _k ) + b _k-1 ) + ... b ₁ , a ₁ (a ₂ (... (a _ky + c _k ) + c _k-1 ) ) +... + C ₁ ) (4) where W0 (x, y) is used for convenience.
= (X, y).

From equation (4), in the coordinate system in the image buffer Comp, the point sequence Traj representing the locus of the center point of the image is as follows: Traj = ｛W0 (W / 2, H / 2), W1 (W / 2, H / 2),..., WN-1 (W / 2, H / 2)｝ (5) Here, W and H represent the width and height of the image, respectively.

The trajectory of the camera operation can be represented by drawing a point sequence Traj using polygonal lines and arrows. In the case of pan and tilt, respectively, 31 and 3 in FIG.
Figure 2 is obtained. As shown in FIG. 19A, in order to indicate the start point and end point of the camera operation, FIG.
It is also preferable to draw a rectangle surrounding the start image and the end image.

In the drawing of the camera operation trajectory, an arrow may be drawn only at the end point of the polygonal line, as shown at 37 in FIG. 3. However, in this case, the speed of the camera operation is not expressed. is there. For the purpose of expressing the speed of the camera operation, it is preferable to draw arrows at regular time intervals, for example, as shown at 31 in FIG. 3, and to express the length of the arrow corresponding to the speed.

Next, a procedure for classifying motions included in a video and selecting a visualization method of motion information according to the result will be described. If the camera is panning or tilting,
Although the trajectory of the image center point may be used as indicated by 31 and 32 in FIG.
In the case of zooming, since the image center point does not change, it is not possible to visualize the direction of zooming (whether zooming in or out). In order to solve this problem, it is effective to classify and determine the motion of the camera based on “whether or not the camera operation includes zoom” and to switch the expression method according to the determination result. This will be described with reference to the processing flowchart of FIG.

In FIG. 5, step 51 is a zoom ratio Zi (i = 1, 1) required to classify camera operations.
2,..., N) are calculated as Zi = a1 × a2 ×. Step 52 determines whether the camera operation includes zoom. Specifically, the zoom ratio Zi
Is Zmin for all i = 1, 2,..., N
If <Zi <Zmax is satisfied, it is determined that the zoom operation is not included. Here, Zmin and Zmax are thresholds given in advance, and are set so as to satisfy Zmin <1 and Zmax> 1. If the camera operation does not include zooming, the locus of the center point of the image is calculated and drawn in step 53 as described above. If the camera operation includes zooming, in step 54, the trajectories of the four corners of the image are similarly calculated using equation (4) and the trajectories are drawn.

At step 54, still images such as 33 and 34 in FIG. 3 are obtained for the zoomed-in and zoomed-out images, respectively. Further, in a scene where the camera is tilted while zooming, a still image as indicated by 35 in FIG. 3 is obtained. Reference numeral 36 in FIG. 3 is an example of the illustrated method when the camera is rotating around the optical axis.

In the above embodiment, the camera operation trajectory is drawn using arrows. However, it is also possible to indicate the direction by the brightness of a straight line or to express the speed by the thickness of an arrow. Also, an example in which the expression method is switched depending on the presence or absence of a zoom operation has been described. For example, in a video in which a camera is panning, it is determined whether or not the scene is following a subject, and a visualization method of the movement of the subject is determined. Can be switched.

FIG. 4 is a processing flowchart of another embodiment of the video still image display method of the present invention. This is also realized in practice with the aid of a computer. In FIG. 1, the motion of the camera is visualized as motion information. In FIG. 4, the motion of the subject is extracted as motion information.

In FIG. 4, a step 41 inputs a continuous image sequence. In the present embodiment, it is assumed that the images I1,..., IN (41A) captured by the fixed camera are input. Of course, it is also possible to consider the movement of the camera as in FIG.

In step 42, a subject area Obj1 is specified for the first image I1. In step 43, k =
The movement of the subject is tracked by sequentially examining the image Ik for 2, 3,..., N. Specifically, the object region Objk-1
And searching for an area having the highest correlation from the image Ik and outputting it as the subject area Objk of the image Ik, for k = 2,..., N. 43 in FIG.
A indicates this. As a method for calculating the correlation, various methods such as a well-known color histogram difference and a luminance correlation can be used.

In step 44, the images I1 and IN are synthesized so that the first and last object areas Obj1 and ObjN stand out, thereby obtaining a synthesized image as shown at 44A in FIG. Specifically, the combined image Comp (x, y) is combined according to the following equation. (i) Point (x, y) is not included in Obj1 or ObjN Comp (x, y) = {I1 (x, y) + IN (x, y)} / 2 (ii) Point (x, y ) Is included in Obj1 Comp (x, y) = I1 (x, y) (iii) Point (x, y) is included in ObjN Comp (x, y) = IN (x, y).

In step 45, the movement locus of the subject is calculated and drawn. The subject motion trajectory is represented by a subject area Obji (i =
1,..., N). This subject movement trajectory is represented by a composite image Comp (x, y).
The final image 46A is drawn on the display and displayed on a display or the like.

FIG. 6 is a block diagram of an embodiment of a video and still image display device according to the present invention. In FIG. 6, a video input unit 61 inputs a video signal, detects a break in the video, and extracts an image sequence for each continuous scene.
Pass to. The motion information extracting means 63 holds a plurality of images in a buffer, and extracts the motion of the camera or the motion of the subject using the method described in the embodiment of FIGS. The image synthesizing unit 64 synthesizes an image sequence based on the extracted motion information and holds the image sequence in an internal buffer. The motion information visualizing means 64 visualizes the motion information by using the same method as described in the embodiment of FIGS. This
At the same time, the motion information is
Select the method of visualizing the information. The image output unit 65 superimposes the images obtained from the image synthesizing unit 64 and the motion information visualizing unit 65 for each video, and outputs and displays the images on a display, a window in a computer screen, or a printer.

The means of each unit of the video / still image display device described with reference to FIG. 6 is similar to the respective steps of the processing flow of FIG. 1 and FIG.
A video input device for inputting image sequences, a hard disk or equivalent device, a memory or equivalent device for calculating motion information, or an image buffer required for image synthesis, or for displaying still images A computer (CPU) that includes a display device such as a display and controls the hard disk, the buffer memory, and the display device based on a predetermined procedure or a device equivalent thereto can be appropriately executed.

Although the embodiment of the present invention has been described above, the processing flow of FIGS. 1 and 4 and the function of each unit in FIG. 6 are realized by a computer-readable recording medium such as a floppy disk as a video still image display program. Memory card, MO, compact disk (CD-ROM),
It can be recorded on a tape or the like and provided.

Further, in the above embodiment, a case where one image is displayed as a still image has been described. However, an image composed of a plurality of scenes is divided into images, and a still image is formed by the method and apparatus described in each embodiment. Creating and displaying a list of a plurality of the still images is also suitable for the purpose of making it possible to grasp video contents in a short time.

[0042]

According to the present invention, a representative image is created by combining images according to motion information, and the motion information is visualized in association with the representative image. Can also extract a representative image representing the entire video, and display the type, direction, and speed of camera operation in association with the representative image, and display the movement of the subject in association with the representative image. There is an effect that can be done. As a result, motion information about the video can be displayed in such a way that the user can understand it intuitively, so that the content of the video can be grasped efficiently and the target video can be visually recognized in a short time. Be able to search.

[Brief description of the drawings]

FIG. 1 is a process flowchart showing an embodiment of the present invention.

FIG. 2 is a schematic diagram for explaining a camera operation visualization method in the related art.

FIG. 3 is a schematic diagram showing a still image display example according to the present invention.

FIG. 4 is a process flowchart showing another embodiment of the present invention.

FIG. 5 is a processing flowchart illustrating an example of an embodiment of a motion information visualization procedure.

FIG. 6 is a block diagram showing a configuration of still another embodiment of the present invention.

[Explanation of symbols]

 61 Video scene input means 62 Motion information extraction means 63 Image synthesis means 64 Motion information visualization means 65 Image output means

Continuation of the front page (72) The inventor Takashi Sato 3-19-2 Nishi-Shinjuku, Shinjuku-ku, Tokyo Nippon Telegraph and Telephone Corporation (56) References JP-A-6-303562 (JP, A) (58) Survey Field (Int.Cl. ⁷ , DB name) H04N 5/76-5/956

Claims (4)

(57) [Claims] 1. A method for displaying a continuous image as a still image, comprising: a first step of extracting motion information included in the image; and synthesizing a plurality of images constituting the image according to the motion information. Then, a second step of generating one representative image, and classifying motion included in the video, and
Select the information visualization method, and select the information visualization method.
A third step of visualizing the motion information in association with the representative image. 2. The video still image display method according to claim 1, wherein the first step extracts a motion speed of the camera or the subject as motion information, and the third step visualizes the motion speed. Video still image display method. 3. An image input means for inputting a continuous image.
And a motion information extracting means for extracting motion information contained in the video.
And a plurality of images forming a video according to the step and the motion information.
An image synthesizing unit that synthesizes and generates one representative image, and classifies motion included in the video, and performs motion according to the result.
Select the information visualization method, and select the information visualization method.
Visualizing the motion information in association with the representative image
Motion information visualization means, obtained from the image synthesis means and the motion information visualization means.
And a video output means for outputting images and information . 4. An image for displaying a continuous image as a still image.
A computer reading the image still image display program
A process for extracting motion information contained in a video,
Combines multiple images that compose a video according to the motion information
Process to create a representative image
And classify the movements and visualize the movement information according to the results
A method, and the motion information is selected by the selected visualization method.
Processing for visualizing information in association with the representative image
Video still image display program storage characterized by including
recoding media.