JPH11261946A - Video display method, device therefor and recording medium recorded with the video display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To display input video images which are classified so as to grasp contents of the images sequentially from the images usable for editing with a high value while displaying all the video images in the case of editing the video images. SOLUTION: In this video display method, after a received video image is classified for each shot period, motion information included in each shot period is analyzed and camera work information and camera-shake information of the received video image are analyzed based on the motion information and a representative image for each video period in the unit of camera works and in the unit of presence/absence of camera-shake is displayed on a display device.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] The present invention relates to a method for analyzing a video image,
The present invention relates to a video display method and apparatus for displaying a partial frame of a video as a representative image, and a recording medium storing a display processing program.

[0002]

2. Description of the Related Art With the spread of small-sized video cameras and VCRs, ordinary people often record and shoot images.

[0003] However, when editing a video, it is necessary to view and understand the contents, and it is necessary to perform repetitive operations such as repeatedly watching the video, which makes the editing operation itself cumbersome and difficult. , Most people do not work.

[0004] It seems that there are some problems. One of the problems is that it is difficult to control the contents of the shot video. For one thing, conventional editing machines edit on tape media,
It is necessary to perform operations in the time axis direction such as rewinding the tape. For this reason, only sequential access is permitted, random access is not possible, and a large amount of time is required for editing work, and it is considered that this was not used because of this.

[0005] To promote video editing, 1. It is easy to grasp the contents of video materials for video editing; 2. The video editing work is easy. As a result of the video editing work, it is considered necessary to easily understand what the content of the produced video is and to view the video easily.

Several conventional methods have been proposed to analyze the above problems.

As one proposal for easily grasping and viewing the contents of a video, information of a cut point of video information is extracted and displayed, and camera work information is extracted. Some methods have been proposed, such as displacing the images, connecting all the scenes captured during the camera work, arranging them on a screen or on paper, and displaying them. (Yukin Taniguti, Yoshinobu T
onomura, `` PanoramaExceps: Extracting and packing Pan
oramas for Video Browsing '' 1996 ACM Multimedia 97)
In this example, a video shot change is detected, and a representative image of a portion corresponding to more stable camera work is displayed. When viewing a video, the content can be grasped to some extent only by showing only a portion corresponding to the shot change and the camera work. But,
According to this method, a portion where camera motion is stable or stable is displayed, but a portion where camera shake information exists is not displayed.

In video editing, it has been empirically known that a stable camera work including a state where the camera is stationary is more easily used for editing. However, even if camera shake occurs, a subject with high utility value may be photographed on the screen, so that it is not always possible to delete the image of these camera shake portions.

Note that the camera shake information generally refers to a case where the camera vibrates little by little in the left, right, up and down directions. In this case, however, a case where the speed rapidly changes even during camera work in the same direction. Shake. This is because if the speed changes even if the direction of the camera does not change other than the slight vibration, the image becomes difficult to use for editing.

[0010]

As described above, in order to edit a video, when analyzing a video and displaying a representative image, it is necessary to display all of the images and to display the contents from those having high value that can be used for editing. It is considered necessary to make it easy to grasp.

SUMMARY OF THE INVENTION The present invention has been made to solve such a situation, and its purpose is to display all of the images while editing the images in order from those having the highest value that can be used for editing. It is an object of the present invention to provide an image display method and apparatus capable of classifying and displaying input images so that the contents can be grasped, and a recording medium recording a display processing program.

[0012]

In order to achieve the above object, the invention of claim 1 includes a video shot classification step of classifying an input video for each shot section, which is included in each shot section. A motion analysis stage for analyzing motion information;
Based on the motion information, a camera shake information analysis step of analyzing camera work information and camera shake information of the input video, and displaying a representative image of each video section corresponding to a camera work unit and a unit of presence or absence of camera shake on a display device. And displaying a representative image.

Further, the invention of claim 2 is characterized in that, in the camera shake information analyzing step, a temporal change of a motion vector is calculated to extract camera shake information and camera work information.

According to these inventions, by analyzing camera work information and camera shake information based on a time-series motion vector, a stable camera work including a stationary state is displayed when a representative image in a video is displayed. By changing the display level based on the presence or absence of camera shake information while displaying a part of the middle image, it is possible to preferentially display an image that is highly useful for editing while displaying all the contents.

According to a third aspect of the present invention, there is provided the apparatus according to the first aspect of the present invention, wherein the shots are classified into shot sections by shot section, and each shot section includes the shot section. Motion analysis means for analyzing motion information;
Based on the motion information, camera shake information analyzing means for analyzing camera work information and camera shake information of the input video, and display a representative image of each video section corresponding to a camera work unit and a unit of presence or absence of camera shake on a display device. And a representative image display means.

A fourth aspect of the present invention relates to a recording medium that describes the first aspect of the present invention with a computer-readable program and enables the image display method of the present invention to be performed using the program. A video shot classification process of classifying the input video for each shot section;
Motion analysis means for analyzing motion information included in each shot section; camera shake information analysis processing for analyzing camerawork information and camera shake information of an input video based on the motion information; A computer-readable program including a representative image display process of displaying a representative image of each video section corresponding to the presence / absence unit on a display device is recorded.

[0017]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an image display method and apparatus according to the present invention will be described in detail with reference to the illustrated embodiments.

FIG. 1 is a flowchart showing one embodiment of the video display method of the present invention.

In FIG. 1, input video data Vi is input to video shot classification means 11. The video shot classification means 11 analyzes the input video based on the input video data Vi and classifies the input video into shot sections 1.

Next, the motion analysis means 12 analyzes the motion in the image of each classified shot section 1 to calculate a motion vector 2. Next, the information of the motion vector 2 is input to the camera shake information analysis unit 13, and the camera shake and camera shake information 3 are extracted by the camera shake information analysis unit 13. Next, the camera work and camera shake information 3 are input to the representative image display means 14, and the representative image display means 14
With this, the representative image 4 is determined and displayed.

FIG. 2 is a block diagram showing an embodiment of a video display device for realizing the method of FIG.

The video display device shown in FIG. 2 includes a recording medium 30 for temporarily storing input video data Vi. The input video data Vi is stored in the recording medium 30.
Is temporarily stored in the video shot classifying unit 40. Alternatively, the input video data Vi is directly input to the on-video shot classification unit 40.

When the input video data Vi is input, the video shot classification section 40 extracts a shot section 21 of the input video from the input video data Vi, and uses the information of the extracted shot section 21 as a video motion analysis section 41. To enter.

The video motion analysis unit 41 determines whether the shot section 21
, The motion vector 22 inside the screen in each shot section is extracted. The information of the motion vector 22 is input to the image blur analysis unit 42. The video camera shake analysis unit 42 analyzes the camera work and camera shake from the distribution of the motion vector 22, and obtains the camera work and camera shake information 23.
Is output. This camera work and camera shake information 23
The image data is stored in the recording medium 30 and input to the representative image selection unit 43. The representative image selection unit 43 selects the representative image 24 from the input camera work and camera shake information 23, displays the representative image 24 on a display device screen (not shown) via the display interface 44, and displays the contents of the video to the user 50. Present.

At this time, what is presented to the user 50 is a representative image classified by the camera work and camera shake information 23.

Here, the input video data Vi may be a digital video which can be analyzed by a computer.
Or JPEG or the like. The parameters in the time and space directions may be arbitrary. Further, when the image is not a digital image, an analog image signal that has been digitized by a capture tool or the like may be input, or a configuration in which a digitizing function is incorporated in the image shot classification unit 40 may be employed. .

The input video data Vi does not need to be one when counted, for example, in file units.
More than one may be used. In the case of a plurality of video shots, if their temporal and spatial formats (size, frame rate, number of audio tracks, etc.) are different, a function to match them is built in the video shot classifying unit 40. do it.

By the way, in the video shot classifying section 40,
The input video is classified into shot units. This is based on the assumption that a difference between the same pixels between frames is calculated, and a shot change has occurred when a large number of pixels having a large difference exist. The shot change includes a shot change such as a gradual change, but may or may not have a detection function for this.

The video motion analysis unit 41 calculates a motion vector 22 in the video. As an example, each screen is subdivided into blocks, and an area similar to another frame is searched for each block while shifting the corresponding position.
As a criterion for the similar area, a difference from the similar area is calculated, and a position shift when the difference is the smallest is calculated as the motion vector 22. Looking at the distribution of the motion vectors 22 calculated in this manner, many vectors of the motion vector “0” appear when the camera is stationary. FIG.
(A) and (b) show typical examples of a case where camera shake has occurred and a case where camera work has appeared, respectively. FIG. 3A shows an original image 30 in a case where camera shake occurs.
1 and its motion vector 302 are shown. FIG. 3B shows an original image 303 and its motion vector 303 when there is camera work. The motion vectors 302, 3
The direction of the arrow 03 indicates the moving direction of the image,
As shown in the figure, when there is camera shake, the direction of the arrow varies in the time direction, and when there is camera work, the direction of the arrow is almost the same.

If camera work exists, a high correlation area that is considered to be similar is searched for by shifting the position between the images of the shots, and a positional shift showing a high correlation is recorded. , Represented by a vector.

When camera work or camera shake exists, vectors on a frame appear intensively with a certain direction and a certain distance. If this is other than camera work due to camera work or camera shake, this vector does not concentrate in a certain direction and has a random value. This is often seen when randomly moving subjects occur.

Next, the video camera shake analysis unit 42 for extracting camera work and camera shake information 23 will be described.

When the distribution of the motion vector 22 is biased, it is necessary to determine whether this is due to camera work or a change due to camera shake. The image blur analysis unit 42 performs this determination. This determination can be made by determining what kind of change has occurred over a plurality of frames. Assuming that the distribution of the bias vector is a local movement direction, in the case of camera work, this movement direction is constant over a plurality of frames, whereas camera work due to camera shake has no regularity,
It is known that the direction of movement changes over time. From this, by counting the movement directions over a plurality of frames, it is possible to determine whether the camera is stationary, intentional camera work, or camera work due to camera shake.

An acceleration is used as one example of the calculation method.

When the camera work = 0, that is, when the camera is stationary, the motion vector is "0" and the average vector is also "0". For stable camera work,
Vectors corresponding to similar camera work appear over several frames. Therefore, the average of the camera work is also similar. Stable camera work generally appears over several frames. Therefore, the difference between the motion vectors of each frame is calculated. This is one acceleration. If the camera work has similar motion vectors, the size of the acceleration vector of such a motion vector is close to “0”.

On the other hand, in the case of camera shake, the motion vector changes randomly, the speed and direction are not stable, and the acceleration vector does not become "0" but becomes large. If the acceleration vector is large, it is determined that camera shake has occurred.

Therefore, the distribution of the acceleration vector is calculated, and a case where a large number of vectors having a large acceleration vector appear in the same shot section is defined as a camera shake state.

That is, when a large number of motion vectors "0" and acceleration vectors "0" are present, a stationary state is set. When the motion vector is other than "0" and the acceleration vector is "0", a stable camera work is performed. If the motion vector is other than "0" and the acceleration vector is also other than "0", it is determined that the camera work is unstable, that is, a camera shake is occurring.

According to this determination method, acceleration is generated by accelerating or decelerating in the same direction, and a negative acceleration is generated if the vehicle moves suddenly in the opposite direction. In addition, camera work involving a rapid change in speed is also determined as camera shake information, and it can be expected that a video with high utility value and stable camera work can be presented preferentially.

A representative image is selected based on the camera work and camera shake information 23 thus obtained.

Next, the representative image selecting section 43 will be described. As a representative image, a frame immediately after a shot change is first selected in each shot section. Further, when there is camera work even in the same shot section, the camera work is started. If there is camera shake, the frame is the part where the camera shake information has occurred. Similarly, when the camera work ends, the camera shake ends, or the camera works in another direction, the frame is selected as the representative image. Expressing this in a time series, images as shown in FIG. 4 are selected as representative images 401 to 405 for each shot section.

Next, an interface 44 for displaying this is displayed.
Will be described.

The user 50 has two requests, that is, a request to preferentially display an image by a stable camera work which is easy to use for editing, and a request to simultaneously watch all the image sections. For this reason, FIG.
As shown in, for each video section, each video section is classified based on the average vector size of the motion vector on each frame and the average vector size of the acceleration vector. The representative images 501 to 504 are displayed at different display positions. For example, a motion vector and an acceleration vector that are both small are displayed as being stationary in the upper part of the vertical axis of the screen, and those that are both larger are displayed below the vertical axis of the screen. The criterion may be a sum of a motion vector and an acceleration vector,
A ratio weighted by a certain ratio may be used. In the example shown in FIG. 5, the head of the entire video section is aligned with the left end of the screen, and is arranged so that the utility value decreases from the top to the bottom of the screen in accordance with the stillness and the moving state. This example has an advantage that the video section in the best state can be selected in order.

FIG. 6 is a diagram showing another display example, in which the order of time is respected and the representative image 6 is displayed in accordance with the stationary and stable states.
This is an example in which display is performed with display levels (display positions) of 01 to 606 changed. By using such a display interface, there is an advantage that it is easier to search for a video section while being aware of the time taken and recorded, as compared with the example shown in FIG. In addition, since it is possible to know the time zone in which the video is stable, it is possible to edit mainly the time zone in which the video is stable.

It is assumed that the smaller the acceleration, the higher the stability. By arranging the arrangement as shown in FIG. 6, the editor using the video editing apparatus for video still images can use a stable camera work which is easy to use for editing. The images can be selected in order from the one from the beginning. As a result, the convenience of the user's selection is improved.

In the display interface 44,
If necessary, a function of reproducing an arbitrary portion (video section) can be added.

The above-described video display method is described in a computer program, recorded on a recording medium such as a CD-ROM, read by a computer used by a video editor, and executed, thereby being stored in the recording medium 30. The edited video data can be edited while selecting a representative image according to the editor's intention.

As described above, the image display method and apparatus according to the present invention have been specifically described based on the embodiments.
The present invention is not limited to the above-described embodiment, and various changes can be made without departing from the gist.

[0049]

As described above, according to the present invention, after the input video is classified for each shot section, the motion information included in each shot section is analyzed, and the inputted video is analyzed based on the motion information. The camera work information and camera shake information of the video that has been analyzed are analyzed, and a representative image of each video section corresponding to the camera work unit and the unit of the presence or absence of camera shake is displayed on the display device. Representative images of each video section are displayed in order from the camera work state. This makes it easy to determine which video section to use and which video section to not use when performing video editing work, and to perform video editing work, making video editing extremely easy. It has the effect of becoming.

[Brief description of the drawings]

FIG. 1 is a flowchart showing an embodiment of a video display method according to the present invention.

FIG. 2 is a block diagram showing an embodiment of an apparatus for realizing the method of the present invention.

FIG. 3 is an explanatory diagram illustrating a principle of extracting camera shake information.

FIG. 4 is a diagram illustrating a selection example of a representative image.

FIG. 5 is a diagram illustrating an example of a display interface of a representative image.

FIG. 6 is a diagram illustrating another example of a representative image display interface.

[Explanation of symbols]

Vi: input video data, 1: video shot section, 2: video motion vector, 3: camera work and camera shake information, 4
... representative image, 11 ... video shot classification means, 12 ... video motion analysis means, 13 ... camera shake information analysis means, 14 ... representative image display means, 401 to 405 ... representative images.

Claims (4)

[Claims] 1. A video display method for analyzing an input video and displaying a representative image on a display device, comprising: a video shot classification step of classifying the input video for each shot section; A motion analysis step of analyzing the motion information that has been input, a camera shake information analysis step of analyzing camera work information and camera shake information of the input video based on the motion information, and a camera work unit and a unit of presence or absence of camera shake. And displaying a representative image of each video section on a display device. 2. The video display method according to claim 1, wherein in the camera shake information analyzing step, a camera motion information is extracted by calculating a temporal change of a motion vector. 3. A video display device for analyzing an input video and displaying a representative image, comprising: a video shot classification means for classifying the input video for each shot section; Motion analysis means for analyzing motion information; camera shake information and camera shake information analysis means for analyzing camerawork information and camera shake information of the input video based on the motion information; and each image corresponding to a camerawork unit and a unit for the presence or absence of camera shake. And a representative image display means for displaying a representative image of a section on a display device. 4. A medium in which a program for executing a video display method for analyzing an input video and displaying a representative image is recorded, the video shot classification processing for classifying the input video for each shot section. A motion analysis means for analyzing motion information included in each shot section; a camera shake information analysis process for analyzing camerawork information and camera shake information of an input video based on the motion information; A representative image display process of displaying a representative image of each video section corresponding to a unit of the presence or absence of camera shake on a display device; and a computer-readable recording program including: Medium.