JP3469122B2 - Video segment classification method and apparatus for editing, and recording medium recording this method - Google Patents

Description

Detailed Description of the Invention

[0001]

TECHNICAL FIELD The present invention analyzes an image and
The present invention proposes an editing video section classification method and apparatus for providing a user with a video suitable for editing and supporting the video editing.

[0002]

2. Description of the Related Art Due to the widespread use of small video cameras and VCRs, the number of people who record and shoot images has increased significantly. However, many people do not edit the image or keep looking at it repeatedly. This is because the editing work is extremely difficult and is not performed easily, and the unedited video is not viewed because it has unnecessary parts.

Therefore, it is necessary to facilitate difficult editing work in order to make more effective use of the video image taken in real time. One of the difficult points of editing is the process of grasping the content of the material video and cutting out the appropriate video section to be used. However, with equipment including conventional non-linear editing equipment, in order to select and cut out the video section to be used for editing, the content of the video is repeatedly viewed, and the start point and the end point of the section called cueing are displayed. I had to make a decision one by one. This became a complicated work and was one of the causes of obstacles to the editing work.

In order to solve these problems, there has been proposed an attempt to classify video sections into appropriate sections in advance and present them to the user as sections of video material. For example, (Yukinobu Ta
niguti, Yoshinobu Tonomur
a, "PanoramaExcerpts: Extra
cting and packing Panoram
as for Video Browsing "1996
ACM Multimedia 97). In this, a shot change corresponding to ON / OFF of the camera at the time of shooting is used as a reference of a section when material is divided, and a representative image that briefly shows the content of the section is configured and presented. Furthermore, if the contents of the section are analyzed, the representative image is selected as the beginning of the section if the contents of the section are still, and if there is camerawork, the camerawork is connected to create a panoramic image. Has the feature of displaying as a representative image.

[0005]

Since the above-mentioned conventional method certainly divides the image into the video sections and directly shows the contents, it seems to have a sufficient function for providing the material video for editing. However, when the editing work is actually performed, there are cases where a portion that is not suitable for editing is included in the same shot. These include camera shake, quick movements, and shots that are very short and long. The value of video itself depends largely on the content itself, and the value fluctuates depending on the purpose of use, but it also depends on the quality at the time of shooting.

Therefore, in terms of easy editing,
It is desirable to extract only the video section that maintains a certain level of quality. For that purpose, the camera work is further estimated in the shot section, it is determined whether or not an image such as a camera shake or a sudden change in content is included, and such a section is automatically removed or specified. I think that a function is necessary. Similarly, for videos with extremely short durations, depending on what is being filmed,
It is often not suitable for editing.
Therefore, it is considered necessary to have a function of removing even such short sections in advance.

That is, in order to edit a video, the video material is analyzed and divided into shot sections corresponding to ON / OFF at the time of shooting, and a video section which deteriorates the quality such as camera shake is automatically selected from among them. There is a need for a technology that removes or clearly shows the remaining video section and presents it so that it can be easily used as an editing material. However, the image may depend on the content regardless of the quality. Therefore, check the degree of the error, and whether it is a camera shake, a sudden movement, or a video section that is too short. It is considered that the most desirable method is to identify and show the appropriate video section, camera shake, section with abrupt changes, and too short section.

Therefore, from the automatically divided shot sections, which is a problem of the above-mentioned prior art, a section including camera work such as low image quality, camera shake, and sudden movement is determined, An object of the present invention is to provide an editing video section classification method and apparatus capable of extracting appropriate video sections and presenting them to a user so that they can be easily selected as an editing material to support video editing. Is.

[0009]

The present invention solves the above problems by the following means.

One of the means is to analyze the input image,
In the editing video section classification method that automatically extracts and presents materials suitable for video editing, the shot change section is automatically detected from the input video, and it is divided into shot sections and classified into shot change section division and classification stages, From the image sequence that makes up the video in the shot section,
Based on the feature amount calculation step of calculating a single or multiple feature amount regarding whether or not is appropriate for editing, and whether the camerawork of the video section in the shot is appropriate for editing from the calculated single or multiple feature amount . And a video material presentation step of presenting only a video section determined to have camerawork suitable for editing in the evaluation in the video evaluation step as a material suitable for editing. A method for classifying video sections for editing, characterized in that

Alternatively, in the feature quantity calculating step, the first feature quantity is calculated by integrating the results of local calculation on the image between two or more consecutive image sequences, and the first feature quantity is calculated on the image. The second feature amount is calculated from the overall calculation of
In the video evaluation step, the video section classification method for editing is characterized in that the video section in the shot is evaluated using the two feature amounts.

Another means is to analyze the input video, automatically extract a material suitable for video editing, and present it in an editing video section classifying device, which automatically performs shot change from the input video. From the shot change section dividing unit that detects, divides and classifies into shot sections, and the image sequence that constitutes the video in the shot section, camera work is performed.
From the feature quantity analysis unit that calculates a single or a plurality of feature quantities related to whether or not is appropriate for editing, and the camera work of the video section in the shot is edited from the plurality of calculated feature quantities.
A video evaluation unit that performs an evaluation regarding suitability, and a material presentation unit that presents only a section determined to have camerawork suitable for editing by the evaluation of the video evaluation unit as a material suitable for editing. An editing video section classifying device characterized by having:

Alternatively, the feature quantity analyzing unit calculates the first feature quantity by integrating the results of local operations on the image between two or more consecutive image sequences, and calculates the first feature quantity on the image. Is to calculate a second feature amount from the overall calculation of, and the video evaluation unit uses the two feature amounts,
An editing video section classification device characterized by evaluating a video section in a shot.

It should be noted that a program for executing the steps in the above-mentioned editing video section classification method on a computer is
Recording on a computer-readable recording medium,
If the recording medium and a computer that can read the recording medium and execute the program exist, the present invention can be implemented.

In the present invention, the evaluation using the feature amount is performed from the automatically divided shot sections, and the section including the camera work such as low image quality, camera shake, and abrupt movement is discriminated. , By extracting an appropriate video section and presenting it to the user, the section can be easily selected and used as an editing material, and video editing can be supported.

Although camera shake and quick camera work are similar errors, the characteristics they show are slightly different. Therefore, in order to distinguish these stably,
The accuracy is improved by using the feature quantity and the evaluation method in which a plurality of algorithms are combined. Therefore, the video evaluation is performed from the feature amount calculated by integrating the results of the local calculation on the image between two or more continuous image sequences and the feature amount calculated from the overall calculation on the image. By doing so, higher accuracy can be expected.

[0017]

BEST MODE FOR CARRYING OUT THE INVENTION A detailed description will be given below of an embodiment of the editing video section classification method and apparatus of the present invention.

FIG. 1 shows a flow chart of an embodiment of an editing video section classification method of the present invention.

The video from the input video 0 is analyzed and divided into video shot sections 1 in the shot change section division / classification step 11 to be classified. From this video shot section 1,
In the feature amount extraction step 12, each image included in the video is analyzed and the feature amount 2 included in the image is calculated. This feature amount may be calculated from information closed in each image, or may be calculated based on correlation information with other images.

On the basis of this characteristic amount 2, the image evaluation step 13 evaluates the content of the image. The evaluation here is whether it is a proper video, that is, a video suitable for editing. The material for determining the evaluation is the feature amount 2. Any meaning may be included in the determination result. In this video evaluation step 13,
The appropriate video section 3 which is a stable video section is calculated.

The appropriate video section 3 is input to the video material presentation step 14, and the appropriate video material 4 is output. Of course, it is up to the user to decide whether or not the user uses this video material for editing.

FIG. 2 shows a block diagram of an embodiment of the editing video segment classification apparatus of the present invention.

The present apparatus is provided with a recording medium 30 for temporarily storing a video, and the shot change section dividing unit 40 extracts the shot section 21 of the video from the input video 20. The feature amount analysis unit 41 extracts the feature amount 22 from the video shot section 21. From the feature amount 22, the video evaluation unit 42 calculates an appropriate video section 23 which is a stable video section in the shot. The appropriate video section 23 is input to the video material presentation unit 43 and presented to the user 50 as video material. Therefore, the video material presentation unit 43 has a representative image creation unit 44 that creates the representative image 24 from the appropriate video section 23, and also has a display interface 45 that displays the representative image 24 so that the user 50 can use it. To

A detailed description will be given below of an example of the operation of the apparatus and an example of an embodiment of the method.

The input image 20 may be any digital image that can be analyzed by a computer, and may be compression-encoded such as MPEG or JPEG. The parameters in the time and space directions may be arbitrary. If the image is not a digital image, the shot change section dividing unit 40 may have a function of digitizing analog image data by a capture tool or the like. For example, when the input video is counted in file units, it is not necessary that the input video is one, and a plurality of input videos may be used. In the case of plural pieces, the shot change section dividing unit 40 has a function of matching them when the temporal and spatial formats (size, frame rate, audio track, etc.) are different.

The shot change section dividing section 40 divides the image into shots. Here, a shot change is detected, and the video is divided into sections according to the shot change points. For detection of shot change, existing, proposed and publicly known methods may be used. As an example, if the difference on the same pixel between frames is calculated and there are many pixels with large differences,
There is a method of assuming that a shot change has occurred. Also, it can be turned on during recording without analyzing the image content.
It is also possible to record / store / OFF, and combine such external information to divide the section.

The feature amount analysis unit 41 calculates the feature amount 22 from the image sequence in the same shot. In the calculation, the feature amount may be detected only from the information inside the image of one frame, or the information with other frames such as the correlation with a plurality of image sequences may be used.

The video evaluation unit 42 calculates the calculated feature quantity 2
Based on 2, it is determined whether the image sequence in the shot section is stable. The feature quantity used may be a single feature quantity,
When a plurality of feature quantities are given, they may be combined to analyze a plurality of feature quantities.

Regarding the characteristic amount analysis unit 41 and the image evaluation unit 42, a description will be added later, including all the appropriate image sections based on the characteristic amount and evaluation.

Regarding the video material presenting section 43, the appropriate video section 23, which is a video material, has, as a section, a section determined to be a video material that can withstand editing within the same shot. Sections that are not suitable for editing shall be removed.

In the video material presentation unit 43, the appropriate video section 2
An image representative of the video section from 3 is represented by the representative image creating unit 44.
Create with. Representative image 24 by the representative image creating unit 44
May be the top one frame of each appropriate video section, a panoramic object composed of a plurality of frames, or an image in which a plurality of frames are reduced and arranged. The representative image 24 thus created is displayed on the display interface 45 to the user 50. Display interface 45
May have an interface that classifies the representative image and the relevant appropriate video section as the editing material and transfers the classified image section to the editing software.

It should be noted that another example of the appropriate video section and the video material presenting section can be considered.

The appropriate video section includes all of the same shots, but information on whether the shot is appropriate or not is recorded in each image sequence, and only the appropriate video section is recorded by the video material presenting section. May be classified and processed, and then processed. Further, in addition to binarizing whether or not it is appropriate, multivalued evaluation may be performed on the basis of the feature amount, the video sections may be ranked, and the video may be displayed according to the ranking.

The video material presenting section 43 may present not only the representative image but also the information such as the scenery, the person, etc. recorded therein. The semantic information may be automatically detected if possible, or may be added from the outside. In the case of adding from the outside, the display interface 45 may have a function of supporting the adding work from the outside.

The operations of the feature quantity analysis unit and the video evaluation unit for the target video shot section will be described in more detail.

As described above, the determination as to whether the image is appropriate or inappropriate depends on the purpose of use. However, it is generally known that camera shake, sudden camera movement, and too short video section are difficult to use for editing. Here, as an example, a description will be given of a feature amount necessary for distinguishing an unsuitable video section, such as a camera shake, a video that moves abruptly, and a video section that is too short, and its evaluation method.

Shake and sharp camera work and normal camera work will be discussed respectively. Each image is shown in FIGS. 3 (a), (b) and (c).

Two types of feature quantities are calculated for the purpose of discriminating these. A similar area between the original image and a continuous image is searched for, and the movement of the area is calculated. The error is calculated in consideration of the movement amount, and the size of the error is calculated as the feature amount. One of the two types of features to be calculated is
FIGS. 5 and 6 show the calculation methods of these feature amounts, which are the total change amount based on the similarity of the entire image and the other is the local set change amount based on the local similarity. A method called block matching is used as a method common to both. First, this method will be described with reference to FIG.

As shown in FIG. 4, the image f to be searched is first
Divide (t) into several regions. The divided areas may be square or other areas, but squares are generally used to simplify the processing. Let each of the divided areas be B (k, t). Note that B (k, t)
Contains several pixels. Let those pixels be p (k,
i, t). What is block matching? B
Separate target images are stitched for each (k, t). Here, the comparison is performed on the image of the next continuous frame f (t + 1).

Block p (k, t) obtained by dividing f (t)
Target. Area B in frame f (t + 1)
The area corresponding to (k, t) is written as B (k, t + 1). A search is made by comparing with a region B (k + m, t + 1) obtained by moving the position of the region by m around this region. In a similar search, p (k, t) included in B (k, t)
i, t) and p (k + contained in B (k + m, t + 1)
m, i, t + 1) will be searched for such that the sum of the differences between them becomes the smallest. The formula is: min {Σ _m {p (k, i, t) −p (k + m, i, t
+1)} ² }.

In this embodiment, a search is performed using this block matching method.

First, the search for the overall similar region will be described with reference to FIG. As shown in FIG. 5A, the images obtained by calculating the average of the original images and reducing the images are regarded as one block, and a block matching method is applied to search for a similar region (Level).

Next, in a state where the image is not reduced so much, the entire image is regarded as one block, and the search is similarly performed (Level 2). Note that only the periphery of the candidate is searched with reference to the motion vector obtained from the one smaller reduced image. In this way, this process is repeated up to the size of the original original image (Level 3) to calculate the moving distance of the entire image. This method is also called coast to fine. The image similarity is calculated by using the difference between the images finally determined to be similar (FIG. 5B). As a mathematical expression, D = Σ _i {p (i, t) -P (i + m, t + 1)} ² As a result, when the entire image moves in the direction accurately searched, the difference becomes small.

Next, the local similarity will be described with reference to FIG.

Here, the original image is decomposed into blocks and exactly block matching is performed. The original image is divided into appropriate blocks and the search is performed. At the time of search, it is said that about 8 × 8 is preferable when the original image has a size of 320 × 240. A similar region is searched for each vector, and the error is calculated for each block. The sum of the errors of each block is used as a feature amount. Therefore,
The mathematical formula is D = Σ _k {B (k, t) −B (k + m, t + 1)} ² .

At this time, when the camera shake to be detected and the abrupt movement occur, the camera shake locally has many similar areas as shown in the camera shake state of FIG. 3A. Therefore, the local set change amount does not become so large. On the other hand, if the total change amount is applied as a change in the whole, distortion occurs, and therefore the total change amount becomes large.

On the other hand, abrupt movements often go out of the block matching search range, so that the error in both the total change amount and the local set change amount becomes large.

FIG. 7 shows these two variations and the effect on the image. Two consecutive images are compared with each other, and a portion where a large difference remains is displayed in black. In the case of a change due to camera shake, the local set change amount is smaller than the total change amount because the strain is absorbed in each region. Further, in the case of abrupt camera work, most of the errors are out of the search range, resulting in large errors.

By applying this fact, it is possible to discriminate between camera shake and sudden movement. That is, if any of the errors is large, the motion is abrupt, and if the local set change amount is small, it is determined that the motion is a camera shake.

This flow configuration is summarized in FIG. That is, on the basis of each change amount, the camera shake, the abrupt camerawork, and the other stable camerawork are calculated by comparing with a given threshold value. If the same shot is classified based on each of these sections, as shown in FIG. 9, it is possible to divide the same shot into a plurality of sections on the same shot section. In such divided sections, a section that is shorter than a predetermined video section length is automatically determined as a section that is too short.

In the video material presentation section 43, as shown in FIG. 9, a representative image is selected from each video section. of course,
The video section that is too short may be removed in advance, or the video section may be displayed together with the shot section, the abrupt movement section, and the camera shake section in a state that the section is too short. It should be noted that various applications such as reproduction of an arbitrary part in the representative image interface can be considered, and easily analogized ones are within the scope of the present invention.

It should be noted that a part or all of each part of the apparatus shown in FIG. 2 can be made to function by using a computer, or the processing steps shown in the operation examples of FIG. 1, FIG. 8 or FIG. Needless to say, can be executed by a computer, a program for causing a computer to function as each part thereof, or a program for causing a computer to execute the processing steps thereof, a recording medium readable by the computer, for example, FD (floppy disk), MO, R
It can be recorded and provided on an OM, a memory card, a CD, a DVD, a removable disk, etc., and distributed. If there is a recording medium recording the computer program, a storage medium for storing images, etc., and a computer for reading and executing the program, it is possible to function.

[0053]

According to the present invention, a video section determined to be suitable for editing work is extracted so as to facilitate video editing,
Since it is presented to the user, it is useful for video editing support.

[Brief description of drawings]

FIG. 1 is a flowchart showing an example of an embodiment of an editing video section classification method of the present invention.

FIG. 2 is a block diagram showing an exemplary embodiment of an editing video section classification device of the present invention.

3 (a), (b), and (c) are diagrams showing examples of images with camera shake and abrupt camerawork.

FIG. 4 is an explanatory diagram of block matching.

5 (a) and 5 (b) are diagrams for explaining calculation of the degree of similarity of the entire screen.

FIG. 6 is a diagram illustrating calculation of local similarity of a screen.

FIG. 7 is a diagram for explaining the difference in the amount of change in the case of camera shake and sudden camerawork.

FIG. 8 is a diagram showing a detection flow of camera shake and abrupt camerawork in the present embodiment example.

FIG. 9 is a diagram illustrating division of a video section according to the present embodiment example.

[Explanation of symbols]

0 ... Input video data 1 ... Video shot section 2 ... Feature quantity 3 ... Appropriate video section 4 ... Video material 11 ... Shot change section division classification step 12 ... Feature amount extraction step 13 ... Video evaluation step 14 ... Video material presentation step 20 ... Input video data 21 ... Video shot section 22 ... Feature amount 23 ... Appropriate video section 24 ... Representative image 30 ... Recording media 40 ... Shot change section division section 41 ... Feature analysis unit 42 ... Video evaluation classification 43 ... Video material presentation section 44 ... Representative image creation unit 45 ... Display interface 50 ... User

─────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-6-209451 (JP, A) JP-A-11-136573 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) H04N 5/76-5/956 H04N 5/262-5/278

Claims (5)

(57) [Claims] 1. A method of classifying a video section for editing in which an input video is analyzed, a material suitable for video editing is automatically extracted, and a shot change is automatically detected from the input video,
A feature amount for calculating a single or a plurality of feature amounts regarding whether or not camera work is suitable for editing from a shot change segment division and classification step of dividing and classifying into shot segments, and an image sequence forming a video in the shot segment. A calculation step, a video evaluation step that evaluates whether or not the camerawork in the video section in the shot is appropriate for editing from the calculated single or multiple feature amounts, and an edit step in the evaluation step of the video evaluation step. And a video material presentation step of presenting only a video section determined to have appropriate camera work as a material suitable for editing, a video section classification method for editing. 2. In the feature amount calculating step, the first feature amount is calculated by integrating the results of local calculation on the image between two or more consecutive image sequences, and the first feature amount is calculated on the image. 2. The second feature amount is calculated from the overall calculation of, and in the video evaluation step, the video section in the shot is evaluated using the two feature amounts. Video segment classification method. 3. An editing video section classifying device which analyzes an input video, automatically extracts a material suitable for video editing, and presents it, and automatically detects a shot change from the input video,
A shot amount section dividing unit for dividing and classifying into shot sections, and a feature amount analysis for calculating a single or a plurality of feature amounts regarding whether or not camerawork is appropriate for editing, from an image sequence forming a video in the shot section Section, a video evaluation section that evaluates whether or not the camerawork in the video section in the shot is appropriate for editing from the calculated plurality of feature amounts, and a camera that is suitable for editing by the evaluation of the video evaluation section. An editing video section classification device characterized in that it has a material presentation section that presents only sections determined to have work as materials suitable for editing. 4. The feature quantity analyzing unit calculates the first feature quantity by integrating the results of local calculation on the image between two or more consecutive image sequences, and calculates the first feature quantity on the image. The second feature amount is calculated from the overall calculation of the image evaluation unit, and the video evaluation unit evaluates the video section in the shot using the two feature amounts. Item 3. The video segment classification device for editing according to Item 3. 5. An editing image, characterized in that a program for executing the steps in the editing image segment classification method according to claim 1 or 2 by a computer is recorded in a computer-readable recording medium. Recording medium recording the section classification method.