JP2000076463A - Analyzing method of video stream structure, structure analyzing process displaying method, and structure displaying method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make an analyzing process verifiable by displaying the analyzing process of video information. SOLUTION: A structure analyzing process display window 301 is provided with a comparing frame displaying part 302 displaying two frames which are successively selected from a video stream and compared, a graph displaying pat 304 successively plotting the difference between featured values obtained by comparing the two frames and displaying the variation of the featured value of each frame in the video stream by a broken line graph, and a bar 305 which shows the progressing situation of analyzing process corresponding to the position of each frame constituting the video stream and shows the position of the dividing point of the video stream to display the video steam structure analyzing process through the use of these.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for analyzing the structure of a video stream, a method for displaying a structure analysis process, and a method for displaying a structure.・ A structure analysis method for a video stream, a structure analysis process display method, and a structure display that enable accurate execution and further enable visual representation of the internal logical structure of the video information using a result of the analysis of the video information. About the method.

[0002]

2. Description of the Related Art In recent years, with the development of computer-related technology and information processing technology, and the spread of electronic networks such as the Internet, intranets, and extranets, all kinds of personal information used in daily life, business, and services have been developed. Have been digitized and transmitted as electronic information, or accumulated in databases in the computer world and used and utilized.

Therefore, electronic information is used and utilized in various forms (for example, information processing, information retrieval, information transmission, etc.) irrespective of whether the information is analog data or digital data. Various techniques for handling the electronic information itself in a format and data structure that are easy to use and utilize have been provided.

[0004] For example, text information, which is one type of electronic information, is represented by SGML,
A structure tag that expresses a logical structure such as HTML or XML is defined, and when information is transmitted, it can be expressed by the content and its logical structure. Therefore, when using and utilizing the text information, for example, by using the internal logical structure, it is possible to efficiently search, generate an appropriate summary sentence, increase the processing speed, and the like.

[0005] As for video information, various techniques have been proposed for analyzing and structuring video information and managing the structured video information.

[0006]

However, in the conventional video information analysis processing, while reproducing the video information,
Since the process where the operator finds the division point of the video information is the mainstream, the operator has to take a very long time to analyze the video information, and if he is not always focused on the video during the video playback Because of this, there was a problem that the burden on the body and mind was great.

In order to reduce the burden on the operator as described above, devices that automatically determine the division point of video information are individually provided. Since only the physical division point is identified and the corresponding thumbnail image is displayed, a logical division point for dividing a set of meaningful videos that is actually required,
There is a problem that it is difficult to simply distinguish the image from a physical division point where the display state of the screen changes greatly.

[0008] Further, in the conventional case of displaying the result of analyzing and structuring video information, generally, a thumbnail image representing a set of videos in a structured unit is simply displayed side by side. In order to grasp the internal logical structure of the video information, it is necessary to imagine the logical structure from the displayed thumbnail image, and there is an inconvenience that the internal logical structure of the video information cannot be intuitively grasped. . In other words, there is a problem in that a user interface method that can easily grasp the logical structure inside the video information has not been provided.

SUMMARY OF THE INVENTION The present invention has been made in view of the above, and has as its object to provide a method of analyzing the structure of a video stream, which reduces the burden on an operator when analyzing video information.

Further, the present invention has been made in view of the above, and displays an analysis process of video information so that the analysis process of the video information can be visually confirmed, thereby facilitating the analysis process. -It is an object of the present invention to provide a method for displaying a structure analysis process of a video stream, which can be performed easily and accurately.

Further, the present invention has been made in view of the above, and by enabling the process of analyzing video information to be reproduced so that the analysis process can be verified, the analysis process of video information can be smoothly performed. -It is an object of the present invention to provide a method for displaying a structure analysis process of a video stream, which can be performed easily and accurately.

Further, the present invention has been made in view of the above, and is capable of expressing a logical structure inside video information using a result of analyzing the video information, and visualizing the logical structure inside the video information. An object of the present invention is to provide a method of displaying a structure of a video stream that can be easily grasped.

[0013]

According to a first aspect of the present invention, there is provided a method for analyzing the structure of a video stream, comprising the steps of: setting an arbitrary frame in the video stream as a reference frame; In the time series, it is determined whether or not a frame exists at a position advanced by the first interval from the set reference frame, and if a frame exists, the corresponding frame is selected and the first selected frame is selected. And the second step of setting the last frame as the first selected frame when no frame exists, comparing the first selected frame and the reference frame, and comparing the first selected frame and the reference frame If the difference between the feature values of the frame image is equal to or greater than a predetermined threshold,
A third step of storing the position of the first selected frame and, if the position is not greater than or equal to a threshold value, setting the first selected frame as the reference frame and returning to the second step; When the position of the first selected frame is stored in the first step, the frames after the frame set as the first selected frame in the second step before the stored first selected frame are stored in the first frame. A fourth step of sequentially selecting the selected frame at a second interval smaller than the interval, and setting the selected frame as a second selected frame; comparing the second selected frame with a reference frame; A fifth step of storing the position of the second selected frame when the difference between the feature amounts of the images of the reference frame is equal to or greater than a predetermined threshold, and returning to the fourth step when the difference is not equal to or greater than the threshold; Previous A sixth step of setting the second selected frame as the reference frame when the position of the second selected frame is stored in the fifth step, and newly setting a reference frame in the sixth step In the video stream, it is determined whether or not a frame exists at a position advanced by the first interval from the set reference frame in the video stream, and if a frame exists, the process proceeds to the second step. Returning, if there is no frame, the seventh step is to return to the second step with the last frame as the first selected frame, and the process is terminated if the reference frame is the last frame. Extracting a set of consecutive frames having a small difference in feature amount from the video stream based on the position of the second selected frame stored in the fifth step. It is one which allows dividing the video stream at any consecutive frames.

According to a second aspect of the present invention, there is provided a video stream structure analysis method comprising: a first step of setting an arbitrary frame in a video stream as a reference frame; and, in a time series of the video stream, a frame subsequent to the reference frame. Is sequentially selected at a first interval, and the second step of setting the selected frame as the first selected frame is compared with the first selected frame and the reference frame, and the first selected frame and the reference frame are compared. A third step of storing the position of the first selected frame when the difference between the feature amounts of the images is equal to or greater than a predetermined threshold, and returning to the second step when the difference is not equal to or greater than the threshold; When the position of the first selected frame is stored in the step (b), the position is set as the first selected frame in the second step before the stored first selected frame. A fourth step of sequentially selecting frames after the selected frame at a second interval finer than the first interval, and setting the selected frame as a second selected frame; and If the difference between the feature amounts of the images of the second selected frame and the reference frame is equal to or greater than a predetermined threshold, the position of the second selected frame is stored.
A fifth step of returning to the fourth step, and a sixth step of setting the second selected frame as the reference frame when the position of the second selected frame is stored in the fifth step When a new reference frame is set in the sixth step, it is determined whether or not a frame exists at a position advanced by the first interval from the set reference frame in the video stream. Returning to the second step when a frame exists, and returning to the second step with the last frame as the first selected frame when no frame exists; An end of the process when is the last frame, and the second step stored in the fifth step.
A set of continuous frames having a small difference in feature value is extracted from the video stream based on the position of the selected frame, and the video stream can be divided into arbitrary continuous frame units.

A third aspect of the present invention is a video stream structure analysis process display method for displaying an analysis process for analyzing the structure of a video stream, wherein the video stream structure analysis process display method comprises the steps of: A first step of setting an arbitrary frame as a reference frame and displaying an image of the set reference frame; and, in a time series of the video stream, a first step from the set reference frame to the first step.
It is determined whether or not there is a frame at a position advanced by the interval of. If the frame exists, the corresponding frame is selected and set as a first selected frame, and the set first frame is set.
Is displayed together with the image of the reference frame, and when there is no frame, the last frame is set as the first frame, and the image of the set first selected frame is displayed together with the image of the reference frame. Comparing the second step of displaying with the first selected frame and the reference frame, and when the difference between the feature amounts of the images of the first selected frame and the reference frame is equal to or greater than a predetermined threshold value, The position of the selected frame is stored. If the position is not equal to or larger than the threshold value, the first selected frame is set as the reference frame, and the image of the set reference frame is displayed instead of the image of the currently displayed reference frame. , The second
A third step returning to the step, and the first step in the third step.
When the position of the selected frame is stored, the frames after the frame set as the first selected frame in the second step before the stored first selected frame are finer than the first interval. Select sequentially at a second interval,
Set the selected frame as the second selected frame,
A fourth step of displaying the set image of the second selected frame in place of the currently displayed image of the first selected frame, and comparing the second selected frame and the reference frame; A fifth step of storing the position of the second selected frame when the difference between the feature amounts of the images of the selected frame and the reference frame is equal to or greater than a predetermined threshold, and returning to the fourth step when the difference is not equal to or greater than the threshold And when the position of the second selected frame is stored in the fifth step, the second selected frame is set as the reference frame, and the image of the set reference frame is currently displayed. A sixth step of displaying in place of the image of the first step, and when a new reference frame is set in the sixth step, the video image stream is deciphered from the set reference frame by the first interval. It is determined whether or not there is a frame at the position where the frame is located. If a frame is present, the process returns to the second step. If no frame is present, the last frame is set as the first selected frame in the second step. 7th return to the process
And an eighth step of terminating the process when the reference frame is the last frame, and further comprising the reference frame obtained in the third step and the fifth step, and The method includes a ninth step of sequentially plotting a difference between the feature amounts of the first selected frame and the second selected frame, and displaying a change in the feature amount of each frame in the video stream in a graph.

A fourth aspect of the present invention is a video stream structure analysis process display method for displaying a video stream structure analysis process for analyzing the structure of a video stream. A first step of setting an arbitrary frame as a reference frame and displaying an image of the set reference frame, and sequentially selecting frames after the reference frame at a first interval in a time series of the video stream, selecting The second step of setting the set frame as a first selected frame, displaying the image of the set first selected frame together with the image of the reference frame, and comparing the first selected frame and the reference frame, When the difference between the feature amounts of the images of the first selected frame and the reference frame is equal to or greater than a predetermined threshold, the first
A third step of returning to the second step when the position of the selected frame is not greater than or equal to the threshold, and storing the position of the first selected frame in the third step.
The frames after the frame set as the first selected frame in the second step before the stored first selected frame are sequentially selected at a second interval finer than the first interval and selected. A fourth step of setting a frame as a second selected frame, and displaying the set image of the second selected frame instead of the currently displayed image of the first selected frame; and And a reference frame, and when the difference between the feature amounts of the images of the second selected frame and the reference frame is equal to or greater than a predetermined threshold,
A fifth step of storing the position of the second selected frame and returning to the fourth step when the position of the second selected frame is not equal to or greater than a threshold value; and storing the position of the second selected frame in the fifth step. A sixth step of setting the second selected frame as the reference frame, and displaying the image of the set reference frame instead of the image of the currently displayed reference frame; and When a reference frame is set, it is determined whether or not a frame exists at a position advanced by the first interval from the set reference frame in the video stream.
Returning to the second step, if no frame exists,
A seventh step of returning to the second step with the last frame as the first selected frame, and an eighth step of ending the processing when the reference frame is the last frame, Furthermore, the difference between the feature amounts of the reference frame obtained in the third step and the fifth step and the first selected frame and the second selected frame is sequentially plotted, and the difference of each frame in the video stream is plotted. This includes a ninth step of displaying a change in the feature amount in a graph.

According to a fifth aspect of the present invention, there is provided the video stream structure analysis process display method according to the third or fourth aspect, wherein the ninth step is performed with respect to the reference frame. The position of a second selected frame having a feature value equal to or larger than a threshold value is displayed in the graph.

According to a sixth aspect of the present invention, there is provided the video stream structure analysis process display method according to any one of the third to fifth aspects, further comprising the video stream. A tenth step of displaying a graph indicating the progress of the analysis process in association with the position of each frame, wherein the tenth step further selects a first selected frame or a second selected frame. The position of the selected frame and the position of a second selected frame having a feature value equal to or greater than a threshold value with respect to the reference frame are displayed in the graph.

A method for displaying a structure analysis process of a video stream according to claim 7 is the method for displaying a structure analysis process of a video stream according to any one of claims 3 to 6, further comprising the steps of: The method includes an eleventh step of recording the process of the process executed in the tenth process, and a twelfth process of reproducing the process of the process recorded in the eleventh process at an arbitrary timing.

According to a still further aspect of the present invention, there is provided a video stream structure display method, wherein video index information for managing the video information is generated separately from the video information, and the video of the video information is generated based on the generated video index information. A method for displaying a structure of a video stream for displaying a structure of a stream, wherein the video index information includes a set of consecutive frames having a small difference in a feature amount extracted from the video stream as one unit of meaningful frames. It has at least one piece of segment information managed as a set and package information for managing a plurality of the pieces of segment information as at least a structural element object. The segment information managed by the package information is arranged at a lower level Te, and displays the structure element objects in a tree structure.

According to a ninth aspect of the present invention, in the video stream structure display method according to the eighth aspect, the segment information further includes a set of frames managed by the segment information. It is possible to manage package information for collectively managing a plurality of other generated segment information, arrange and display the package information below one segment information, and The segment information managed by the package information is arranged and displayed in a tree structure.

According to a tenth aspect of the present invention, there is provided the video stream structure display method according to the eighth or ninth aspect, further comprising arbitrarily selecting from a set of frames managed by the segment information. At least one frame is displayed as a representative frame as a thumbnail.

[0023] A video stream structure display method according to claim 11 is the video stream structure display method according to any one of claims 8 to 10, further comprising a video stream created in advance for each video information. When a search process is performed for video information that matches a desired search condition from a plurality of video information using the index information, the search is performed based on the video index information of the video information that matches the search condition. The structure of the video stream of the video information that matches the condition is displayed in the tree structure.

According to a twelfth aspect of the present invention, in the method of displaying a video stream structure according to the eleventh aspect, the structure of the video stream of the video information matching the search condition is displayed in a tree structure. At that time,
The structural element object that matches the search condition is displayed separately from other constituent element objects.

[0025]

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, one embodiment of a method for analyzing the structure of a video stream, a method for displaying a structure analysis process, and a method for displaying a structure according to the present invention will be described. 3) Video index information (4) Structure display of video stream (5) Display of search result of video information (6) Application example The order will be described in detail with reference to the accompanying drawings.

(1) System Configuration FIG. 1 is a schematic configuration diagram showing an example of a system for realizing a video stream structure analysis method, a structure analysis process display method, and a structure display method according to the present embodiment. The system shown in FIG. 1 includes a video information library 101 storing a plurality of video information, and a plurality of video index information (“(3) video index information” generated corresponding to each video information in the video information library 101). Video index information database 1 that stores
02, and a computer 103 that analyzes the structure of the video stream of the video information in the video information library 101 and visually displays the structure of the video stream based on the analysis result. .

The video information library 101 shown in FIG.
The video index information database 102 may be built in the computer 103, but is assumed here to be connected to the computer 103 via a network as an example.

FIG. 2 shows the computer 103 shown in FIG.
FIG. 3 is a block diagram of the configuration of FIG. The computer 103 includes a CPU 201 that executes processing based on software described later.
ROM 202 storing a boot program and the like;
RAM 20 used as work area of PU 201
3 and a keyboard 204 for performing various input operations
And a mouse 205, a display device 206 such as a CRT or a liquid crystal display for displaying a video stream analysis process and a video stream structure, and a video stream structure analysis method and a structure analysis process display method according to the present embodiment. Structural analysis software 20 for realizing the structure display method
7. Index generation software 208, structure display software 2
A network adapter 211 for connecting the computer 103 to the hard disk 210 storing various application programs and data such as the image data 09, the video information library 101 and the video index information database 102 shown in FIG.
And at least.

The system configuration shown in FIGS. 1 and 2 is merely an example, and if a method according to the present embodiment described later can be realized,
Any system configuration may be used.

(2) Structure Analysis of Video Stream Next, a structure analysis process of a video stream realized in the above-described system will be described. When executing the structure analysis processing of the video stream, the structure analysis software 207 shown in FIG. 2 is used.

The structure analysis software 207 compares the frames in the video stream to be subjected to the structure analysis and obtains the difference between the feature amounts of both images, thereby obtaining the feature amount of each frame image in the video stream. Is performed to determine the change in In other words, the structural analysis software 207
Calculates a change in the feature value of the image of each frame, specifies a division point at which the video stream is divided into a plurality of portions, extracts a set of consecutive frames having a small difference in the feature value from the video stream, The stream can be divided into arbitrary continuous frame units. Note that the feature amount here is the color of the frame image
There are color scheme, structure, shape, curvature, etc.

When the structural analysis software 207 is activated, FIG.
A structure analysis process display window 301 as shown in FIG. The structure analysis process display window 301 displays to the user the process in which the structure analysis software 207 performs the structure analysis of the video stream. Here, before describing the structure analysis processing of the video stream, the structure analysis process display window 30
1 will be described.

The structure analysis process display window 301 has a comparison frame display section 302 for displaying two frames to be compared in the video stream. In the present embodiment, the frame displayed on the left side of the comparison frame display section 302 is referred to as a reference frame, and the frame displayed on the right side is referred to as a selected frame (a first selected frame or a second selected frame). In the comparison frame display section 302, an area for displaying a reference frame is referred to as a reference frame display section 302a, and an area for displaying a selected frame is referred to as a selected frame display section 302b.

The structure analysis software 207 executes the structure analysis processing of the video stream while switching between two modes, a normal comparison mode and a detailed comparison mode. In the normal comparison mode, the structural analysis software 207 selects a selected frame to be compared with the reference frame at a preset first interval, and displays the selected frame on the selected frame display unit 302b. In the present embodiment, the first interval is set to 30
As frames, the structure analysis software 207 executes a process of selecting a frame to be a selected frame at intervals of 30 frames and comparing the frame with a reference frame. By executing the structure analysis processing in the normal comparison mode as described above, it is possible to roughly grasp the change in the feature amount of each frame in the video stream, and it is possible to compare each frame with the reference frame. It is possible to speed up the structural analysis processing. Note that, in this normal comparison mode, the frame selected as the above-described selected frame is referred to as a first selected frame.

On the other hand, in the detailed comparison mode, when the first selected frame in which the feature amount is greatly changed so as to be a division point of the video stream during execution of the normal comparison mode, the first selected frame is selected. The frames subsequent to the frame previously selected as the first selected frame
Are sequentially selected at a second interval smaller than the interval of, and a comparison process with the reference frame is executed. In the present embodiment, the second interval is one frame. By executing such a structure analysis process in the detailed comparison mode, it is possible to reliably specify a frame to be a division point of the video stream. Note that the frame selected as the selected frame in the detailed comparison mode will be referred to as a second selected frame with respect to the first selected frame in the normal comparison mode.

Here, in the normal comparison mode, frames are compared at intervals of 30 frames, and in the detailed comparison mode, frames are compared at intervals of one frame. In addition, for example, a process of comparing frames at 5-frame intervals to narrow down a position of a frame where a feature amount greatly changes so as to be a division point of a video stream is performed,
Finally, a process may be performed in which frames are compared at one-frame intervals to specify a position of a frame where a feature amount greatly changes so as to be a division point of the video stream.

The structure analysis process display window 301 displays which of the normal comparison mode and the detailed comparison mode the structure analysis process is being executed during the structure analysis process below the selected frame display section 302b.

The structure analysis process display window 301
Has three frame display units 303a to 303c on the left side of the comparison frame display unit 302. The frame display unit 303 sequentially displays the frames for which the comparison processing has been completed. Specifically, when the comparison process is completed, the reference frame displayed on the reference frame display unit 302a is sent to the frame display unit 303a, and the selected frame displayed on the selected frame display unit 302b is displayed on the reference frame display unit 302a. And a new selected frame is displayed on the selected frame display section 302b. That is, each time the comparison process is completed, the comparison frame display unit 302 and the frame display units 303a to 303a
The frames displayed in 3c are respectively sent to the left.

The structure analysis process display window 301 plots the difference between the feature amounts of the two frames obtained by comparing the two frames displayed on the comparison frame display unit 302, and displays each frame in the video stream. Has a graph display unit 304 for displaying a line graph representing a change in the feature amount of the above. In the graph display unit 304, a dividing line 304a is displayed.
Area A, area B, area C, and area D surrounded by 4a respectively correspond to the frames displayed above them. For example, a difference between the feature amounts of the reference frame and the selected frame is displayed in an area D below the reference frame display unit 302a.

Areas such as area A, area B, and area D of the graph display unit 304 in which the line graph is indicated by a straight line are compared in the normal comparison mode in which frames are selected at the above-described first intervals. Has been executed. For example, the region A is the frame display unit 303c
3 shows the difference between the feature amounts of the images of the two frames when the frame displayed in the frame display unit 303b and the frame displayed in the frame display unit 303b are compared in the normal comparison mode.

On the other hand, an area C in which a plurality of plot points are displayed in one area corresponds to the corresponding frame display section 30.
This means that the frame displayed in 3a and a plurality of frames existing before the frame displayed in the reference frame display unit 302a have been compared in the detailed comparison mode. That is, the plot points in the region C are selected at the second interval and compared with the frame displayed on the frame display unit 303a (displayed on the reference frame display unit 302a during the comparison process). It corresponds to the number of frames.

Further, as is apparent from the sharp change of the line graph in the area C, when the difference between the feature values of the compared two frames is equal to or larger than a predetermined threshold, the structural analysis software 207 executes the processing. The interval between the compared frames is determined as a division point of the video stream, and a division line 304 b indicating the determined division point is displayed on the graph display unit 304.

It should be noted that the graph displayed in the graph display section 304 is similarly sent to the left and displayed, as the corresponding frame is sent to the left every time the comparison process is performed.

The “color”, “color scheme”, “structure”, “shape”, and “curvature” shown below the graph display unit 304 indicate the feature values used when comparing frames. By checking the corresponding check box, a line graph indicating the difference between the feature amounts is displayed on the graph display unit 304. However, since the values of the respective feature values are different, all the differences between the feature values are not displayed on the graph display unit 304 shown in FIG. 3, but the magnification is changed by selecting the button in the magnification setting unit 306. By doing so, the range of the line graph displayed on the graph display unit 304 can be changed, so that all the feature values can be displayed, the change in the feature value between each frame can be displayed in detail, or displayed roughly. It is possible to do.

When a check box corresponding to “score” is checked, “color”, “color scheme”, “structure”,
A line graph comprehensively showing the difference between all the feature amounts of “shape” and “curvature” is displayed on the graph display unit 304. In the present embodiment, the term “feature amount” simply means the amount indicated by the “score”.

Further, the structural analysis process display window 30
1 has a bar 305 that displays the progress of the processing from the first frame to the last frame in the video stream and the processing result. The bar 305 sequentially displays the position of the frame currently displayed on the comparison frame display unit 302 and subjected to the comparison processing with respect to the entire video stream. As a result, the bar 305 displays all the positions of the frames used for the comparison processing.
In the bar 305, a plurality of wavy lines indicated by 305a indicate frame position display lines indicating the positions of frames used for the comparison processing. When the bar 305 is filled with the frame position display line 305a, the analysis processing of one video stream ends.

Reference numeral 305b denotes a divided frame position display line.
Corresponds to the dividing line 304b of the graph display unit 304, and indicates the position of the frame immediately after the dividing point. This divided frame position display line 305b is indicated by a bar 3
By displaying during the video stream 05, it is possible to recognize at a glance where the division point is located in the video stream.

In FIG. 3, reference numeral 307 denotes a set value display unit for displaying various set values; and 308, a reproduction speed adjustment unit for setting a reproduction speed when reproducing an analysis process described later. 309, a video display unit for displaying video of video information to be analyzed, 310, a start button for instructing start of analysis processing, 311, a restart button for instructing restart of interrupted analysis processing. 312, the analysis processing is completed and the structural analysis process display window 301 is displayed.
Respectively indicate a close button for closing the.

Next, the structure analysis processing of the video stream will be specifically described. FIG. 4 is a flowchart showing the process of analyzing the structure of a video stream. For example, when the video information for which the structure of the video stream is to be analyzed is selected from the video information library 101 and the start button 310 in the structure analysis process display window 301 is clicked,
The structure analysis software 207 starts the processing shown in FIG. Here, “flower report” is an example of video information, and the first frame (ID:
The processing in the normal comparison mode is started from 1).

First, the structural analysis software 207 determines that the ID is 1
Is set as a reference frame, and as shown in FIG. 5, the image is displayed on the reference frame display section 302a, and a frame position display line 305a indicating the corresponding position in the video stream is displayed on a bar 305 ( S401).

After displaying the reference frame, the structure analysis software 207 determines whether or not this reference frame is the last frame in the video stream (S401a).
If it is determined that the reference frame is the last frame in the video stream, the structure analysis software 207 ends the structure analysis processing shown in FIG.

On the other hand, when it is determined that the reference frame is not the last frame in the video stream, it is determined whether or not a frame exists at a position advanced by the first interval (30 frames) from the set reference frame. (S402).
If no frame exists at a position advanced by the first interval (30 frames), the structure analysis software 207 sets the last frame in the video stream as the first selected frame to be compared with the reference frame (S415). The process proceeds to step S404.

When a frame exists at a position advanced by the first interval, the structure analysis software 207 sets the corresponding frame as the first selected frame to be compared with the reference frame, and as shown in FIG. Then, the image is displayed on the selected frame display unit 302b, and a frame position display line 305a indicating the corresponding position in the video stream is displayed on the bar 305 (S403).

Subsequently, the structural analysis software 207 compares the image of the reference frame and the image of the first selected frame shown in FIG. 5 (S404), and finds that the difference between the feature amounts of the image of the reference frame and the image of the first selected frame is small. It is determined whether it is equal to or more than a predetermined threshold (S405).

If it is determined in step S405 that the difference between the feature values of the image of the reference frame and the image of the first selected frame is not greater than or equal to the predetermined threshold, the structure analysis software 207
As shown in FIG. 5, in the area of the graph display section 304 corresponding to the reference frame, the difference between the feature amounts obtained in comparison with the first selected frame is recorded as a line graph (S40).
6). However, since the processing mode is the normal processing mode, the recorded graph is a straight line.

Thereafter, the structure analysis software 207 returns to step S401, displays the image of the reference frame displayed on the reference frame display section 302a on the frame display section 303a on the left, and then displays the image on the selected frame display section 302b. The first selected frame is set as a reference frame, and as shown in FIG.
a is displayed (S401). The structure analysis software 20
Reference numeral 7 shifts the corresponding area of the graph display unit 304 to the left along with the process of displaying the image of the reference frame displayed on the reference frame display unit 302a on the frame display unit 303a.

Then, the structure analysis software 207 determines whether or not the set reference frame is the last frame in the video stream (S401a). If it is determined that the reference frame is the last frame in the video stream, the structure analysis software 207 ends the structure analysis processing shown in FIG. It is determined whether or not there is a frame at a position advanced by the first interval (30 frames) from the reference frame (S402). If no frame exists at a position advanced by the first interval (30 frames), the structure analysis software 207 sets the last frame in the video stream as the first selected frame to be compared with the reference frame (S415). The process proceeds to step S404.

On the other hand, if a frame exists at a position advanced by the first interval, the structural analysis software 207 sets the corresponding frame as the first selected frame to be compared with the reference frame, and as shown in FIG. Then, the image is displayed on the selected frame display unit 302b, and a frame position display line 305a indicating the corresponding position in the video stream is displayed on the bar 305 (S403).

If it is determined in step S405 that the difference between the feature amounts is not equal to or larger than the threshold value, the process returns to step S401 to perform the processing using the first selected frame as the reference frame. It is not always necessary to change the reference frame unless it is not less than the threshold. That is,
The fact that the difference between the feature amounts of the image of the reference frame and the image of the first selected frame is not greater than or equal to the threshold value means that the images of the two compared frames are almost equal, and the same applies regardless of which one is used when comparing the next frame This is because such a comparison process can be performed.

However, different structural analyzes can be performed depending on whether the reference frame is changed or not. When the reference frame is changed, the comparison process is performed between substantially adjacent frames. Therefore, there is a feature that the division point is set only when the difference in the feature amount between the frames is significantly different. On the other hand, if the reference frame is not changed in the above-described case, the difference in the feature amount between the frames at distant positions will be seen, so the difference in the features between the frames is accumulated, and compared with the case where the reference frame is changed. There is a feature that the number of division points is likely to increase.

Returning to the description of the structure analysis processing, the structure analysis processing of the video stream is continued until the state shown in FIG. 7 without determining that the difference between the feature amounts of the images of the reference frame and the first selected frame is equal to or larger than the threshold value. It shall be advanced. FIG.
Shows a frame with an ID of 931 as a reference frame, and a frame with an ID of 961 as a first selected frame to be compared with the reference frame.

The structural analysis software 207 compares the two frames in step S404 of FIG.
Assume that it is determined in 405 that the difference between the feature values of the two frames is equal to or greater than a predetermined threshold. That is, as shown in FIG. 7, the reference frame is an image of "flower", but the first selected frame is an image of "reporter", and it can be seen that these two images are significantly different.

Subsequently, the structure analysis software 207 stores the position of the first selected frame displayed on the selected frame display section 302b (S407), and selects any one of the frames located between the compared two frames. A detailed comparison mode process is performed to check whether the feature amount of the image is significantly different from the feature amount of the image of the reference frame. In other words, by executing the detailed comparison mode processing, the structure analysis software 207 can find a place where a frame such as the first selected frame shown in FIG. 7 first appears and divide the video stream. Identify appropriate split points.

The structure analysis software 207 is located before the first selected frame (ID: 961) storing the position,
A frame after the reference frame (ID: 931) is selected at a second interval (one frame) smaller than the first interval, and the selected frame is set as a second selected frame to be compared with the reference frame, and the selected frame is displayed. In addition to the display on the unit 302b, a frame position display line 305a indicating the corresponding position in the video stream is displayed on the bar 305 (S408). The structure analysis software 207 sequentially selects frames located after the reference frame as the second selected frame, and compares the selected second selected frame with the reference frame.

Then, the structural analysis software 207 compares the images of the reference frame and the second selected frame (S4).
09), it is determined whether or not the difference between the feature amounts of the images of the reference frame and the second selected frame is equal to or greater than a predetermined threshold (S410).

If it is determined in step S410 that the difference between the feature amounts of the image of the reference frame and the image of the second selected frame is not greater than or equal to the predetermined threshold, the structure analysis software 207
Determines whether all frames between the reference frame and the first selected frame whose position has been stored in step S407 have been processed (S412). That is, the structure analysis software 207 determines whether or not the process of comparing all the frames between the reference frame and the first selected frame and corresponding to the second interval with the reference frame has been completed. .

If it is determined in step S412 that all the frames between the reference frame and the first selected frame have not been processed, a comparison is made with the second selected frame in the area of the graph display section 304 corresponding to the reference frame. The difference between the obtained characteristic amounts is recorded in a line graph (S41).
3).

Then, the structure analysis software 207 returns to step S408, selects a frame at the second interval, and
Is set as the selected frame, and the selected frame display unit 30
2b, and a bar 305 indicates a frame position display line 305a indicating a corresponding position in the video stream.
Is displayed.

FIG. 8 is an explanatory diagram showing an example of the analysis process displayed in the structural analysis process display window 301 after the processes of steps S408 to S413 are performed a plurality of times. In the case of the normal comparison mode, since only the comparison between the reference frame and the first selected frame is performed, the line graph displayed in the area of the graph display unit 304 corresponding to the reference frame at that time is a straight line. (See graphs corresponding to the three left frames in FIG. 8). On the other hand, in the detailed comparison mode,
Since the comparison processing between the reference frame and the plurality of second selection frames is performed, as shown in FIG. 8, the area of the graph display unit 304 corresponding to the reference frame includes the second selection frame on which the comparison processing has been performed. Points are plotted according to the number of
A line graph consisting of a plurality of points will be displayed.
That is, the reference frame (ID: 931) and a plurality of second frames
Are selected based on the result of the sequential comparison of the selected frames (ID: 936 to 938), the difference between the feature amounts of the image of the reference frame and the image of the selected frames is plotted, and a graph corresponding to the reference frame is displayed as a line graph. Display 3
04 is displayed in the area.

FIG. 9 is an explanatory diagram showing an example of the analysis process displayed on the structural analysis process display window 301 after the processes of steps S408 to S413 of FIG. 4 are further repeated. Further, as a result of the comparison processing with the plurality of second selected frames, a line graph including more points than the one shown in FIG. 8 is displayed in the area of the graph display unit 304 corresponding to the reference frame. Have been. As shown in FIGS. 8 and 9, in the area of the graph display unit 304 corresponding to the reference frame, the second selected frame and the reference frame selected from the frames located between the reference frame and the first selected frame are displayed. Will be displayed. That is, the structure analysis software 307 determines whether the graph display unit 304
The line graph display processing is performed while adjusting the interval between the points of the line graph displayed in the area of.

Returning again to step S408 in FIG.
The frame (ID: 9) shown in FIG.
51) is assumed to be selected. In FIG. 10, the reference frame is an image of “flower”, and the second selected frame is an image of “reporter”. Structural analysis software 2
07 compares the image of the reference frame and the image of the second selected frame (S409), and determines whether or not the difference between the feature amounts of the image of the reference frame and the image of the second selected frame is greater than or equal to a predetermined threshold (S410). ).

If it is determined in step S 410 that the difference between the feature amounts of the image of the reference frame and the image of the second selected frame is equal to or greater than a predetermined threshold, the structure analysis software 207
Stores the position of the second selected frame as the division point of the video stream (S411), and records the difference between the feature amounts of the image of the reference frame and the image of the second selected frame in a graph (S406).

After that, the structure analysis software 207 returns to step S401 and executes the structure analysis processing in the normal comparison mode again. In other words, the second selected frame whose position is stored in step S411 is set as a reference frame, and if the set reference frame is not the last frame in the video stream (S401a), the first interval (from the set reference frame) It is determined whether or not a frame exists at a position advanced by 30 frames (S40) (S40).
2). When a frame exists at a position advanced by the first interval, the structure analysis software 207 sets the corresponding frame as a first selected frame to be compared with the reference frame (S403), and sets the position advanced by the first interval. If no frame exists, the structure analysis software 207 sets the last frame as the first selected frame (S415).

FIG. 11 is an explanatory diagram showing an example of the analysis process displayed on the structural analysis process display window 301 after proceeding to step S403 (or step S415) via step S411. The structure analysis software 207 records in a graph the difference between the feature amounts of the images of the second selected frame and the representative frame whose positions have been stored in step S411 in step S406, and determines that the video stream is a division point that can be divided. The displayed dividing line 304b is displayed in the graph display unit 304.
In addition, the structure analysis software 207 displays a divided frame position display line 305 at a position in the bar 305 corresponding to the frame stored in step S411 in order to clarify where in the entire video stream the division point is located.
b is displayed.

If it is determined in step S412 in FIG. 4 that all frames between the reference frame and the first selected frame have been processed, the structure analysis software 207
Stores the first selected frame in which the position is stored in step S407 as a division point of the video stream (S
414), the difference between the feature amounts of the image of the reference frame and the image of the first selected frame is recorded in a graph (S40).
6), and return to step S401. Although there is a difference between the first selected frame and the second selected frame,
In this case, the same processing as that described with reference to FIG. 11 is performed.

FIG. 12 shows that, in step S401a of FIG. 4, when the reference frame set in step S401 is determined to be the last frame, and the structural analysis process is completed, the frame is displayed in the structural analysis process display window 301. FIG. 5 is an explanatory diagram showing an example of an analysis process in which the present invention is performed. A point to be noted in FIG. Since the bar 305 displays the frame position display line 305a in accordance with the positions of all the frames used for the comparison processing, the user understands from the bar 305 the progress of the analysis processing and the end of the analysis processing. be able to. Also,
Since the division frame position display line 305b is displayed in the bar 305, the user can easily understand where the division point is located in the video stream on which the structural analysis processing has been performed. Further, the user can recognize a frame located between the divided frame position display lines 305b as a set of consecutive frames having a small difference in feature amount. The structure analysis software 207 extracts a set of continuous frames having a small difference in the feature amount from the video stream, and allows the video frame to be divided into arbitrary continuous frame units.

The structure analysis software 207 records all the steps of the analysis processing described with reference to the flowchart of FIG. The user can reproduce the process of the analysis process any number of times by clicking the start button 310 after the end of the analysis process. At this time, the user can reproduce the process of the analysis process at a desired speed by adjusting the reproduction speed by the reproduction speed adjustment unit 308.

Here, the process of the analysis process is recorded,
By making the process of the analysis process reproducible, the following effects can be obtained. In other words, the analysis process is not actually performed, but the recorded analysis process is reproduced, so that the time required for the analysis can be reduced, and the analysis process can be reproduced at a desired speed as described above. can do.

The division point of the video stream determined by the structure analysis software 207 is determined based on a physical viewpoint of the feature amount of the image of each frame. Therefore, for example, in video information in which the background changes rapidly, it is expected that many division points will be set according to the changing background. In other words, this division point does not always indicate a logical division point for dividing the video stream into arbitrary continuous frame units that are actually required. Therefore, by reproducing the process of the analysis process, it is possible to check whether or not the division point intended by the user is obtained, so that the division point can be easily modified or changed. In particular, two frames before and after the division point are displayed in the comparison frame display section 302 in the structural analysis process display window 301.
And the difference in the feature amount is displayed on the graph display unit 304.
Is displayed as a line graph, so that it is easy to understand what kind of analysis processing has been performed and the division point has been determined, and it is possible to easily correct or change the division point.

Note that even during the process of actually analyzing the structure of the video stream, the two frames to be compared are displayed on the comparison frame display section 302 in the structure analysis process display window 301, and the characteristic Since the difference in the amount is displayed as a line graph on the graph display unit 304, it is possible to easily grasp what kind of analysis processing is being performed. Corrections and changes can be made easily.

(3) Video Index Information Next, video index information generated based on the result of the video stream structure analysis performed by the structure analysis software 207 will be described. The video index information is used to structure and manage the video information and to express the internal logical structure of the video information. The video index information is generated by the index generation software 208 shown in FIG. 2 at an arbitrary timing based on the analysis result by the structure analysis software 207. Hereinafter, the video index information will be described in the order of the data structure of the video index information and the method of generating the video index information.

Data Structure of Video Index Information FIG. 13 is an explanatory diagram showing a data structure of video index information. As shown in FIG. 13, the video index information 500 has a structure in which a plurality of structural element objects are hierarchized, and the video index information 500 itself can be handled as one data file (index file). it can. Therefore, the video index information 500 can be stored integrally with the video information, or can be stored separately from the video information.

The structure element objects of the video index information 500 include tree information 501 and frame information 50
2 (502A to 502F) and sound information 503 (5
03A to 503F) and segment information 504 (504
A to 504F) and package information 505 (505A to 505F).
505C).

The tree information 501 is information set at the top of the tree structure of one piece of video index information 500, and all structural element objects related to one piece of video information are arranged in a lower layer. In other words,
This is information declaring the entrance of one piece of video index information 500. Below the tree information 501, one piece of package information 505A representing the entire video information (audio and video) is arranged.
One segment information 504A representing the entire video information whose structure has been analyzed by the above-described video stream structure analysis processing is arranged below 05A.

The frame information 502 manages an arbitrary continuous frame in the video information (for example, a set of continuous frames having a small difference in feature amount extracted from the video stream by the structural analysis software 207) as one unit of information. The information includes a node ID in the tree structure, a range of continuous frames managed by the frame information, and a pointer indicating a position in the video information.

The sound information 503 is information for managing sound corresponding to an arbitrary range in the video information as one unit of information, and includes a node ID in the tree structure, an arbitrary range managed by the sound information, And a pointer indicating a position in the video information. In the above-described structure analysis processing of the video stream, audio is not mentioned, but the audio is attached to the video information.

The segment information 504 is information for classifying the frame information 502 and the sound information 503, which are one unit of information, and managing them as a set having a desired meaning, and the frame information managed by the segment information 504. This is information for managing package information 505 for collectively managing a plurality of other segment information 504 generated using frames and sounds in the same range as the information 502 and the sound information 503. The segment information 504 includes node IDs in the tree structure, identification information for identifying frame information, sound information, and package information managed by the segment information 504;
Further, it has a pointer to higher-level package information 505 to which the segment information 504 belongs.

In FIG. 13, segment information 504A
Is information for managing the frame information 502A, the sound information 503A, and the package information 505B. The node ID of the segment information 504A, the specific information for specifying the frame information 502A, the sound information 503A, and the package information 505B; And a pointer to higher-level package information 505A to which the segment information 504A belongs.

The package information 505 is information for managing a plurality of pieces of segment information 504 collectively, and includes a node ID in the tree structure, a list of segment information 504 managed by the package information 505, and the package information 505. Top segment information 50 to which it belongs
4 is provided.

For example, in FIG. 13, package information 505B includes three pieces of segment information 504B to 504D.
And a list of the segment information 504B to 504C managed by the package information 505B and a pointer to the higher-level segment information 504A to which the package information 505B belongs. Have.

As described above, in the video index information 500, the package information 505 in addition to the frame information 502 and the sound information 503 is arranged below one piece of segment information 504 in the tree structure composed of the structural element objects. be able to.

Further, in the video index information 500, it is possible to arrange an arbitrary attribute object in the tree structure. Using this attribute object, it is possible to hold necessary data corresponding to the structural element object. it can. By setting, as the attribute object, search information for searching for the contents of frames or audio managed directly or indirectly by each structural element object, the details of the video information structured by the video index information are set. Search can be easily performed.

Next, an example of a process of generating the video index information 500 having the above-described data structure will be briefly described. The following process will be described as a process performed by the index generation software 208.
08 can be operated.

The segment information 504 in the video index information 500 represents a lump of continuous frames and continuous audio of the divided video (moving image and audio), and can be defined as one unit of information of a structured video. One piece of long video information can be considered as a whole at first, and therefore exists as one piece of segment information 504A (see FIG. 13). The other segment information 504 other than the segment information 504A is the parent segment information 5
04 is a divided part. For example, each of the segment information 504B to 504D is a part obtained by dividing the parent segment information 504A.
4E and 504F are parent segment information 50, respectively.
4C is a part obtained by dividing 4C.

When the parent segment information 504 (for example, the segment information 504B) is divided into a plurality of pieces, the segment information 504 which is a part of the division is obtained.
(Here, 504E and 504F) are collectively managed as package information 505.

The index generation software 208 generates segment information 504 and package information having the above-described properties based on the division points of the video stream specified by the structure analysis software 207.

FIGS. 14 (a) to 14 (c) show the case where the dividing point for dividing the video information step by step using the above-described structural analysis software 207 is specified and generated by the index generating software 208. Segment information 5
FIG. 4 is an explanatory diagram showing a package information 504 and package information 505. 14 shows the segment information 504 and the package information 505 generated by the index generation software 208, and the left side shows the state of video information structurally analyzed by the structural analysis software 207 in a series of frames. . However, the state of the video information is represented by video index information 500 for easy understanding.
This is an abstraction (image) of the frame information 502 and the sound information 503 in the middle, and does not actually indicate the state of raw video information (original video information).

First, as shown in FIG. 14A, the index generation software 208 corresponds to the entire video stream of the video information analyzed by the structure analysis software 207 and includes tree information 501, one package information 505, The segment information 504 is generated. That is, the index generation software 208 generates tree information 501 that declares the entrance of one piece of video index information 500, and generates one piece of package information 505A representing the entire video stream below the tree information 501. Then, one piece of segment information 504A representing the entire video stream is generated below the package information 505A.

Subsequently, the index generation software 208
Divides the video stream into a set of continuous frames according to the division point of the video stream determined by the structure analysis software 207, as shown on the left side of FIG.

The left side of FIG. 14B shows an example in which the entire video stream of video information is divided into three based on the division points determined by the structure analysis software 207, and the index generation software 208 In response to this, three pieces of segment information 504B to 504D are generated below the segment information 504A, and
The package information 505B that collects the two pieces of segment information 504B to 504D is generated. The index generation software 208 associates the segment information 504A with the package information 505B, and
B is managed by the segment information 504A.

Further, by analyzing the structure of the video stream in detail or from another viewpoint using the structure analysis software 207, the divided video information can be obtained, for example, as shown in FIG.
Can be further divided at the portion indicated by the arrow to the left of.

When such a structure analysis process is performed,
As shown on the right side of FIG. 14C, the index generation software 208 associates the segment information 504B to 504D with the package information 505C to 505E, respectively.
Is generated, and the segment information 504E and 50
4F is associated with the package information 505C, the divided segment information 504G and 504H is associated with the package information 505D, and the divided segment information 504I is obtained.
And 504J are associated with the package information 505E.

[0103] The video index information 500 generated in this manner is structured more finely as it goes to lower-level constituent object objects.

Referring to FIGS. 15A to 15C, a method of generating video index information having a structure different from that of video index information 500 shown in FIG. 14 will be briefly described.

Here, the structure analysis processing is performed twice from different viewpoints using the structure analysis software 207 on the video stream of the same video information, and the video is analyzed from one analysis result as shown on the left side of FIG. It is assumed that a division point for dividing the entire stream into three is obtained, and a division point for dividing the entire video stream into two is obtained from another analysis result as shown on the left side of FIG. 15B.

The index generation software 208 is the same as that shown in FIG.
As shown in (a), tree information 501, package information 505A, and segment information 504A for managing the entire video information are generated. Then, three pieces of segment information 504B to 504D for managing the video information divided into three by the division points are generated, and package information 50 for managing the segment information 504B to 504D is generated.
5B is generated and associated with the segment information 504A.
Subsequently, the index generation software 208
As shown in (b), two pieces of segment information 504 for managing video information divided into two by a division point
E and 504F are generated, and package information 505C for managing the two pieces of segment information 504E and 504F is generated. As a result, as shown in FIG. 14C, it is possible to obtain video index information 500 having a structure in which two pieces of package information 505B and 505C are associated below the segment information 504A.

The frame information 502 and sound information 503 corresponding to each segment information 504 are shown in FIG.
As shown in FIG. 3, it is held below the corresponding segment information 504. The sound information 503 is divided according to the division point of the video stream when the index generation software 208 structures the video information based on the result analyzed by the structural analysis software 207.

Thus, the index generation software 20
8 generates tree structure information 501, frame information 502, sound information 503, segment information 504 and package information 505 based on the analysis result obtained from the structure analysis software 207, and generates video index information 500. Structure video information. The generated video index information 500 is stored in the video index information database 102 shown in FIG.

(4) Display of Structure of Video Stream Next, how to display the structure of the video stream of the video information based on the video index information 500 generated by the index generation software 208 will be described.

FIG. 16 is an explanatory diagram showing a screen display example of the structure of video information displayed based on video index information 500. The user operates the structure display software 2 shown in FIG.
09 is started, and by specifying the video index information 500 stored in the video index information database 102 shown in FIG. 1, the structure display window 601 shows what structure the desired video information has.
Can be displayed inside.

The structure display software 209 refers to the information indicating the relationship between the structure element object information included in each structure element object in the video index information 500, and forms each structure element in a tree structure as shown in FIG. Display objects.

FIG. 16 is a screen display of the video index information 500 generated as shown in FIG. 15, in which the file name of the video information or the title of the video information managed by the video index information 500 is displayed. The tree information 501 to be displayed is arranged at a higher position, the package information 505A is arranged below the tree information 501, and the package information 5 is arranged below the package information 505A.
Segment information 504A managed by 04A
Place. Further, package information 505B and 505C managed by the segment information 504A are arranged below the segment information 504A, and segment information 504B to 504I managed by the package information 505B are arranged below the package information 505B. The segment information 504 </ b> J to 504 </ b> M arranged under the package information 505 </ b> C and managed by the package information 505 </ b> C
Is arranged.

Note that the frame information 502 shown in FIG.
The sound information 503 is displayed together with the segment information 504 that manages them.

As described above, the segment information 504 can manage package information for collectively managing a plurality of other segment information generated using a set of frames managed by the segment information. It is. Although not shown, such segment information 5
04, the structure display software 209 arranges and displays the package information 505 below the segment information 504, and arranges the segment information 504 managed by the package information below the package information 505. To display in a tree structure.

As described above, since each constituent object is displayed in a hierarchical structure by the tree structure, the user
It is possible to grasp the logical structure inside the video information at a glance.

Here, it is assumed that the user has clicked on the package information 505B shown in FIG. The structure display software 209 refers to the frame information 502 managed by each segment information 504 according to a user's input operation, and displays a thumbnail of a representative frame in a continuous frame managed by each segment information 504. . FIG.
FIG. 8 is an explanatory diagram showing an example in which a representative frame 602 of each of the segment information 504B to 504I managed by the package information 505B is displayed. FIG. 17 shows an example in which the leading frame of a continuous frame managed by each segment information 504 is displayed as a representative frame 602. It is not limited to the first frame of the managed frame. That is, a representative frame can be arbitrarily selected from a set of frames managed by the segment information 504, and the number of representative frames is not limited to one but may be two or more.

Further, even if any of the segment information 504 is clicked by the user, the structure display software 209 displays the clicked segment information 50.
4 representative frame 602 is displayed.

As described above, when the user sets the package information 50
By clicking on 5 or the segment information 504, the representative frame 602 of the corresponding segment 504 is displayed as a thumbnail, so that it is possible to grasp the contents of each segment information 504 without playing an actual video. .

In addition, the user selects package information 505,
When reproduction is designated (or double-clicked) from a menu (not shown), a video managed by all segment information 505 displayed below the package information 505 is reproduced. Also, segment information 504
Is selected and playback is designated (or double-clicked) from a menu (not shown), the video managed by the segment information 504 is played back.

Although detailed description is omitted,
Each segment information 504 manages the frame information 502 and the sound information 503 shown in FIG.
There are two types of video reproduction here: reproduction using both frame information 502 and sound information 503 (when audio is included), and reproduction using only frame information 502 (when audio is not included). Can be Also,
By using only the sound information 503, it is also possible to reproduce only the sound in the unit of the package 505 or the unit of the segment 504.

(5) Display of search result of video information As described above, when segment information 504 and package information 505 are generated, desired information is added to an attribute object (for example, search information) according to the purpose of use and utilization. ), And can be used later for searching. For example, it is possible to set a keyword or the like which is a feature of each structural element object using text information as an attribute object, and later search or classify using the keyword. For example, in addition to searching only for titles, the content of video information may include "video that appeared and talked to", "video with scenes similar to this scene", and " A search that extends to the information of the content of the video information, such as “video that exists”.

The structure display window 601 shown in FIG.
Above, input of search conditions, designation of execution of search processing, display of search results, and the like can be performed. In this case, based on the video index information 500, the structure of the video stream of the video information that matches the search condition can be displayed as a search result in a tree structure shown in FIG. FIG. 18 shows that the two pieces of video information match the search condition. As described above, by displaying the structure of the video information of the search result, the search result can be verified from various angles, and editing work or the like can be performed using the search result.

In addition, when the structure of the video information that matches the search condition is displayed in a tree structure, the structural element object that matches the search condition can be displayed separately from other constituent object objects. In FIG. 18, a matching mark 603 is displayed for the segment information 504 that matches the search condition. In this way, by displaying the component object objects that match the search condition in a distinguished manner, it is possible to immediately reproduce a scene that matches the search condition.

(6) Application Example Next, an application example relating to the structure display of a video stream will be described. FIG. 19 is an explanatory diagram showing the data structure of the video index information 500. Components common to the video index information 500 shown in FIG. 13 are denoted by the same reference numerals, and description thereof will be omitted.

The video index information 500 includes at least a plurality of segment information 504 for managing a range of an arbitrary continuous frame in the video information, and a segment information indicating the order of the plurality of segment information 504. View information 701 consisting of a link list between
(701A to 701C).

A plurality of view information items 701 can be set.
A plurality of segment information 5 which constitutes video information in advance
04, a plurality of (here, three) link lists that trace a desired number of segment information 504 in a desired order.
View information 70 generated and corresponding to each link list
Set as 1A to 701C.

The view information 701 expresses how to follow the segment information 504 (or package information 505) as a link list in order to express one piece of video information from a certain viewpoint. The segment information 50
Any of a link list of only four, a link list of only package information 505, and a mixed link list can be expressed, but is basically the same as the link list of the segment information 504. Although a method of creating the view information 701 is omitted, it can be easily created by a dedicated creation module.

Next, with reference to FIG.
The link list 1 will be specifically described. Depending on how the link list of the view information 701 is set, a digest (summary) of the entire video information viewed from a specific viewpoint can be created with one view information 701, or two pieces of segment information 504 can be simply connected. Is also free.

In FIG. 20, the link list will be described by taking as an example the case where the segment information of the package information B obtained by dividing the entire video information for a certain purpose is roughly reproduced. Here, the package information B is composed of three pieces of segment information, and the segment information at the head of the package information B includes lower package information E and F.

First, when only the package information B is designated and simply reproduced, the three pieces of segment information B1 to B3 in the package information B are reproduced as they are. Here, "Segment information E1 → E2 → B2 → B
When "3" is created, the segment information is reproduced in the order shown in (1) of FIG. That is, segment information B
1 is the same as that of the portion corresponding to the segment information F1 is not reproduced, which means that the segment information B1 is roughly reproduced.

By creating a link list of view information 701 in this way, segment information can be specified in a desired order.

Further, the link list is changed to “segment information G”.
20 is created as 1 → G2 → G3 → C3 → D1 ”
The segment information is reproduced in the order shown in (2).
Change the link list to "Segment information G1 → G2 → G3 → C
If "3.fwdarw.I1.fwdarw.I2" is created, the segment information is reproduced in the order shown by (3) in FIG.

If the representative frame of each segment information (for example, frame information for a predetermined time at the head of each segment information) is reproduced instead of reproducing the segment information itself, digest reproduction (summary) Reproduction) can be remarkable. In particular,
If it is assumed that frame information in the segment information is traced instead of tracing the segment information itself, the effect of the link list can be easily realized. For example, in the case of (1) in FIG.
Package information E obtained by dividing the segment information B1 into three pieces of segment information B1 to B3 is composed of two pieces of segment information E1 and E2 of five minutes each, and the package information F is composed of 10 minutes of segment information F1. In this case, the frame information of each one minute is reproduced from each segment information, so that the content is 1/5, 1 /
The roughness becomes 5, 1/20, 1/20, and the intention of the digest can be reflected.

Although FIG. 20 shows only an example in which the link list traces the segment information in chronological order, for example, as the link list, “segment information E2 → G3 →
The link may be jumped like “H2 → I2 → D2”. Further, the link list may be created in a desired order ignoring the time series.

Further, according to the contents of the video information in advance, for example, view information having a link list linking only segment information excluding violent scenes, or view information having a link list linking only segment information of intensely shot scenes View multiple pieces of view information according to the purpose of use and utilization, and select desired view information as needed to view one video information (content) from a multipurpose viewpoint. Becomes possible.

The structure display software 209 can also display the view information 701 having the above-described functions when displaying the structural element objects constituting the video index information 500 in a tree structure as shown in FIG. it can. As a result, when the user selects the view information 701 displayed in the structure display window 601,
Video information can be reproduced by the method described with reference to FIG. That is, by displaying the view information 701 having the link list as a structural element object in the structure display window 601, the degree of freedom of the method of reproducing the video information is increased, and the video information can be used and utilized efficiently and effectively. A user interface can be provided.

As described above, according to the embodiment of the present invention, the burden on the operator when analyzing video information can be reduced by the video stream structure analysis processing described with reference to FIG. .

In analyzing the structure of the video stream, the structure analysis process display window 301 is displayed to display the analysis process of the video information, so that the analysis process of the video information can be visually confirmed. Thus, the analysis process can be verified, and the analysis processing of the video information can be performed smoothly, easily and accurately.

Furthermore, the logical structure inside the video information can be expressed in a tree structure by using the video index information generated based on the analysis result of the video stream, thereby simplifying the logical structure inside the video information. Will be able to understand.

As shown in FIG. 2, in the embodiment of the present invention, the structure analysis of the video stream, the generation of the video index information, and the structure display of the video stream based on the video index information are performed by the structure analysis software. 20
7. The description has been made on the assumption that the application is realized by three application programs, namely, the index generation software 208 and the structure display software 209. However, these application programs are only examples,
It is not intended to limit the application programs for implementing the present invention.

The application program is a floppy disk, CD-ROM, MO, DV
D, a computer-readable recording medium such as a memory card. Further, the application program can be provided to the computer 103 shown in FIGS. 1 and 2 via a network. The computer 103 reads and executes the application program provided in this manner to realize the above-described various processes.

[0142]

As described above, according to the video stream structure analysis method of the present invention (claim 1), an arbitrary frame in the video stream is set as a reference frame, and in the time series of the video stream, It is determined whether or not a frame exists at a position advanced by a first interval from the set reference frame, and when a frame exists, the corresponding frame is selected and set as a first selected frame,
If no frame exists, the last frame is set as the first selected frame, the first selected frame and the reference frame are compared, and the difference between the feature amounts of the images of the first selected frame and the reference frame is determined by a predetermined threshold value. In the above case, the position of the first selected frame is stored. When the position is not equal to or larger than the threshold value, the first selected frame is set as the reference frame, and the process returns to the process of selecting the first selected frame. When the position is stored, the frames after the frame set as the first selected frame before the stored first selected frame are sequentially selected at a second interval finer than the first interval, and the selected frame is selected. Is set as the second selected frame, the second selected frame and the reference frame are compared, and the feature amount of the image of the second selected frame and the reference frame is calculated. If There when more than a predetermined threshold value, and stores the position of the second selected frame, not more than the threshold value,
Returning to the process of selecting the second selected frame, when the position of the second selected frame is stored, the second selected frame is set as a reference frame, and when a new reference frame is set, the video stream is set. It is determined whether or not a frame exists at a position advanced by the first interval from the set reference frame, and if there is a frame, the process returns to the process of selecting the first selected frame, and the frame does not exist In this case, the last frame is set as the first selected frame, and the process returns to the process of setting the first selected frame. When the reference frame is the last frame, the process is terminated, thereby storing the stored second selected frame. Based on the position of the video stream, extract a set of consecutive frames with a small difference in feature value from the video stream, and convert the video stream into arbitrary continuous frame units. In order to be divided, it is possible to reduce the burden on the worker at the time of analyzing the video information.

According to the video stream structure analysis method of the present invention (claim 2), an arbitrary frame in the video stream is set as a reference frame, and in the time series of the video stream, frames subsequent to the reference frame are set. First
, The selected frame is set as the first selected frame, the first selected frame and the reference frame are compared, and the difference between the feature amounts of the images of the first selected frame and the reference frame is determined by a predetermined value. If the value is equal to or greater than the threshold value, the position of the first selected frame is stored. If the value is not equal to or greater than the threshold value, the process returns to the process of selecting the first selected frame. The frames after the frame set as the first selected frame before the one selected frame are sequentially selected at a second interval finer than the first interval, and the selected frame is set as a second selected frame. The second selected frame and the reference frame are compared, and if the difference between the feature amounts of the images of the second selected frame and the reference frame is equal to or greater than a predetermined threshold, the position of the second selected frame Stored, a second if not greater than or equal to the threshold
When the position of the second selected frame is stored, the second selected frame is set as a reference frame, and when a new reference frame is set, the process returns to the video stream. It is determined whether or not there is a frame at a position advanced by the first interval from the set reference frame. If there is a frame, the process returns to the process of selecting the first selected frame. Let the last frame be the first selected frame,
Returning to the process of setting the first selected frame, by ending the process when the reference frame is the last frame, the difference between the feature amounts from the video stream based on the stored position of the second selected frame is returned. Since a set of consecutive frames with a small number of frames is extracted and the video stream can be divided into arbitrary continuous frame units, it is possible to reduce the burden on the operator when analyzing the video information.

According to the video stream structure analysis process display method of the present invention (claim 3), an arbitrary frame in the video stream is set as a reference frame, and the image of the set reference frame is displayed. In the time series of the stream, it is determined whether or not a frame exists at a position advanced by the first interval from the set reference frame, and if a frame exists, the corresponding frame is selected and set as a first selected frame. The image of the set first selected frame is displayed together with the image of the reference frame, and if there is no frame, the last frame is set as the first frame, and the image of the set first selected frame is set as the reference. Displaying with the image of the frame, comparing the first selected frame and the reference frame, the image of the first selected frame and the reference frame If the difference between the feature amounts is equal to or larger than a predetermined threshold, the position of the first selected frame is stored. If the difference is not equal to or larger than the threshold, the first selected frame is set as a reference frame, and the image of the set reference frame is currently displayed. The process returns to the process of selecting the first selected frame by displaying in place of the image of the reference frame that is present, and when the position of the first selected frame is stored, the first selected frame is displayed before the stored first selected frame. Frames subsequent to the frame set as one selected frame are sequentially selected at a second interval finer than the first interval, the selected frame is set as a second selected frame, and an image of the set second selected frame Is displayed in place of the currently displayed image of the first selected frame, the second selected frame and the reference frame are compared, and the images of the second selected frame and the reference frame are displayed. If the difference in the amount of collection is equal to or larger than the predetermined threshold, the position of the second selected frame is stored. If the difference is not equal to or larger than the threshold, the process returns to the process of selecting the second selected frame, and the position of the second selected frame is set. When storing, the second selected frame is set as the reference frame, the image of the set reference frame is displayed instead of the image of the currently displayed reference frame, and when a new reference frame is set, In the video stream, it is determined whether or not a frame exists at a position advanced by the first interval from the set reference frame, and if there is a frame, the process returns to the process of selecting the first selected frame. If not, the last frame is set as the first selected frame, and the process returns to the process of setting the first selected frame. If the reference frame is the last frame, the process ends. In addition, in the process of the above process, the difference between the feature amounts of the reference frame, the first selected frame, and the second selected frame is sequentially plotted, and the change of the feature amount of each frame in the video stream is displayed in a graph. Therefore, the analysis process of the video information can be visually confirmed, and the analysis process can be easily verified, so that the analysis process of the video information can be performed smoothly, simply, and accurately.

According to the video stream structure analysis process display method of the present invention (claim 4), an arbitrary frame in the video stream is set as a reference frame, and the image of the set reference frame is displayed. In the stream time series, the frames after the reference frame are sequentially selected at a first interval, the selected frame is set as the first selection frame, and the image of the set first selection frame is displayed together with the image of the reference frame. Then, comparing the first selected frame and the reference frame, storing the position of the first selected frame when the difference between the feature amounts of the images of the first selected frame and the reference frame is equal to or greater than a predetermined threshold, If not, the process returns to the process of selecting the first selected frame, and if the position of the first selected frame is stored, the stored first selected frame is stored. A first set frame and subsequent frames as the selected frame are sequentially selected with a fine second gap than the first interval earlier, the selected frames 2
Is set as the selected frame, the set image of the second selected frame is displayed in place of the currently displayed image of the first selected frame, and the second selected frame and the reference frame are compared. When the difference between the feature amounts of the images of the selected frame and the reference frame is equal to or larger than a predetermined threshold, the position of the second selected frame is stored.
Returning to the process of selecting the selected frame, when the position of the second selected frame is stored, the second selected frame is set as the reference frame, and the image of the set reference frame is set as the reference frame of the currently displayed reference frame. It is displayed in place of an image, and when a new reference frame is set, it is determined whether or not a frame exists at a position advanced by the first interval from the set reference frame in the video stream,
When there is a frame, the process returns to the process of selecting the first selected frame. When there is no frame, the last frame is set as the first selected frame, and the process returns to the process of setting the first selected frame. In the case of the last frame, the process ends, and in the process of the above process, the reference frame, the first selected frame, and the second
In order to plot the difference of the feature value from the selected frame in order and display the change of the feature value of each frame in the video stream in a graph, the analysis process of the video information can be visually confirmed and the analysis process Can be easily verified, so that the analysis processing of the video information can be performed smoothly, simply, and accurately.

Further, according to the video stream structure analysis process display method of the present invention (claim 5), the position of the second selected frame having a feature amount equal to or larger than the threshold value with respect to the reference frame is displayed in the graph. Therefore, the position of the division point of the video stream can be easily specified, and it is possible to easily know what analysis has determined the division point.

According to the video stream structure analysis process display method (claim 6) of the present invention, further, a graph indicating the progress of the analysis process is displayed in association with the position of each frame constituting the video stream. In addition to displaying, in the graph, the position of the frame selected as the first selected frame or the second selected frame and the position of the second selected frame having a feature amount equal to or larger than the threshold with respect to the reference frame are displayed in a graph. The positional relationship of each frame with respect to the entire video stream can be easily grasped. In particular, since the position of the second selected frame having a feature value equal to or larger than the threshold value is displayed in the graph, it is easy to grasp where the dividing point of the video stream is and how many times the video stream can be divided. Can be.

According to the video stream structure analysis process display method (claim 7) of the present invention, it is possible to record the analysis process and reproduce the recorded analysis process at an arbitrary timing. It is possible to check whether the division point intended by the user has been obtained, and it is possible to easily correct or change the division point. Also, since the analysis process is not actually performed again, but the recorded analysis process is reproduced, the time required for the analysis can be reduced, and the analysis process can be reproduced at a desired speed. It becomes possible.

Further, according to the video stream structure display method of the present invention (claim 8), the video index information is
At least one segment information for managing a set of consecutive frames having a small difference in feature amount extracted from the video stream as a set of meaningful frames in one unit, and package information for collectively managing a plurality of segment information; At least as a structural element object, the package information is arranged at a higher level, the segment information managed by the package information is arranged at a lower level of the package information, and the structural element object is displayed in a tree structure. Can be used to express the logical structure inside the video information, and the logical structure inside the video information can be easily visualized.

According to the video stream structure display method of the present invention (claim 9), the segment information further includes a plurality of other segment information generated by using a set of frames managed by the segment information. It is possible to manage package information for collective management, arrange and display package information below one segment information, and display segment information managed by the package information below the package information. Are arranged and displayed in a tree structure, so that the logical structure inside the video information can be expressed using the result of analyzing the video information, and the logical structure inside the video information can be easily visualized. Become.

According to the video stream structure display method of the present invention, at least one frame arbitrarily selected from a set of frames managed by the segment information is set as a representative frame. Since it is displayed, the contents of each segment information can be grasped at a glance.

Further, according to the video stream structure display method of the present invention (claim 11), the video stream structure is further displayed using video index information created in advance for each video information.
When a search process is performed for video information that matches a desired search condition from a plurality of video information, the video of the video information that matches the search condition is determined based on the video index information of the video information that matches the search condition. Since the structure of the stream is displayed in a tree structure, search results can be verified from various angles, and editing work or the like can be performed using the search results.

Furthermore, according to the video stream structure display method of the present invention (claim 12), when the video stream structure of the video information that matches the search condition is displayed in a tree structure, the structure that matches the search condition is displayed. Since the element object is displayed while being distinguished from other component objects, it is possible to immediately reproduce a scene that matches the search condition.

[Brief description of the drawings]

FIG. 1 is a schematic configuration diagram showing an example of a system for implementing a video stream structure analysis method, a structure analysis process display method, and a structure display method according to an embodiment of the present invention.

FIG. 2 is a block diagram of a computer constituting the system of FIG. 1;

FIG. 3 is an explanatory diagram showing a configuration example of a structure analysis process display window for displaying a process of the analysis process in the structure analysis process of the video stream according to the embodiment of the present invention.

FIG. 4 is a flowchart showing a video stream structure analysis process according to the embodiment of the present invention.

FIG. 5 is an explanatory diagram showing an example of an analysis process displayed in a structure analysis process display window in a video stream structure analysis process according to the embodiment of the present invention.

FIG. 6 is an explanatory diagram showing an example of an analysis process displayed in a structure analysis process display window in a video stream structure analysis process according to the embodiment of the present invention.

FIG. 7 is an explanatory diagram showing an example of an analysis process displayed in a structure analysis process display window in a video stream structure analysis process according to the embodiment of the present invention.

FIG. 8 is an explanatory diagram showing an example of an analysis process displayed in a structure analysis process display window in a video stream structure analysis process according to the embodiment of the present invention.

FIG. 9 is an explanatory diagram showing an example of an analysis process displayed in a structure analysis process display window in the video stream structure analysis processing according to the embodiment of the present invention.

FIG. 10 is an explanatory diagram showing an example of an analysis process displayed in a structure analysis process display window in a video stream structure analysis process according to the embodiment of the present invention.

FIG. 11 is an explanatory diagram showing an example of an analysis process displayed on a structure analysis process display window in a video stream structure analysis process according to the embodiment of the present invention.

FIG. 12 is an explanatory diagram showing an example of an analysis process displayed in a structure analysis process display window in the video stream structure analysis processing according to the embodiment of the present invention.

FIG. 13 is an explanatory diagram showing a data structure of video index information according to the embodiment of the present invention.

FIG. 14 is an explanatory diagram showing a process of generating video index information according to the embodiment of the present invention.

FIG. 15 is an explanatory diagram showing a process of generating video index information according to the embodiment of the present invention.

FIG. 16 is an explanatory diagram showing a screen display example of the structure of video information displayed based on video index information according to the embodiment of the present invention.

17 is an explanatory diagram showing a state in which the package information shown in FIG. 16 is selected and a representative frame of each segment information is displayed.

FIG. 18 is an explanatory diagram showing a state in which a structure of video information obtained as a search result is displayed when video information search processing is performed using video index information according to the embodiment of the present invention.

FIG. 19 is an explanatory diagram showing a data structure of video index information used as an application example in the embodiment of the present invention.

20 is an explanatory diagram showing a link list of view information provided in the video index information shown in FIG.

[Explanation of symbols]

 Reference Signs List 101 Video information library 102 Video index information database 103 Computer 207 Structure analysis software 208 Index generation software 209 Structure display software 301 Structure analysis process display window 302 Comparison frame display 302a Reference frame display 302b Selected frame display 303a to 303c Frame display 304 Graph display section 304a Separator line 304b Division line 305 Bar 305a Frame position display line 305b Division frame position display line 308 Reproduction speed adjustment section 500 Video index information 501 Tree information 502 Frame information 503 Sound information 504 Segment information 505 Package information 601 Structural display Window 602 Representative frame 603 Match mark

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Takayuki Kunieda 1-3-6 Nakamagome, Ota-ku, Tokyo Inside Ricoh Company, Ltd. (72) Masajiro Iwasaki 1-3-6 Nakamagome, Ota-ku, Tokyo F term in Ricoh Co., Ltd. (reference) 5B050 BA11 CA07 EA18 EA24 FA02 FA12 FA13 FA14 FA19 5L096 DA04 GA19 HA02 MA07

Claims (12)

[Claims] 1. A first step of setting an arbitrary frame in a video stream as a reference frame, and a frame exists at a position advanced by a first interval from the set reference frame in a time series of the video stream. It is determined whether or not to perform the operation. If a frame exists, the corresponding frame is selected and set as the first selected frame. If no frame exists, the last frame is set as the first selected frame. A second step, comparing the first selected frame and the reference frame,
When the difference between the feature amounts of the images of the first selected frame and the reference frame is equal to or greater than a predetermined threshold, the position of the first selected frame is stored. A third step of setting as the reference frame and returning to the second step; and, if the position of the first selected frame is stored in the third step, Frames subsequent to the frame previously set as the first selected frame in the second step are sequentially selected at a second interval finer than the first interval, and the selected frame is set as a second selected frame. A fourth step, comparing the second selected frame and the reference frame,
If the difference between the feature amounts of the images of the second selected frame and the reference frame is equal to or greater than a predetermined threshold, the position of the second selected frame is stored. If not, the process returns to the fourth step. A fifth step, when the position of the second selected frame is stored in the fifth step, a sixth step of setting the second selected frame as the reference frame, and a sixth step When a new reference frame is set, it is determined whether or not a frame exists at a position advanced by the first interval from the set reference frame in the video stream. Returning to the second step, and when there is no frame, a seventh step of returning to the second step with the last frame as the first frame; and wherein the reference frame is the last frame. An eighth step of terminating the processing when the second frame is stored in the fifth step, based on the position of the second selected frame stored in the fifth step. A structure analysis method for a video stream, wherein a set is extracted and the video stream can be divided into arbitrary continuous frame units. 2. A first step of setting an arbitrary frame in a video stream as a reference frame, and sequentially selecting frames after the reference frame at a first interval in a time series of the video stream, and selecting the selected frames. A second step of setting a frame as a first selected frame; and comparing the first selected frame and a reference frame;
If the difference between the feature amounts of the images of the first selected frame and the reference frame is equal to or larger than a predetermined threshold, the position of the first selected frame is stored. If not, the process returns to the second step. A third step, and in a case where the position of the first selected frame is stored in the third step, the position of the first selected frame is stored as the first selected frame in the second step before the stored first selected frame. A fourth step of sequentially selecting frames subsequent to the set frame at a second interval finer than the first interval, and setting the selected frame as a second selected frame; Compare frames,
If the difference between the feature amounts of the images of the second selected frame and the reference frame is equal to or greater than a predetermined threshold, the position of the second selected frame is stored. If not, the process returns to the fourth step. A fifth step, when the position of the second selected frame is stored in the fifth step, a sixth step of setting the second selected frame as the reference frame, and a sixth step When a new reference frame is set, it is determined whether or not a frame exists at a position advanced by the first interval from the set reference frame in the video stream. Returning to the second step, and when there is no frame, returning to the second step the last frame as the first selected frame; and the reference frame is the last frame. An eighth step of terminating the process when there is a continuous frame having a small difference in the feature amount from the video stream based on the position of the second selected frame stored in the fifth step. A video stream structure analysis method, wherein a set of the video streams is extracted and the video stream can be divided into arbitrary continuous frame units. 3. A method for displaying a structure analysis process of a video stream for displaying an analysis process for analyzing a structure of a video stream, the method comprising: setting an arbitrary frame in the video stream as a reference frame; A first step of displaying an image of the video stream; and determining whether or not a frame exists at a position advanced by a first interval from the set reference frame in the time series of the video stream. The corresponding frame is selected and set as the first selected frame, the image of the set first selected frame is displayed together with the image of the reference frame, and if no frame exists, the last frame is set to the first frame. A second step of setting the image of the first selected frame and displaying the image of the set first selected frame together with the image of the reference frame; Compare one selected frame with a reference frame,
When the difference between the feature amounts of the images of the first selected frame and the reference frame is equal to or greater than a predetermined threshold, the position of the first selected frame is stored. A third step of setting as the reference frame, displaying the set reference frame image instead of the currently displayed reference frame image, and returning to the second step; When the position of the first selected frame is stored, frames after the frame set as the first selected frame in the second step before the stored first selected frame are stored in the first interval. The selected frames are sequentially selected at smaller second intervals, the selected frame is set as a second selected frame, and the image of the set second selected frame is replaced with the image of the currently displayed first selected frame. And comparing the second selected frame and the reference frame,
If the difference between the feature amounts of the images of the second selected frame and the reference frame is equal to or greater than a predetermined threshold, the position of the second selected frame is stored. If not, the process returns to the fourth step. A fifth step, when the position of the second selected frame is stored in the fifth step, the second selected frame is set as the reference frame, and an image of the set reference frame is currently displayed. A sixth step of displaying in place of the image of the reference frame being set, and, when a new reference frame is set in the sixth step, in the video stream, the first interval from the set reference frame. It is determined whether or not a frame exists at the position advanced by one minute. If a frame exists, the process returns to the second step. If no frame exists, the last frame is replaced with the first selected frame. A seventh step of returning to the second step as a frame, and an eighth step of terminating the process when the reference frame is the last frame. The difference between the feature amounts of the reference frame and the first and second selected frames obtained in the fifth step is sequentially plotted, and the change in the feature amount of each frame in the video stream is displayed in a graph. And displaying a structure analysis process of a video stream. 4. A method for displaying a structure analysis process of a video stream for displaying an analysis process of analyzing a structure of a video stream, the method comprising: setting an arbitrary frame in the video stream as a reference frame; A first step of displaying the image of the above, and in the time series of the video stream, frames after the reference frame are sequentially selected at a first interval, and the selected frame is set and set as a first selected frame. A second step of displaying an image of a first selected frame together with an image of the reference frame; and comparing the first selected frame and the reference frame;
If the difference between the feature amounts of the images of the first selected frame and the reference frame is equal to or larger than a predetermined threshold, the position of the first selected frame is stored. If not, the process returns to the second step. A third step, and in a case where the position of the first selected frame is stored in the third step, the position of the first selected frame is stored as the first selected frame in the second step before the stored first selected frame. The frames after the set frame are sequentially selected at a second interval finer than the first interval, the selected frame is set as a second selected frame, and the image of the set second selected frame is currently displayed. A fourth step of displaying instead of the image of the first selected frame, and comparing the second selected frame and the reference frame,
If the difference between the feature amounts of the images of the second selected frame and the reference frame is equal to or greater than a predetermined threshold, the position of the second selected frame is stored. If not, the process returns to the fourth step. A fifth step, when the position of the second selected frame is stored in the fifth step, the second selected frame is set as the reference frame, and an image of the set reference frame is currently displayed. A sixth step of displaying in place of the image of the reference frame being set, and, when a new reference frame is set in the sixth step, in the video stream, the first interval from the set reference frame. It is determined whether or not a frame exists at the position advanced by one minute. If a frame exists, the process returns to the second step. If no frame exists, the last frame is replaced with the first selected frame. A seventh step of returning to the second step as a frame, and an eighth step of terminating the process when the reference frame is the last frame. The difference between the feature amounts of the reference frame and the first and second selected frames obtained in the fifth step is sequentially plotted, and the change in the feature amount of each frame in the video stream is displayed in a graph. And displaying a structure analysis process of a video stream. 5. The method according to claim 3, wherein, in the ninth step, a position of a second selected frame having a feature amount equal to or larger than a threshold value with respect to the reference frame is displayed in the graph. The method of displaying the structure analysis process of the described video stream. 6. The method according to claim 6, further comprising: a tenth step of displaying a graph indicating the progress of the analysis process in association with the position of each frame constituting the video stream. The tenth step further includes: Displaying, in the graph, a position of a frame selected as a first selected frame or a second selected frame and a position of a second selected frame having a feature amount equal to or larger than a threshold value with respect to the reference frame. The method for displaying a structure analysis process of a video stream according to any one of claims 3 to 5. 7. An eleventh step of recording the steps of the processing executed in the first to tenth steps, and reproducing the step of the processing recorded in the eleventh step at an arbitrary timing. The method according to any one of claims 3 to 6, further comprising a twelfth step of: 8. A video stream structure for generating video index information for managing the video information separately from the video information, and displaying a video stream structure of the video information based on the generated video index information. A display method, wherein the video index information includes at least one piece of segment information for managing a set of consecutive frames having a small difference in feature amount extracted from the video stream as a set of meaningful frames in one unit; Package information for managing a plurality of pieces of segment information together as at least a structural element object, the package information is arranged at a higher position, and the segment information managed by the package information is arranged below the package information. To display the structural element objects in a tree structure. Structure display method of the video stream, characterized by. 9. The segment information can manage package information for collectively managing a plurality of other segment information generated using a set of frames managed by the segment information, The package information is arranged and displayed below one piece of segment information, and the segment information managed by the package information is arranged and displayed below the package information in a tree structure. Item 9. The method for displaying a structure of a video stream according to Item 8. 10. The video stream according to claim 8, wherein at least one frame arbitrarily selected from a set of frames managed by the segment information is displayed as a thumbnail as a representative frame. Structure display method. 11. When a search process is performed for video information that matches a desired search condition from a plurality of video information using video index information created for each video information in advance, The structure of the video stream of the video information that matches the search condition is displayed in the tree structure based on the video index information of the video information that matches the search condition. 2. The method for displaying the structure of a video stream according to item 1. 12. When a structure of a video stream of video information that matches the search condition is displayed in a tree structure, a structural element object that matches the search condition is displayed while being distinguished from other component objects. The method for displaying a structure of a video stream according to claim 11, wherein: