JPH10112835A - Video image summarizing method and video image display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow the method to have provision for versatility of preference of the users and video image contents by extracting scenes consecutive for a prescribed time and grouping scenes in time series. SOLUTION: When a scene change detection means 105 discriminates occurrence of a scene change in a frame image received by a image fetch means 104, the means decides a representative image in all scene modes and a head frame. Then a time length of a scene just before a scene change is calculated and a time discrimination processing means 106 stores a preceding scene representative image as a representative image in the time discrimination mode to a file server 113. Then the user selects in which mode a summarized video image is to be displayed via a user interface means 116, a head frame number of a scene to be displayed and information of the representative image are sent from the server 113 to a summarized video image reproduction means 115, which displays the summarized video image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for supporting search, editing, processing, and quick viewing of images, and more particularly, to a method for summarizing images stored on a video tape or a video disk.
The present invention relates to a video summarizing method for reproducing or displaying and a video displaying method.

[0002]

2. Description of the Related Art In video editing and quick viewing, video summarizing means for efficiently searching for a desired portion from a video is indispensable. As a video summarizing means for selecting a specific representative image from images, for example, Japanese Patent Application Laid-Open
No. 4-68084 (hereinafter referred to as scene list display), rush reproduction method for each shot,
A method described in US Pat. No. 2,236,153 (hereinafter, referred to as a color difference correlation value method) and a method described in JP-A-6-149902 (hereinafter, referred to as a time length designation summarizing method) are known.

[0003] The scene list display is a method of examining scene changes in advance and displaying a list of images immediately after the scene changes as shown in FIG. In FIG. 14, (1) to (9) represent images immediately after a scene change. In this method, one scene is represented by the leading image of the scene.

[0004] A "scene" is a unit of video that is often used in the field of video editing and the like, and is often defined as a "portion of a video camera taken continuously in time". In the present application, “scene” is used in a broader sense, and “scene” refers to “a unit of an image formed by dividing an image according to some reference”. Scenes are sometimes called shots and cuts. The “scene change” often refers to a portion where the scene changes, such as a portion connected by editing or a portion where shooting by a video camera is interrupted. In this application, it shall be used in a broader sense, and "scene change"
It refers to “parts divided when dividing a video according to some criteria”. A scene change is sometimes called a cut. In addition, “frame” refers to each image constituting a video.

As a method of automatically dividing an image into scenes, a common color ratio method (Yamada, Fujioka, Kanamori, Matsushima, "Study Method for Scene Change Focusing on Common Color for Each Partial Area", The Institute of Television Engineers of Japan) Technical Report, Vol. 17, No. 55),
Video change model method (Yamada, Fujioka, Kanamori, Matsushima, Sakauchi,
"Scene change detection method for video including editing effect", The Institute of Television Engineers of Japan, Multimedia and Video Processing Symposium '94), etc. have been proposed.

[0006] The rush reproduction method for each shot is a video summarization method in which a head portion of each scene is reproduced one after another at a standard speed. This is a way to see a summary of the video,
Play from the beginning to the end without interrupting playback midway.

[0007] The time length designation summarization method is a method of reproducing all or a part of a scene at a fixed time, and uses the time length of each scene so that the time length of the summary moving image becomes a designated value. The playback time or the number of playback scenes for each scene is determined.

[0008] The color difference correlation value method is a video summarizing method that enables a more efficient video search than a scene list display by grouping scenes having similar contents while changing only the camera angle. In this method, a color difference histogram correlation value between cut images is obtained while obtaining a leading image of a scene (hereinafter, referred to as a cut image), and cut images having a color difference histogram correlation value equal to or more than a threshold value are cut images of the same group. Consider At this time, cut screens having similar backgrounds belong to the same group. When a list of cut screens is displayed, a cut screen first detected in the same group is set as a parent screen, and the remaining cut screens in the same group are displayed as sub-screens. For example, when the second to third scenes belong to the same group and the sixth to eighth scenes belong to the same group, they are displayed as shown in FIG. In FIG. 15, (1) to (9)
Represents the first to ninth cut screens, respectively.

A conventional video summarization system using the color difference correlation value method will be described below. FIG. 16 is a block diagram showing a conventional video summarizing system. In FIG. 16, reference numerals 1501 and 1502 denote video input devices.
Reference numeral 501 denotes a video disk device, and 1502 denotes a VTR. Reference numeral 1503 denotes a video disk device 1501 or VTR 1
A video summarizing apparatus for summarizing a video using a scene change detection and a color difference correlation value method while processing a video signal from a video signal 502.
504, a scene change detecting unit 1505 for detecting a scene change, and a group generating unit 1506 for performing grouping of the cut screen (the leading image of the scene). A control device 1507 controls the video disk device 1501 and the VTR 1502. 1508
Is a video compression device for compressing video. Reference numeral 1509 denotes the cut screen, scene change data and group data created by the video summarizing device 1503, and the video compression device 1
A file server that stores the video data compressed in step 508. Reference numeral 1510 denotes a video display device that displays data, images, and videos stored in the file server.

The overall operation of the video summarizing system configured as described above will be described with reference to the flowchart shown in FIG.

In step 1601, the control unit 1507 in FIG.
02 is started to reproduce the video, and at the same time, the video compression by the video compression device 1508 is started. The compressed video is stored in the file server 1509.

In step 1602, the control device 1507 determines whether or not the video has ended. If the video has ended, the procedure proceeds to step 1607; otherwise, the procedure 1603
Proceed to.

In step 1603, the image capturing means 15
04 captures the frame image being reproduced.

In step 1604, the scene change detecting means 1505 processes the frame image captured by the image capturing means 1504, and detects whether a scene change has occurred using the above-described common color ratio method or the like.

If it is determined in step 1604 that "a scene change has occurred", the flow advances to step 1605; otherwise, the flow returns to step 1602.

In step 1605, the frame number and the cut screen at the time when the scene change detected by the scene change detecting means 1505 occurs are stored in the file server 1509.

In step 1606, the group generation unit 15
06 calculates a color difference histogram correlation value between the time-series cut screens, and regards the cut screens that make the color difference histogram correlation value equal to or larger than the threshold value as cut screens of the same group. After saving the grouping result in the file server 1509, the process returns to the step 1602.

In step 1607, the control device 1507 stops the reproduction of the video and the compression of the video. In step 1608,
The user selects an image display method. If the user wants to view a representative portion as a moving image, the process proceeds to step 1610. If the user wants to search for a desired part using a still image, the process proceeds to step 1609.

In step 1609, the system causes the image display device 151 to allow the user to efficiently search for the desired portion.
A summary of the video is displayed on 0. For example, a list of cut screens is displayed as shown in FIG. Alternatively, a cut screen determined to be in the same group may be displayed in the form of a child screen by being linked to a cut screen first detected in the same group. When the second scene and the third scene are in the same group and the sixth to eighth scenes are in the same group, they are displayed as shown in FIG.

In step 1610, the system displays the head of each video group on the video display device 1510 for a predetermined time, for example, 5 seconds.

[0021]

Conventionally, a video summarizing apparatus characterized in that a specific representative image is selected from images is disclosed in which content of the video is important in one program. Despite the differences, the representative image is selected based on only one criterion. For example, in a video summarizing apparatus using the color difference correlation value method described in the section of "Prior Art", a program is only based on a video summarizing standard such that "scenes having similar contents differing only in camera angle are in the same group". The representative image was selected. However, in this method, as is often the case in news programs, important information such as a scene where a person appears even in the same background, such as a scene where an announcer explains the incident, etc. Cannot be selected as a representative image. This is because these images are all collected as long as the background is similar. In addition, in the same method, a digest created by extracting moving images of two or three scenes from each article of the main part, such as a satellite broadcast program "High Tech Shower International", flows first, and then the main part flows. The main part cannot be grouped so that the digest image described above becomes the representative image. This is because the similarity is determined only between adjacent scenes without distinguishing between the digest part and the main part.

There has been a problem not only with the video summarizing apparatus but also with a video display apparatus that displays video information including the summary information obtained by the video summarizing apparatus. That is, in a video display device based on a lash reproduction method for each shot or a time length designation summarization method for reproducing part or all of all scenes by some method, when the number of video scenes is large, the number of representative images increases, and Was a burden on the participant. Also, in a video display device that displays video information including summary information obtained by a video summary device that selects a representative image from images according to a predetermined criterion, the number of representative images increases when the number of video scenes is large. As a result, the number of images displayed in a list increases, and it is often difficult to find a desired portion. For example, when summarizing a news program, in many cases, representative images of dozens of scenes are displayed from each article. There was an issue that needed to be seen.

The video display device in the system described with reference to FIGS. 16 and 17 can search for a desired portion using still images, but cannot search for a desired portion using moving images and sounds. There was a problem that. Furthermore, in the case of the shot-by-shot rush reproduction method and the time length designation summarization method, not only can a part to be viewed using a still image cannot be searched, but also a scene that does not reach a certain time is reproduced.
Since the first scene of the next scene is included in the certain time, the next scene is played until the end of the certain time,
Thereafter, the next scene is reproduced again from the beginning for a certain period of time, so that there is a problem that the beginning video of the next scene is reproduced continuously and redundantly.

As described above, a video summarizing apparatus for uniformly selecting a representative image only based on a certain video summarization standard regardless of the content of the video, the user's preference, and the like, and displaying the representative image as the length of the video, etc. However, the video display device that displays the video data regardless of the user has a problem that it cannot respond to the user's various needs for the diversity and the summary depending on the content of the video and the like.

The present invention solves the above-mentioned problems of the prior art, and provides a video summarizing apparatus corresponding to a variety of video contents and a variety of user preferences and a video display apparatus for efficiently displaying summary information. The purpose is to provide.

[0026]

In order to solve the above-mentioned problems, an image summarizing apparatus for realizing an image summarizing method according to the present invention comprises a plurality of images obtained by dividing a captured image based on a predetermined standard. A plurality of video summarizing means are provided for grouping scenes into a plurality of time series groups or for selecting a predetermined scene from the plurality of scenes. The plurality of video summarizing means includes a means for detecting only a scene that continues for a certain time or more and extracting a representative image thereof, and a scene for a certain time or more for a result of grouping using a color difference correlation value method or the like. Means to extract representative images of each group after correcting so that they do not belong to the same group as the preceding and following scenes, and between each representative image of dozens of scenes including the digest of the video and each representative image of other scenes Means for calculating a similarity, detecting scenes in which the similarity is equal to or more than a threshold value, and extracting only a representative image thereof.The video summary information extracted by each means is stored in an internal or external recording. It is recorded in the means together with the video information itself. In the video display apparatus for realizing the video display method according to the present invention, a list of representative images, which are summary images of video, which is video summary information recorded in the recording unit, is displayed. When an image of a part is specified, the video after that part is reproduced. further,
The priorities of the video summarizing means and the priorities of image selection in each video summarizing means are determined in advance, and a representative image is determined using these priorities. Can be extracted. Further, when the number of representative images extracted by one video summarizing means is equal to or more than a threshold value, the representative image of the video summarizing means is selected and extracted from the video before the representative image by the other video summarizing means. Determine the image, if not, select an image to be extracted from the video by selecting from representative images of other video summarization means other than the video summarization means, and determine the number of images less than the number specified by the user. Can be extracted.

Further, instead of displaying the representative images in a list, a part of the video near the position of each representative image can be reproduced one after another (hereinafter, the reproduced video is referred to as a summary video). Then, the video before the summarization can be reproduced with an arbitrary frame on the summary video at the top. Also,
When the reproduction of the summary video is interrupted, a list of a plurality of representative images including a representative image representing the contents of the frame image at the interrupted position is displayed.

However, in the summary video, when a part of the video near the position of the representative image is included in a part of the video near the position of the immediately preceding representative image, one of the videos near the position of the immediately preceding representative image is included. The end of the set is the beginning of a part of the video near the position of the representative image.

[0029]

According to the first aspect of the present invention, a plurality of scenes formed by dividing a captured video by detecting a scene change of the captured video are determined in a time series by a predetermined number of scenes before and after the scene. A time-series group generation process of calculating the similarity of an image (hereinafter, a predetermined image of a scene is referred to as a representative image) to group scenes including the representative image into a time-series group. For a scene that continues for a certain period of time or more, a time series group adding step of making it a time series group independent of the preceding and following scenes, and video summary information of each time series group obtained in the above two steps are output. Is a video summarization method comprising the steps of outputting video summarization information. It has the effect that grouping.

According to a second aspect of the present invention, the method of calculating the similarity between representative images of scenes in the time series group generation process of the video summarizing method according to the first aspect is common to representative images to be compared. This is a video summarization method characterized by calculating the ratio of pixels having the same color, and extracts all scenes that last for a certain period of time or more, and groups time-series scenes based on the similarity regarding color. It has the action of:

According to a third aspect of the present invention, a predetermined image (hereinafter, referred to as a reference scene) of a plurality of scenes (hereinafter, referred to as reference scenes) selected according to a predetermined reference from a plurality of scenes constituting a captured image. A similarity between a predetermined image of a scene is referred to as a representative image) and representative images of all scenes constituting a video according to a predetermined standard, and the similarity with the representative image of the reference scene is equal to or greater than a threshold value And a video summary information output step of outputting video summary information of the scene obtained in the above process, which is a digest scene of the video. This has the effect of extracting the main part scene corresponding to the digest scene by obtaining the similarity between the highly likely time-series scene and all the scenes constituting the video.

According to a fourth aspect of the present invention, there is provided the video summarizing method according to the third aspect, wherein a similarity between the representative image of the reference scene and the representative image of the main scene is calculated in the similarity calculating step. Is that the representative image is divided into a plurality of image regions, and the RGB components of the average color of the pixels in each image region of both representative images are compared. By calculating the similarity by comparing the RGB components of the average color of the pixels in each image region of the representative image between the time-series scene having high probability, which is the digest scene of the video, and all the scenes constituting the video, And has the effect of extracting the main part scene corresponding to the digest scene.

According to a fifth aspect of the present invention, there are provided a plurality of video summarization processes for combining a plurality of scenes formed by dividing a captured video by detecting a scene change thereof into a plurality of time series groups, and / or Alternatively, a video summarization method comprising: a plurality of video summarization processes for selecting a predetermined scene from the plurality of scenes; and a video summary output process for outputting video summary information of each scene selected by each video summarization process. In addition, there is an effect that the user prepares a plurality of pieces of video summary information for determining the representative image according to the characteristics of the video.

According to a sixth aspect of the present invention, there is provided the video summarizing method according to the fifth aspect, wherein a plurality of video summarizing processes for combining a plurality of scenes into a plurality of time series groups and / or the plurality of scenes are performed. A plurality of video summarization processes for selecting a more predetermined scene include a video summarization process of selecting all scenes, and a video summarization process of selecting only a scene that continues for a predetermined time or more among the plurality of scenes of the captured video. The process and the similarity of a predetermined image of a scene before and after the scene in time series (hereinafter, a predetermined image of the scene is referred to as a representative image) are calculated according to a predetermined reference, and the similarity is equal to or larger than a threshold. A video summarization process for grouping scenes including images into a time-series group, a video summarization process according to claim 1 or 2, and a video summarization process according to claim 3 or 4. And characterized in that at least two or more video summarizing process, it has the effect of user to prepare a plurality of video summary information to determine a representative image in accordance with the characteristics or the like of the video.

According to a seventh aspect of the present invention, there is provided the video summarizing method according to the fifth or sixth aspect, wherein the video summarizing information is a predetermined image of a scene (hereinafter, a predetermined image of a scene is referred to as a representative image). .) Or a predetermined frame number (hereinafter, referred to as
It is called a representative frame number. ) In which the representative image or the representative frame number is selected in ascending order of the number of frames between the representative images or the representative frame numbers, and a predetermined number of scenes or time-series scenes are selected. .

The invention according to claim 8 is the video summarizing method according to claim 7, wherein when a predetermined number of scenes or time-series scenes cannot be selected by one image summarizing process, the remaining scenes are selected. Alternatively, it has an effect of selecting a time series scene or a scene selected by another video summarizing means or a time series scene.

According to a ninth aspect of the present invention, there is provided the video summarizing method according to the seventh or the eighth aspect, wherein a user selects the priority of the video summarizing process and the number of scenes or time series scenes to be selected. Is input, and video summarization is performed according to the information.

According to a tenth aspect of the present invention, in a video display method for displaying a video and a representative image of the video selected by any one of the first to ninth methods, the video near the position of the representative image is displayed. Video connecting a part of
It is called a summary video. This is a video display method characterized by reproducing a video with the frame specified above at the head, and has an effect that a portion to be viewed can be specified while taking into account sound and movement of a subject.

According to an eleventh aspect of the present invention, when reproduction of a video (hereinafter, referred to as a summary video) in which a part of a video near the position of an image extracted from a video is interrupted, a frame at the interrupted position is interrupted. A video display method characterized by displaying a list of a plurality of representative images including representative images representing the contents of an image, and has an effect of intensively displaying information of a part desired by a user.

According to a twelfth aspect of the present invention, there is provided the video display method according to the tenth or eleventh aspect, wherein a part of the head of the summary video is included in a part of the tail of the immediately preceding summary video. Has the effect of setting the frame next to the last frame included in a part of the end of the previous summary video in the summary video as the head of the summary video. This method can avoid duplicative reproduction of the once reproduced summary video without a pause.

According to a thirteenth aspect of the present invention, a plurality of scenes formed by dividing a captured video based on a predetermined criterion are grouped into a plurality of time-series groups to extract video summary information. And / or a plurality of video summarization means for extracting video summary information by selecting a predetermined scene from the plurality of scenes, and the one or more video summarization means. A video summarizing system comprising a summary information selecting means and a summary information displaying means for selecting the summarized information, which has an effect that it is possible to freely select on which basis the summarized information is to be displayed.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. (First Embodiment) FIG. 1 shows a first embodiment, in which a video summarization apparatus for extracting summary information from a video based on a plurality of video summarization criteria, and a video summary information selected by the apparatus. FIG. 2 is a block diagram showing a video summarization system configured by combining a video display device that displays the video itself.

In FIG. 1, 101 and 102 are video output devices, 101 is a video disk device, and 102 is a VTR. 103 is a video disk device 101 or V
A video summarizing apparatus for summarizing a video while processing a video signal from a TR 102, comprising: an image capturing unit 104 for capturing a frame image; a scene change detecting unit 105 for detecting a scene change; Time determination processing means 106 for detecting, and group generation means 10 for performing grouping of representative images of scenes
7, time determination processing means 106 and group generation means 10
7, the group addition means 108 for correcting the grouping result in response to the output of the image 7, the image similarity calculation means 109 for calculating the similarity between the representative images of the scenes, and the similarity calculated by the similarity calculation means. It comprises an image reference processing means 110 for detecting a scene having a threshold value or more. A control device 111 controls the video disk device 101 and the VTR 102. Reference numeral 112 denotes a video compression device that compresses a video. Reference numeral 113 denotes a file server that stores video data compressed by the video compression device 112, data of various scenes and groups detected by the video summarization device 103, and respective representative images. Reference numeral 114 denotes a video display device for displaying a video using data stored in the file server 113. The video summary device 1 reproduces videos near the position of the image extracted by the video summary device one after another for a certain period of time.
15, a user interface means 116 for controlling the reproduction of the video, and a video reproduction means 117 for reproducing the video after the designated position.

The image summarizing device 103 of the image summarizing system
Can be realized, for example, by a combination of computer hardware and software. Also,
Of the video display device 114, the summary video playback unit 115 and the video playback unit 117 can be realized on a computer, and the user interface unit 116 is a CR
It can be realized by a combination of a monitor such as T and computer hardware and software.

The video summarizing apparatus 103 according to the present embodiment has four video summarization standards. The first video summarization criterion is a video summarization criterion in which the top screen of all scenes including a video is a representative image. Hereinafter, this video summarization standard is referred to as an all scene display mode. The second video summarization criterion is a video summarization criterion in which a top screen of a scene having a time length equal to or longer than a certain value is used as a representative image. Hereinafter, this video summarization reference is referred to as a time determination mode. The third video summarization criterion is that scenes other than the scene selected in the time determination mode are grouped by the color difference correlation method, and the scene selected in the time determination mode and the top screen of the grouped scenes are defined as a representative image. This is the video summarization criterion. Hereinafter, this video summarization standard is referred to as a hybrid mode. The fourth video summarization criterion is that a representative image of a portion including the digest of a video, such as a specific news program, which is known in advance that the digest of the video is at the beginning of the program and the top screen of other scenes This is a video summarization criterion in which the similarity between pixels is calculated by comparing RGB components of pixels in a plurality of image areas, and the top screen of a scene having a value equal to or greater than a predetermined threshold value is used as a representative image. Hereinafter, this video summarization standard is referred to as an image standard mode.

The operation of the video summarizing system configured as described above will be described with reference to the flowchart shown in FIG.

In step 201, the control device 1 shown in FIG.
11 controls the video disk device 101 and the VTR 102 to start video playback, and at the same time,
Start compressing video in.

In step 202, the control device 111 determines whether or not the image has been completed. If the video has ended, the procedure proceeds to step 212; otherwise, the procedure proceeds to step 203.

Steps 203 to 211 are processes performed in the video summarizing apparatus 103.

In step 203, the image capturing means 104
Captures the frame image being played.

In step 204, the scene change detecting means 105 processes the frame image captured by the image capturing means 104, and detects a scene change using the common color ratio method described in the section of "Prior Art". Determine if it has occurred. However, in the common color ratio method, since a scene change is detected after processing two frame images of the next scene, the determination that “a scene change has occurred” is “the previously captured frame image is a cut screen”. It represents that.

FIG. 3 is an explanatory diagram showing a method of detecting a scene change by the scene change detecting means 105 just described and a previous scene. In FIG. 3, a is the frame image captured in step 203, and b is the previous step 203.
Is the frame image captured in step 2, and c is the procedure 2 before
03 is the frame image fetched, and d is the frame image fetched in the previous procedure 203 three times. in this case,
Although a scene change has occurred between b and c, it indicates that it is detected after a is taken. The scene immediately before the detected scene change is called a previous scene.

The scene change may be detected by using a video change model method or the like. Alternatively, the operator may determine a scene change in advance and input a frame number at the time when the scene change occurs.

In step 204, the scene change detecting means 10
If it is determined that the scene change has occurred, the process proceeds to step 205; otherwise, the process returns to step 202.

In step 205, the representative screen and the first frame of the scene in the all scene mode are determined. Specifically, the image captured in step 204 and the frame number at the time when the scene change occurs are stored in the file server 113 as the representative screen and the top frame of the scene in all scene modes. In addition, the image is transferred to the file server 1
In the case where the data is stored in the file 13, the data may be reduced and stored.

In step 206, the representative screen and the first frame of the scene in the time determination mode are determined. Specifically, after calculating the time length of the scene immediately before the scene change detected in step 204 (hereinafter referred to as the previous scene),
The time determination processing means 106 detects a previous scene having a time length of 8 seconds or more, and stores a representative image of the detected previous scene in the file server 113 as a representative image in the time determination mode. The head frame number of the previous scene detected by the time determination processing means 106 is stored in the file server as the head frame number of the scene to be displayed in the time determination mode.

In the present embodiment, the previous scene of 8 seconds or more is detected, but it is not always necessary to be 8 seconds.

In steps 207 to 209, the representative screen and the first frame of the scene in the hybrid mode are determined.

In step 207, the group generation means 107
However, the representative image of the scene immediately after the scene change (hereinafter, referred to as the next scene) detected in step 204 and the representative image of the previous scene are extracted from the file server 113, and the color difference histogram correlation value between them is obtained. If the value is equal to or larger than the threshold value, the representative image of the next scene is regarded as the representative image of the hybrid mode and stored in the file server 113. If the color difference histogram correlation value is equal to or greater than the threshold value, the first frame number of the next scene is stored in the file server 113 as the first frame number of the scene to be displayed in the hybrid mode.

Step 208 is a branching process. If the time length of the previous scene is 8 seconds or more, proceed to step 209,
Otherwise, proceed to step 210.

In step 209, the group adding means 108
Saves the representative image of the previous scene and the representative image of the next scene having a time length of 8 seconds or more as the representative image of the hybrid mode in the file server 113. Further, the head frame number of the previous scene and the head frame number of the next scene are stored in the file server 113 as the head frame number of the scene to be displayed in the hybrid mode.

In steps 210 to 211, the representative screen and the first frame of the scene in the image reference mode are determined.

In step 210, the representative image I _N of the next scene and the representative image I of the first 20 scenes of the video obtained by the scene change detecting means 105 in step 204.
_M (M is a natural number from 1 to 20) S (M,
N) is calculated by the image similarity calculation means 110. Various methods can be considered as a method of calculating the similarity. An example is shown in FIG. FIG. 4 is an explanatory diagram showing that the screen is divided into blocks when calculating the color similarity between the representative image of the portion including the digest and other scenes in the image reference mode. That is, the representative image I _M
Is divided into 4 × 4 blocks as shown in FIG. 4, and the RGB components R _i (M) of the average color of the pixels in the i-th block are
G _i (M), B _i and (M) may be calculated by the following equation using.

[0064]

(Equation 1)

In equation (1), | x | represents the absolute value of x. Further, the color difference histogram correlation value described in the conventional example may be used as the similarity.

In the above description, the representative image of the first 20 scenes of the video is used as an image for calculating the degree of similarity with the representative image of the next scene. However, this is changed depending on the structure of the program. You can also do so. Specifically, the image similarity calculation means 109 by a command from the user interface unit 116 of video display device 114 may be pick a representative image I _N follows a scene from the portion not at the beginning of the video. For example, when there is a digest at the end of the video, a plurality of scenes at the end of the video are used.

In step 211, the image reference processing means 110
Then, when M takes any value from 1 to 20,

[0068]

(Equation 2)

It is checked whether or not the following holds. When (Equation 2) holds, the first frame number of the next scene is stored in the file server 113 as the first frame number of the scene to be displayed in the image reference mode, and the representative image of the next scene is stored. Is regarded as a representative image in the image reference mode. In (Equation 2), θ
_SIM is a preset threshold value.

After the end of step 211, the process returns to step 202.
In step 212, the reproduction of the video and the compression of the video are stopped.

With the above procedure, the file server 113
Contains several video summarization criteria: full scene mode,
The summary information extracted in the time determination mode, the hybrid mode, and the image reference mode, that is, the representative image and the top frame number are stored. Procedure 213 below
Steps 215 to 215 are processes performed in the display device 114.

In step 213, the user selects which mode to display the summary video through the user interface means 116. The options are the time determination mode, the hybrid mode, the image reference mode, and the all scene display mode, as described above. The user interface unit 116 outputs a signal to the file server 113 to send the representative image of the scene in the selected mode and the information of the first frame to the summary video reproducing unit 115.

In step 214, the head frame number of the scene to be displayed in the mode selected in step 213 and the information of the representative image are sent from the file server 113 to the summary video reproducing means 115. Based on this information,
The summary video reproducing means 115 displays the summary video. Here, a case where the user selects the hybrid mode will be described. FIG. 5 is an explanatory diagram showing a method of creating a summary video according to the first embodiment of the present invention. Of the three rows of video information, the upper row shows the video itself, the middle row shows the summary video for 5 seconds extracted from the upper row video, and the lower row shows that only the middle summary video is reproduced continuously. That is,
As shown in the lower part of FIG. 5, the video data for 5 seconds after the head of the scene of each representative image in the hybrid mode is taken out from the file server 113 and reproduced one after another at a standard speed.

Note that the length of the video data after the head of the scene of each representative image does not have to be 5 seconds. Also, instead of the five seconds after the beginning of the scene of each representative image, the video before and after the position of each representative image may be played back one after another, such as "play back five seconds from two seconds before each representative image". . Also,
Instead of playback at the standard speed, playback may be performed at fast forward. Further, the fast-forward speed may be changed according to the video content.
Further, the five seconds after the head of the scene of each representative image may include a plurality of scenes.

The user operates the user interface means 1
While displaying the summary video using the search button 16, search for a desired part. Control buttons for the summary video include play, reverse play,
Still, fast-forward, rewind, one-frame forward, one-frame reverse, and the like can be considered.

In step 215, the user designates the head of the part to be viewed while viewing the summary video. FIG. 6 is an explanatory diagram showing a playback part of the summary video. In other words, when the user specifies the scene by the user interface means 116 at the point of time when the user views the top of the representative image of the i-th scene for 2 seconds, the reproduction start time of the video by the video reproducing means 117 is " At the time when two seconds have elapsed from the head of the representative image ".

FIG. 7 shows an example of a screen of the user interface 116 for designating a part to be viewed. In FIG. 7, a is a portion for displaying the summary video, b is a button for controlling the summary video, and c is a button for specifying a portion to be viewed. As soon as the button for designating the part to be viewed is pressed, the image displayed in a is regarded as the head of the part to be viewed. Therefore, keep the summary video still at the part you want to see,
When the button c is pressed, the head of the part to be surely viewed can be designated.

In the above description, the representative image of the head scene of the group is set as the representative image in the hybrid mode. However, the representative image may be selected by another method. For example, a representative image of a scene having the longest time length in each group may be regarded as a representative image in the hybrid mode. Further, in the present embodiment, four modes to be selected are prepared, but the number may not be four. The order in which the representative images are obtained is also arbitrary, and the representative images do not necessarily have to be determined in the order of the all scene display mode, the time determination mode, the hybrid mode, and the image reference mode.

In step 214, when the summary video is frozen, a list of representative images before and after the representative image immediately before the still position may be displayed. Then, in step 215, one of the displayed representative images may be designated as the head of a part to be viewed. The number of representative images displayed at the same time is
For example, the number of sheets may be 24.

In step 214, the video data for 5 seconds after the head of the scene of each representative image is extracted.
The leading frame of the scene of the representative image may be included in the five seconds after the beginning of the scene of the immediately preceding representative image. FIG. 8 is an explanatory diagram showing a playback portion of the summary video in such a situation. As described above, when the extracted video data is successively reproduced, the portion A in FIG. 8 is reproduced twice. Therefore, as shown in the figure, when the “first frame of the scene of the representative image” is included in the “5 seconds after the beginning of the scene of the previous representative image”, instead of the head of the scene of the representative image, (For example, FIG. 8
May be reproduced immediately after (A part).

(Second Embodiment) FIG. 9 shows a second embodiment, in which summary information is extracted from a video based on a plurality of video summarization criteria, and the number of representative images less than or equal to the number specified by the user. FIG. 1 is a block diagram showing a video summarization system configured by combining a video summarization device extracting a video and a video display device that displays summary information of a video selected by the device and a video itself.

In FIG. 9, reference numerals 801 and 802 denote video output devices, 801 denotes a video disk device, and 802 denotes a VTR. 803 is a video disk device 801 or V
A video summarizing device for extracting a number of images less than or equal to a number specified by a user while processing a video signal from the TR 802 and summarizing the video, an image capturing means 804 for capturing a frame image, and a scene for detecting a scene change. A change detection unit 805, a time determination processing unit 806 that detects a scene that continues for a predetermined time or more, a group generation unit 807 that performs grouping of representative images of the scene,
Group addition means 808 for correcting the result of grouping in response to the output of the time determination processing means 806 and group generation means 807, image similarity calculation means 809 for calculating the similarity between representative images of scenes, The output of the image reference processing means 810 for detecting a scene in which the similarity obtained by the calculation means is equal to or more than the threshold value, the scene change detection means 805, the time determination processing means 806, the group addition means 808, and the image reference processing means 810 It comprises image determining means 811 for receiving and determining a representative image. A control device 812 controls the video disk device 801 and the VTR 802. Reference numeral 813 denotes a video compression device that compresses a video. A file server 814 stores the representative image determined by the video summarizing device 803 and its frame number.
An image display device 815 displays data, images, and images stored in the file server 814.

An image summarizing device 803 of the image summarizing system
Can be realized on a computer, for example.
The video display device 815 can be realized by, for example, a combination of a computer, a CRT, and a monitor.

The video summarizing apparatus according to the present embodiment has four video summarization standards. Since they are the same as those described in the first embodiment, description thereof will be omitted.

The operation of the video summarizing system configured as described above will be described with reference to the flowchart shown in FIG.

In step 901, similarly to step 201, the control device 812 in FIG.
And the VTR 802 to start video reproduction, and at the same time, start video compression by the video compression device 813.

In step 902, as in step 202, it is determined whether or not the video has been completed. If the video has ended, the process proceeds to step 912; otherwise, the process proceeds to step 903.

In step 903, similarly to step 203, the image capturing means 804 captures the frame image being reproduced.

In step 904, as in step 204, the scene change detecting means 805 determines whether the scene
In step 4, the frame image captured is processed to determine whether a scene change has occurred using a common color ratio method or the like.

If it is determined in step 904 that "a scene change has occurred", the flow advances to step 905; otherwise, the flow returns to step 902.

In step 905, a representative screen and a top frame of a scene in the all scene mode are determined. That is, the image captured in step 904 and the frame number of the image are stored in the file server 814. The saved image is used as a representative image of the scene. Also, the representative image of the scene is regarded as a representative image in the all scene display mode.

In step 906, the representative screen and the first frame of the scene in the time determination mode are determined. That is, as in step 206, the time length of the scene immediately before the scene change detected in step 904 (hereinafter referred to as the previous scene) is calculated, and then the time determination processing unit 806 sets the time length to 8
A previous scene that is longer than a second is detected, and a representative image of the detected previous scene is regarded as a representative image in the time determination mode.

In steps 907 to 909, the representative screen and the first frame of the scene in the hybrid mode are determined.

In step 907, similarly to step 207, the group generation means 807 calculates the color difference histogram correlation value between the scene immediately after the scene change detected in step 904 (hereinafter referred to as the next scene) and the representative image of the previous scene. Asked,
If the color difference histogram correlation value exceeds the threshold,
The representative image of the next scene is regarded as the representative image of the hybrid mode and stored in the file server 814. Instead of using the color difference histogram, time-series scene grouping may be performed using the common pixel method, and the representative image of the head scene of the group may be regarded as the representative image in the hybrid mode. The common pixel method will be described after the operation of the video summarizing apparatus according to the present embodiment is described.

In step 908, a branching process is executed as in step 208. If the time length of the previous scene is 8 seconds or more, the process proceeds to step 909; otherwise, the process proceeds to step 910.
Proceed to.

In step 909, as in step 209, the group adding means 808 regards the representative image of the previous scene and the representative image of the next scene as a representative image in the hybrid mode, and stores the representative image in the file server 814.

In steps 210 to 211, a representative screen and a head frame of a scene in the image reference mode are determined.

[0098] In step 910, similarly to the procedure 210, the similarity S between the representative image I _M of the head 20 scene representative image I _N and the video of the next scene (natural number of M is from 1 to 20) ( M, N) is calculated by the image similarity calculating means 809.

In step 911, similarly to step 211, the image reference processing means 810 checks whether or not the expression (2) is satisfied when M takes any value from 1 to 20, and the expression (2) Is established, the representative image of the next scene is regarded as the representative image in the image reference mode. After the end of the procedure 911, the process returns to the procedure 902.

In step 912, as in step 212, video reproduction and video compression are stopped. In the procedure up to this point, the file server 814 stores a plurality of video summary standards, that is, summary information extracted by the all scene mode, the time determination mode, the hybrid mode, and the image reference mode, that is, the representative image and the first frame number. Was done. The following steps 913 to 918 are processes performed in the video display device 815.

In steps 913 to 916, the representative images, which are the summary information of the video, are selected up to a predetermined number. Hereinafter, a case will be described in which the number of representative images extracted by the video summarizing apparatus is set to 24.

In step 913, the image determining means 811
Based on the priority determined in advance, the mode in which the representative image is extracted is determined. Hereinafter, a case where the mode with the highest priority is the image reference mode, the second highest mode is the time determination mode, the third highest mode is the hybrid mode, and the fourth highest mode is the all scene display mode State. It should be noted that this priority can be set freely by providing an interface unit that allows the user to send a command to the image determination unit 811.

First, a representative image is determined by the image reference mode having the highest priority. When the procedure returns to the procedure 913 by the branching process of the procedure 916 after the representative image is determined in the image reference mode, the representative image is determined in the mode having one lower priority than the previously determined mode.

In step 914, the representative images in the mode determined in step 913 are ordered based on the priority of image selection. For example, among the representative images determined in the procedure 913, younger numbers are assigned in order from the representative image of a scene having a longer time length. The representative image having the longer length up to the next representative image in the same mode may be numbered in ascending order.

In step 915, the images are regarded as representative images extracted by the video summarizing apparatus in ascending order of numbers. However, when the number of representative images extracted by the video summarizing apparatus exceeds 24, the operation is interrupted.

Step 916 is a branching process. If the number of representative images already determined to be extracted by the video summarizing apparatus is less than 24, the procedure returns to step 913. Otherwise, go to step 917.

In the above description, the number of representative images extracted by the video summarizing apparatus is set to 24, but the number is not necessarily limited to 24. If the number of representative images is less than 24 in all modes, the process may enter an infinite loop in step 916. Therefore, when the number of scenes becomes less than 24 even after all the modes are selected, the number of representative images at that time may be made equal to the number of scenes.

In step 917, the system displays a summary of the video on the video display device so that the user can efficiently search for a desired portion. For example, a list of the representative images determined in step 915 is displayed.

In step 918, a part desired by the user is specified. For example, using a pointing device such as a mouse, a representative image of a desired portion is specified. The video display device receives the video data from the file server and reproduces the video from the specified portion.

In the above description, a list of representative images is displayed in step 917. However, as described in step 214 of the first embodiment, a summary video may be displayed using the summary video reproducing means. Further, an image display means for displaying a list of representative images may be provided and incorporated in the video summarizing apparatus 803. In this case, in step 917, the system displays the video summary using the image display means.

In the above description, only representative images of a set number or less are displayed in a list. However, a list display of representative images in each of the time determination mode, the hybrid mode, and the image reference mode can be selected. Is also good.

In the following, grouping by the common pixel method which can be employed when grouping time-series scenes in step 907 in the video summarizing system of the present embodiment will be described. The common pixel method has been disclosed by the applicant in Japanese Patent Application No. 7-46970.

The common pixel method focuses on the color common to the scenes, and has a similar background scene such as a scene taken on a sandy beach or a bust shot of various persons (the upper part is reflected from the chest of the person). This is a method of detecting a case where scenes of similar subjects are continuous in a time series, such as a case where scenes continue, and integrating them into one group.

A scene is a minimum unit of contents. Therefore,
Frame images in one scene have common features such as "the same person appears". Therefore, the moving image at the head of each scene is regarded as representing a scene, and this moving image is referred to as a representative spatiotemporal image.

If the "pixels having the same color" are regarded as the same object, there is a problem that the scenes of the "objects of the same color moving differently" are grouped. In order to solve this problem, the same color object is regarded as the same object only when the same color object is stationary in two different scenes and when the same color object is moving together in two different scenes. I reckon. That is,
“Pixels having the same color as the presence or absence of motion” are regarded as the same object. Also, scenes in the same group have common color ratio conditions,
That is, in the representative spatiotemporal image of each scene, the value obtained by normalizing the total number of pixels of “the same object appearing in the representative spatiotemporal image of the scenes in the group” by the total number of pixels is equal to or greater than the threshold θ _SHOT . Suppose that the condition is satisfied.

FIG. 11 is an explanatory diagram showing the state of the common color ratio condition. As shown in FIG. 11, only the A (background) portion appears in the representative spatiotemporal images of the scenes SC _{1 to} SC ₄ in common, and the ratio of the number of pixels of the A portion is different from that of the scene SC ₄ . When the threshold θ _{SHOT is} exceeded, the scene S
C _{1 to} SC ₃ satisfy the common color ratio condition, but the scene SC
_{1 ~SC 4} do not satisfy the common color ratio conditions. SC _{3 to} SC ₄
Whether or not satisfies the common color ratio condition is indeterminate. In the example of FIG. 11, SC ₃ to SC ₄ of A (background) and B "same object appearing in common to the representative spatiotemporal image of the scene SC ₃ to SC _4" (rectangular object) portion of next, the pixel 100% in both scenes SC ₃ and SC ₄
Since the scene SC ₃ to SC ₄ satisfy the common color ratio conditions.

At this time, if scenes satisfying the common color ratio condition are regarded as the same group, scenes SC _{1 to} SC ₃ and scenes SC _{3 to} SC ₄ belong to the same group.
Although C _{1 to} SC ₄ should belong to the same group, scenes SC _{1 to} SC ₄ do not satisfy the common color ratio condition, and contradictions occur. Therefore, even if the common color ratio condition is satisfied, the scenes are not necessarily in the same group.

In the common pixel method, it is assumed that the similarity between adjacent scenes in the same group has a larger value than the similarity between scenes in different groups. Find the boundary. However, M = 1 and the initial value of L is 1.

In the procedure 1, it is determined whether or not the scenes SC _{M to} SC _{M + L} satisfy the common color ratio condition. If satisfied, the determination is repeated while adding 1 to L until the common color ratio condition is no longer satisfied. In the example of FIG. 11, the process proceeds to the procedure 2 when L = 4.

In the procedure 2, for each of the scenes SC _{M to} SC _{M + L-1} satisfying the common color ratio condition, the similarity with the next scene is obtained, and the portion having the minimum similarity is defined as the group boundary. I reckon. FIG. 12 is an explanatory diagram showing a method of determining a group boundary. In the figure, the similarity of other similarity SC ₂ and SC _3, i.e., SC of ₁ and smaller than the similarity of the SC similarity and SC ₃ of ₂ and SC _4, SC ₂ and SC ₃
Is regarded as a group boundary.

Note that a representative image may be used instead of the representative spatiotemporal image. In this case, "pixels having a common color" may be regarded as the same object.

(Third Embodiment) In a third embodiment, in the video summarizing system described in the second embodiment, selection of a mode from which to extract a representative image is performed according to the second embodiment. Instead of determining the mode based on a predetermined priority as in the procedure 913 of the embodiment, the mode is determined by analyzing the video and determining the priority of the mode.

The video summarizing system shown in FIG.
The operation will be described with reference to the flowchart shown in FIG.

The operations in steps 1201 to 1212 are the same as the operations in steps 901 to 912 shown in FIG. 10, and a description thereof will be omitted.

At step 1213, the image judging means 811 determines the priority of the mode. Hereinafter, when the number of representative images in the image reference mode becomes 5 or more, the mode with the highest priority is set to the image reference mode, the second highest mode is set to the time determination mode, and the third highest mode is set to the hybrid mode. When the number of representative images in the image reference mode is less than 5, the mode with the highest priority is the time determination mode, and the second highest mode is the hybrid mode. An example in which the third highest mode is set to the all scene display mode will be described. Here, it is checked whether or not the number of the representative images in the image reference mode is five or more, and the priority of the mode is changed. However, it is not always necessary to set the number to five. The automatic criterion for determining the mode as in this example is based on the presumption that if the number of summary videos in the image reference mode is five or less, it is highly likely that the video to be summarized does not include a headline. ing. Note that such a mode automatic determination criterion may be configured such that the user can send an instruction to the image determination unit 811 by providing an interface unit and freely change the setting.

In step 1214, the image determining means 811 determines the mode based on the priority determined in step 1213. First, a representative image is determined based on the image reference mode having the highest priority. When the process returns to the procedure 1214 by the branching process of the procedure 1217 after the representative image is determined in the image reference mode, the representative image is determined in a mode having one priority lower than the mode determined last time.

In the following, a case where the number of representative images extracted by the video summarizing apparatus is set to 24 will be described.

In step 1215, similar to step 914,
The representative images in the mode determined in step 1214 are ordered based on the image selection priority. For example, Procedure 1
In the representative images determined in 214, the representative images of scenes having a longer time length are numbered in ascending order.

In step 1216, similar to step 915,
In order from the youngest number, it is regarded as a representative image extracted by the video summarizing apparatus. However, when the number of representative images extracted by the video summarizing apparatus exceeds 24, the operation is interrupted.

At step 1217, the same branch processing as at step 916 is executed. If the number of representative images already determined to be extracted by the video summarization device is less than 24,
The procedure returns to step 1214. Otherwise, go to step 1218.

In step 1218, similar to step 917,
The system displays a summary of the image on the image display device so that the user can efficiently search for a desired portion. For example, a list of representative images determined in step 1216 is displayed.

At step 1219, similar to step 918,
Specify the part that the user wants to see. For example, using a pointing device such as a mouse, a representative image of a desired portion is specified. The video display device receives the video data from the file server and reproduces the video from the specified portion.

[0133]

The present invention described above has the following advantageous effects. For this reason, the conventional image summarizing apparatus and the conventional image summarizing apparatus for uniformly selecting a representative image only based on a certain image summarizing standard are used. Compared to a conventional video display device that selects irrespective of length or the like, it is possible to cope with a variety of video contents and a variety of user preferences. (1) The video summarizing apparatus according to the first and second aspects of the present invention has the same background as the scenes before and after the scene, such as a scene explaining an incident by an announcer at the site, as often occurs in a news program, for example. Even if there is a scene with a high possibility of having important information for a certain period of time or more, a scene with a high possibility of having important information such as the scene and the scene immediately after the scene It can be extracted from scenes before and after having similar backgrounds. Scenes longer than the grouping determination time (8 seconds in the embodiment of the present specification) include scenes of an announcer of a news program, scenes of interviews, scenes of explanation by characters, scenes of flips, and the like. is there. According to our analysis, the scene immediately following the long scene is often as important as the long scene.

For example, when a case is reported from the site in a news program, a video of a "scene related to the case" flows after "a scene in which characters dispatched to the site explain the background of the case". . In this case, the image of the "scene related to the incident" more accurately represents the content of the incident than the image of "the scene where the characters dispatched to the scene explain the background of the incident," is there. However, in the representative image determination method using the color difference histogram correlation, since the scene is the same background of the scene, the "scene related to the incident" and "the scene where the characters dispatched to the scene explain the background of the incident" In the same group, there is a problem that the image of the "scene related to the incident" does not become the representative image. Also, even in the time determination mode, the “scene related to the incident” does not always become 8 seconds or more, and thus does not always become the representative image.

On the other hand, in the hybrid mode, a scene longer than 8 seconds and a scene immediately after it are representative images.
The image of the “scene related to the incident” is selected as the representative image, and an effect that is not in the representative image determination method using the color difference histogram correlation and the time determination mode can be obtained. Of the scenes with a long duration, the contents of an announcer scene, an interview scene, and a character explanation scene cannot be understood unless they are displayed together with the immediately following scene. When the representative image determination method using the color difference histogram correlation is used, scenes before and after a scene with a long time length are often displayed, but when the threshold is inappropriate or the background is similar, The next scene is not displayed.
However, in the hybrid mode, there is an effect that the scene immediately after is surely displayed. Therefore, the hybrid mode has an advantage in summarizing such a type of program as compared with the color difference correlation value method and the time determination mode. (2) The video summarizing apparatus based on the invention described in claims 3 and 4 is a program in which a digest created by extracting moving images of two or three scenes from each article of the main part flows first, and then the main part flows. Thus, the main part can be grouped so that these digest images become representative images. This is because the representative image is determined by comparing the digest part and the main part. That is, the image reference mode has a tremendous effect on such types of programs. Therefore, in the image reference mode, the contents of the video are represented by the scene of the digest portion, and the portion to be viewed can be easily searched.

[0136] Further, by using a video summarizing system in combination with a video display device, when a scene to be viewed is specified, the video subsequent to that scene in the main part can be reproduced, and the article to be viewed can be easily reproduced. This has an effect that the conventional video summarization system does not have. When the representative image is hardly selected in the image reference mode, a portion (for example, a headline such as news) for introducing the content of the video in the video in a summary manner
Will not exist. Therefore, this mode has an effect that it is possible to determine whether or not summary information is attached to a video. (3) Since the video summarizing apparatus based on the invention described in claim 5, claim 6, claim 7, claim 8, claim 9, and claim 13 includes a plurality of video summarization standards, As compared with a conventional video summarizing apparatus that summarizes video according to one video summarization standard, it is possible to appropriately cope with the diversity of video content. The user can freely change or combine the video summarization standards depending on the program or the like. (4) In the video summarizing apparatus based on the invention described in claim 7, claim 8, and claim 9, the user can specify the upper limit of the number of representative images, so that the number of representative images can be reduced. You can easily find the part you want to see compared to the conventional method. Further, when the priorities of the modes are determined in advance, the user can determine the representative image according to the characteristics of the video. In addition, when the method of determining a representative image is switched depending on whether the number of representative images in the image reference mode is equal to or greater than a threshold value, the digest image can be preferentially extracted from a video in which a digest exists. It also has the effect. Therefore, the efficiency with which the user searches for the summary video is improved as compared with the conventional video summary system. (5) The video display device based on the invention described in claim 10 utilizes the information of the video and the information of the file server that stores the summary information of the video for a certain period of time around the position of each representative image. A desired part can be specified while watching the video one after another, that is, a desired part can be searched using a moving image and a sound. Therefore, a desired part can be specified while taking into account sound and movement of a subject. In addition, since representative images are selected from scenes with high importance,
The user does not feel redundant when watching the reproduced video. (6) The video summarizing apparatus based on the invention described in claim 11 and claim 12, when reproducing the representative image of the video and the summary video consisting of the preceding and succeeding images, while eliminating the duplicated portion. Since the summary video is reproduced, there is no redundant reproduction. Furthermore, when the reproduction is interrupted at a desired portion while watching the video for a fixed time in the vicinity of the position of each representative image one after another, a plurality of representative images including the representative image representing the contents of the frame image at the interrupted position are listed. Since the display is performed, it is possible to easily search for “contents related to the part to be viewed” existing near the interrupted position.

[Brief description of the drawings]

FIG. 1 is a block diagram of a video summarizing system according to a first embodiment of the present invention.

FIG. 2 is a flowchart showing the operation of the video summarizing system according to the first embodiment of the present invention.

FIG. 3 is an explanatory diagram showing a method of detecting a scene change by a scene change detecting means and a previous scene.

FIG. 4 is an explanatory diagram showing that a screen is divided into blocks in an image reference mode.

FIG. 5 is an explanatory diagram showing a method of creating a summary video according to the first embodiment of the present invention.

FIG. 6 is an explanatory diagram showing a state of starting reproduction from a head of a part to be viewed according to the first embodiment of the present invention;

FIG. 7 is an explanatory diagram showing a screen for designating a portion to be viewed via a user interface unit according to the first embodiment of the present invention;

FIG. 8 is an explanatory diagram showing a playback part of a summary video according to the first embodiment of the present invention.

FIG. 9 is a block diagram of a video summarizing system according to a second embodiment of the present invention.

FIG. 10 is a flowchart showing the operation of the video display system according to the second embodiment of the present invention.

FIG. 11 is an explanatory diagram showing a state of a common color ratio condition according to the second embodiment of the present invention.

FIG. 12 is an explanatory diagram showing a method for determining a group boundary according to the second embodiment of the present invention;

FIG. 13 is a flowchart showing the operation of the video display system according to the third embodiment of the present invention.

FIG. 14 is an explanatory diagram showing a scene list display in a conventional video display device.

FIG. 15 is an explanatory diagram showing a list display of video summary results in a conventional video display device.

FIG. 16 is a block diagram of a conventional video summarizing system.

FIG. 17 is a flowchart showing the operation of a conventional video summarization system.

FIG. 18 is a diagram showing a scene change detected by a conventional common color ratio method.

[Explanation of symbols]

 101, 801 video disk device 102, 802 VTR 103, 803 video summarization device 104, 804 image capturing means 105, 805 scene change detection means 106, 806 time judgment processing means 107, 807 group generation means 108, 808 group addition means 109, 809 Image similarity calculation means 110, 810 Image reference processing means 811 Image determination means 111, 812 Control device 112, 813 Video compression device 113, 814 File server 114, 815 Video display device 115 Summary video reproduction device 116 User interface device 117 Video Reproduction means

Claims (13)

[Claims] A plurality of scenes formed by dividing a captured video by detecting a scene change of the captured video, and a predetermined image of a scene before and after the scene in a time series (hereinafter, a predetermined image of the scene is represented as a representative image). A time-series group generation process in which scenes including a representative image are grouped into a time-series group by calculating the similarity of the image. A video sequence summarizing process for modifying the video sequence into a time series group independent of the scene described above, and a video summary information output process for outputting video summary information of each time series group obtained in the above two processes. Method. 2. The method according to claim 1, wherein the similarity between the representative images of the scene in the time series group generation process is calculated by calculating a ratio of pixels having a color common to the representative images to be compared. Video summarization method. 3. A predetermined image of a plurality of scenes (hereinafter, referred to as a reference scene) selected according to a predetermined reference from a plurality of scenes constituting the captured video (hereinafter, a predetermined image of the scene is represented). A similarity is calculated according to a predetermined criterion between a representative image of all scenes (hereinafter, referred to as a main part scene) constituting a video according to a predetermined criterion, and the similarity with the representative image of the reference scene is determined. A video summarization method comprising: a similarity calculation process of selecting a scene including a representative image having a value greater than or equal to a value; and a video summary information output process of outputting video summary information of the scene obtained in the above process. 4. In the similarity calculation process, a criterion for calculating the similarity between the representative image of the reference scene and the representative image of the main scene is to divide the representative image into a plurality of image areas, 4. The video summarizing method according to claim 3, wherein the RGB components of pixels in each image area of the image are compared. 5. A plurality of video summarization processes for grouping a plurality of scenes formed by dividing a captured video by detecting a scene change thereof into a plurality of time series groups and / or a predetermined process from the plurality of scenes. A video summarization method comprising: a plurality of video summarization processes for selecting a scene; and a video summary output process of outputting video summary information of each scene selected by each video summarization process. 6. A plurality of video summarization processes for grouping a plurality of scenes into a plurality of time-series groups and / or a plurality of video summarization processes for selecting a predetermined scene from the plurality of scenes include all scenes. A first video summarizing step of selecting, a second video summarizing step of selecting only a scene that continues for a predetermined time or more from a plurality of scenes of the captured video, and a predetermined A third video that calculates a similarity of an image (hereinafter, a predetermined image of a scene is referred to as a representative image) according to a predetermined criterion, and collects scenes including the representative image whose similarity is equal to or greater than a threshold into a time-series group. A summarizing process, a fourth video summarizing process according to claim 1 or claim 2, and a video summarizing process of at least two of the fifth video summarizing process according to claim 3 or claim 4. 6. The video summarizing method according to claim 5, wherein: 7. The video summary information includes a predetermined image of a scene (hereinafter, a predetermined image of a scene is referred to as a representative image) or a predetermined frame number (hereinafter, referred to as a representative frame number).
And selecting a representative image or a representative frame number in descending order of the number of frames between representative images or representative frame numbers, and selecting a predetermined number of scenes or a predetermined number of time-series scenes. 7. The video summarizing method according to claim 5, wherein the video summarizing method is characterized in that: 8. When a predetermined number of representative images or frame numbers cannot be selected by one video summarization process,
8. The video summarizing method according to claim 7, wherein a representative image or frame number selected by the other video summarizing means for the remaining representative images or frame numbers is selected. 9. The user inputs the priority of the video summarization process and the number of scenes or time series scenes to be selected,
9. The video summarizing method according to claim 7, wherein the video summarizing is performed according to the information. 10. A video display method for displaying a video and a representative image of the video selected by any one of claims 1 to 9, wherein a video connected to a part of a video near a position of the representative image is displayed. (Hereinafter referred to as a summary video.) A video display method characterized by reproducing a video starting from a frame specified above. 11. When the reproduction of a video (hereinafter referred to as a summary video) connecting a part of a video near the position of an image extracted from a video is interrupted, a representative representing the contents of the frame image at the interrupted position. A video display method characterized by displaying a list of a plurality of representative images including images. 12. When the first part of the summary video is included in the last part of the immediately preceding summary video, the last part of the summary video included in the last part of the immediately preceding summary video is included. The video display method according to claim 10, wherein a frame next to the frame is set as a head of the summary video. 13. A plurality of video summarizing means for extracting summary information of a video by grouping a plurality of scenes formed by dividing a captured video based on a predetermined standard into a plurality of time series groups, and / or A plurality of video summarization means for extracting video summary information by selecting a predetermined scene from the plurality of scenes; a summary information selection means for selecting the summary information extracted by the video summary means; and summary information A video summarization system having a display means.