JPH08251540A - Video summarizing method - Google Patents

Abstract

PURPOSE: To summarize automatically video by solving a problem that a desired content cannot be grasped when the video summarized by the user is viewed so as to provide efficiently a content among the operation of an object with respect to the method summarizing and displaying the video to be used for a device supporting retrieval, editing, processing and glance of the video. CONSTITUTION: The frame image of a video signal received by a video disk device 1 or a VTR 2 is received by a frame memory 3, a 1st computer 5 gives a control signal to the frame memory 3 to receive a frame image and the video is summarized by processing a time series frame image and date are stored in a file server 6. A 2nd computer 7 calls a summarized video from the file server 6 on request by the user and reproduces the video.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for supporting search, editing, processing, and quick viewing of videos, and more particularly to videos that are reproduced or displayed by summarizing videos stored on videotapes or videodiscs. It is about the summarization method.

[0002]

2. Description of the Related Art In recent years, researches on methods for assisting retrieval, editing, processing, and quick viewing of images by applying a computer have become popular. As an example, there is a video summarization method of reproducing a part of the video or the entire video in a short time in order to grasp the content of the video or the video of the movie in a short time.

As a conventional method, a fast-forward reproduction method, a video content compression display processing method described in Japanese Patent Laid-Open No. 4-237284, a variable speed reproduction method for each shot (Otsuji, Tonomura, "Subjective evaluation of high-speed moving image browsing", 1993 Spring Meeting of the Institute of Electronics, Information and Communication Engineers, SD-9-3), a rush reproduction method for each shot is proposed. Note that a shot is a unit of video often used in fields such as video editing, and is close to the minimum unit of video content.

The conventional fast-forward reproduction method is a video summarization method in which frame images are thinned out and reproduced at regular time intervals.

The video content compression display method uses the amount of change between time-series frame images to determine the display importance of each frame image, and the video summary that spends more time displaying the frame images with higher importance. Is the way. For practical use,
The part that determines the display importance of the frame image is important.

The amount of change in luminance in pixel units between adjacent frame images is sensitive to the movement of an object on the screen. Therefore, as a display importance level determination method used when skipping areas with little motion and slowing areas with many motions, a method has been proposed in which the brightness change amount in pixel units is regarded as the display importance level.

On the other hand, the amount of change in brightness in frame units between adjacent frame images is relatively insensitive to the movement of an object within a shot, and the tendency of brightness distribution in the entire frame changes as when the shot changes. In such a case, a large value is given. Therefore, as a display importance level determination method used when it is desired to watch changes in shots carefully, a method has been proposed in which the brightness change amount in units of frames is regarded as the display importance level.

The conventional variable speed reproduction method for each shot is a video summarization method for reproducing a video while controlling the reproduction speed so as to keep the display time of each shot constant.

The shot-by-shot rush reproduction method is a video summarization method in which the head portion of a shot is reproduced one after another at a standard speed.

Finally, the shot will be supplemented. A portion that is continuously shot with one video camera is called a shot. The shot is close to the minimum unit of the video content, as described above. In addition, the portion that is connected by editing and the portion where the video camera has stopped shooting are "changes in shot".

However, there is a case where a portion where the image content changes due to a camera operation such as pan and zoom is regarded as a "shot change" as an exception. In this case, the shot is closer to the minimum unit of the video content as compared with the case where the exception is not considered.

As a method of automatically dividing an image into shots, an image change model method (Yamada, Fujioka, Kanamori, Matsushima,
Sakauchi, "Scene change detection method of video including editing effect", multimedia and video processing symposium '94), etc. are proposed.

[0013]

However, in the fast-forward reproduction method described above, since reproduction is performed at a constant speed regardless of the video content, there are "subjectively high and low reproduction speed portions", and it is difficult to grasp the content. The problem is that it is difficult and the user is tired easily.

Further, in the above-mentioned video content compression display method,
As a method of determining the display importance, only a method of using a luminance change amount in pixel units between adjacent frame images and a method of using a luminance change amount in frame units have been proposed. When the former is used, the amount of luminance change in pixel units is unsuitable for subjective motion evaluation, so there are "subjectively high and low playback speed portions" and it is difficult to understand the contents. so,
Moreover, there is a problem that the user is easily tired.
Further, in the case of using the latter, only the first frame image of each shot is displayed, so there is a problem that it cannot be used when it is desired to grasp the contents centering on the motion of the subject.

In the above-mentioned variable speed reproduction method for each shot, since the reproduction speed is determined by the time length of the shot, the user cannot grasp the contents of some shots and wants to know the temporal flow of the contents included in the video. There was a problem that it could not be used sometimes.

[0016] If you want to know the temporal flow of contents,
It is important to understand the content of each shot, which is a unit of content, and it is important to be able to predict shot changes in order to avoid overlooking shots. However, in the variable speed playback method for each shot, "the shot changes depending on the part that is connected by editing and the part where the video camera shooting is interrupted."
Except in some cases, shot changes cannot be predicted. When a part shot by the same camera operation is treated as the above shot, or when one scene of a scenario is treated as the above shot, the change of the shot cannot be predicted and the shot may be overlooked. Had challenges.

In a video, a combination of a plurality of shots becomes a unit of contents of higher order than a shot such as a scene. With the above-mentioned variable speed playback method for each shot, all shots are played back little by little, so parts with similar contents are played back in succession, and it is not possible to use parts with different contents as efficiently as possible. Had challenges.

The present invention solves the above-mentioned problems of the prior art, and an object of the present invention is to provide a video summarizing method by which a user can grasp desired contents.

[0019]

In order to achieve this object, firstly, a time section of an image containing an object moving at high speed is detected and regarded as a high speed operation section. In addition, a time section in which similar images continue for a certain time or more is detected and regarded as a long-time similar section. Then, the high-speed operation section is reproduced at a lower speed than the time section other than the high-speed operation section, and the long-time similar section is reproduced at a higher speed than the time section other than the long-time similar section. Furthermore, after the video is grouped by content and divided into a plurality of shots, a lower limit is set for the display time length of each shot.

Secondly, the images are grouped by content and divided into shots, and then time-series shots are reproduced one after another while recalling a rhythm of a preset cycle. However, an upper limit is set for the display time length of each shot. Further, the reproduction speed is determined so as to satisfy at least one of the following two conditions.

Rhythm condition: A shot boundary and a rhythm of a preset cycle have a correlation.

Content condition: When the speed upper limit value is set so as to guarantee that "the content reproduced at a speed equal to or lower than the speed upper limit value which is the criterion for determining the reproduction speed can be grasped", each shot The playback speed of at least a part of is below the upper speed limit.

Thirdly, the video to be summarized is divided into shots. After that, similar shots are integrated to create a shot group by using the correlation between the shots in time series. Then, one partial moving image is selected from each shot group, and the partial moving images are reproduced one after another.

[0024]

With these configurations, firstly, since there is no object moving at high speed and each content is displayed for a certain time or longer, the "subjectively high reproduction speed" disappears. Also, since the video changes one after another in a short time, the "subjectively slow playback speed" disappears. Therefore, user fatigue is reduced as compared with the conventional method. In addition, this method subjectively reproduces the entire video while keeping the reproduction speed within the allowable range, so that it is possible to grasp the contents centering on the motion of the subject.

Secondly, portions having the same contents are collected into shots, and the shots are reproduced in a short time. Therefore, when summarizing the video so as to satisfy the rhythm condition, changes in shots can be predicted, so that all shots can be viewed without being overlooked.

On the other hand, when summarizing the video so that the content conditions are satisfied, the user
At least a part of the contents can be grasped.
A shot is a collection of parts with the same contents, so you can understand the contents of all shots.

Since this means creates shots by collecting the parts having the same contents and reproduces them one after another,
It is used when you want to know the temporal flow of the contents included in the video. However, if the rhythm condition is not satisfied, the change of the shot cannot be predicted, so that the shot is overlooked. If the content condition is not satisfied, the content of some shots cannot be grasped. It is desirable to satisfy both the rhythm condition and the content condition at the same time.

Thirdly, since a group of shots having similar contents in time series is created and these shot portions are reproduced one after another, it is possible to efficiently see the portions having different contents as much as possible.

[0029]

An embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 is an overall system diagram of a video summarizing device according to an embodiment of the present invention. In FIG. 1, 1, 2
Is an input device for an image to be processed (hereinafter referred to as an image to be processed), 1 is a video disk device, and 2 is a VT
R. Further, 3 is a video disc device 1 and a VTR 2
It is a frame memory that captures the frame image of the video signal output from the. 4 is a video disc device 1 or VT
This is a video compression device that compresses the video signal output from R2. Reference numeral 5 is a first computer, which is a video disk device 1
Controls the VTR 2, the frame memory 3, and the video compression device 4. In addition, a video is summarized by sending a control signal to the frame memory 3 to capture a frame image and processing the time-series frame image. Reference numeral 6 is a file server that stores the video compressed by the video compression device 4, the data and the frame image sent from the first computer 5. Reference numeral 7 denotes a second computer which calls up the summary video and the processing target video from the file server 6 in accordance with the user's request and reproduces the video.

The overall operation of the video summarizing device configured as described above will be described with reference to the flowchart shown in FIG.

In step 201, the first computer 5 in FIG. 1 summarizes the video by processing the time-series frame images while controlling the video disk device 1, the VTR 2 and the frame memory 3.

In step 202, the first computer 5 compresses the summarized video and the video to be processed and stores them in the file server 6 while controlling the video disk device 1, the VTR 2 and the video compression device 4.

As a method of storing the summary video, various methods other than the method of directly compressing the video can be considered. For example, as shown in FIG. 3, when the summary video can be expressed in a file format using the frame number of the processing target video, the frame server information may be stored in the file server 6 instead of the summary video. . Further, while controlling the video disk device 1, the VTR 2 and the frame memory 3, the first frame image of each shot group described later may be fetched and stored in the file server 6 instead of the summary video. However, in this case, reducing the stored first frame image and displaying the list is equivalent to presenting a summary of the video.

In step 203, the second computer 7 calls the summary video and the video to be processed from the file server 6 in response to the user's request and reproduces the video.

The video summarization process of the first computer 5, which is a specific operation of step 201 in FIG. 2, will be described below.

FIG. 4 is a flow chart of an embodiment of the video summarization process of the first computer 5 in FIG.

First, in step 401, various parameters used in the video summarization process are initialized and threshold values are set. Further, the frame memory 3 is used to calculate the feature amount such as “similarity between frame images”, high-speed operation pixel, still constituent color, moving constituent color, still common color, and moving common color in the procedure described later. To the frame image In to be processed
(N is a natural number of 1 or more) is fetched and this frame image is stored.

Next, in step 402, it is determined whether or not the video to be processed has ended. If the video has ended, the video summarization processing ends. If the video has not ended, the procedure proceeds to step 403.

In step 403, the frame image In + 1 next to the current frame image to be processed is regarded as a new frame image In to be processed. Then, a control signal is sent to the video disk device 1 and the VTR 2 to reproduce the updated frame image In to be processed, and to fetch the frame image In from the frame memory 3.

In step 404, the end of a shot, which is a portion captured by one video camera continuously in time, is detected. When the end of the shot is detected, this shot is regarded as a new processing target shot SHk (k is an integer of 1 or more), and then the procedure proceeds to step 405. If the end of the shot is not detected, the process proceeds to step 405 without doing anything. After proceeding to step 405, this step 404
Is executed again for the next frame image In + to be processed.
After capturing 1. Therefore, when the end of the shot cannot be detected, the next frame image to be processed In + 1
After capturing, you will try again to detect the end of the shot.

Note that one scene or the like of the scenario may be regarded as a shot. Therefore, all the "shots" used in the explanations after step 404 may be replaced with "one scene in the scenario".

As a method of detecting the end of a shot,
The image change model method introduced in the related art has been proposed. Furthermore, in the present embodiment, the video is automatically divided into shots, but the user may divide the shot while viewing the video. If the video is divided into shots in advance, step 404 may be omitted.

From step 405 to step 408 is the video summarization method using the following three assumptions based on the subjective evaluation result of the fast-forward video. The playback speed is determined so that the "part" disappears.

A portion where similar images continue for a long time (hereinafter,
It is called a similar section for a long time), and the playback speed is slow.

It is felt that the reproduction speed is high in the part of the image including the object moving at high speed (hereinafter referred to as the high speed operation section).

When an image is divided into shots according to contents, it is felt that a shot having a short display time (called a short shot) has a high reproduction speed.

The summary video created by this method will be referred to as a section shift summary video. Further, in the present embodiment, a time section other than the long-time similar section and the high-speed operation section is referred to as a standard section. Further, the reproduction speed of the standard section is set to 4 times speed.

Step 405 is a process for detecting a long-term similar section. Since the video of the 4 × fast-forward reproduction is composed of images at intervals of 4 frames, the detection processing is executed by sampling the images at intervals of 4 frames and checking the similarity between the images. A specific operation in step 405 will be described with reference to FIG.

In step 501, first, the frame number of the frame image In to be processed is checked. Next, it is determined whether or not this frame image is included in the image sequence of 4 frame intervals. Since the long-time similar section is detected using only the images at 4-frame intervals, if it is determined to be "included in the image sequence", the procedure 50 is performed to detect the long-term similar section.
Go to 2. If it is determined that it is not included in the image sequence, the procedure 405 is ended and the procedure proceeds to the procedure 406 in FIG.

In step 502, the processing target frame image I
n and the frame image In-4 of the previous image sequence in the above image sequence
Compute the similarity S (n, n-4) between and. However,
The calculation method of the similarity between frame images was decided so that there is no correlation with the motion that can be subjectively ignored by the user viewing the summary video.

Here, a method of calculating the similarity between frame images will be briefly described. χ2 test method (Nagasaka, Tanaka,
Various methods are conceivable such as "Automatic detection of scene change in video work", 40th National Convention of Information Processing Society of Japan, 1Q-5, 1990). In this embodiment, the common color ratio method (Yamada, Fujioka) is used. , Kanamori, Matsushima, "Study on Scene Change Detection Method Focusing on Common Color of Partial Areas", Technical Report of Television Society, September 1993, Vol.17, No.55, pp1-
Use the same method as 6).

In the case of a still image or an image containing only subjectively negligible movements, the distance that an object on the screen moves on the screen in 4 frame time should be much smaller than the width of the screen. In our experience, this distance is less than 2% of the width of the screen. Therefore, as shown in FIG. 6, the frame image In is divided into partial regions R (j, n), (where j is 1
The above is an integer of 16 or less). At this time, still images, images In and In-4 including only motions that can be subjectively ignored
The color histogram hardly changes between the corresponding partial regions R (j, n) and R (j, n-4). Although the number of partial areas is 16 in the present embodiment, the number of partial areas does not have to be 16.

On the other hand, when a shot change or the like occurs between the frame images In and In-4, the subject changes between the corresponding partial areas of In and In-4, and the colors forming the partial areas change. To do. Therefore, the similarity Sp (j, In, In-4) of the corresponding partial region is reduced according to the area of the color that has newly appeared or disappeared between the corresponding partial regions,
This similarity is calculated.

Then, the average of the similarities of the corresponding partial areas of the frame images In and In-4 is calculated and regarded as the similarity S (n, n-4) of the frame images. That is,

[0056]

[Equation 1] Is calculated.

Step 503 is a process for updating the leading frame number of the long-term similar section. Since the images in the long-term similar section are similar to each other, the similarity S between the time-series images is S.
Whether or not (n, n-4) becomes equal to or larger than the threshold value θstill is determined by the conditional expression S (n-4, n) ≧ θstill (2) To find out. When the expression (2) is established, (n-
4) is considered to be in the middle of the similar section for a long time, and if the similar section end Nb representing the head frame number is not set, Nb is
Substitute (n-4).

Step 504 is a process for detecting long-term similar section candidates. Whether or not the degree of similarity between images at 4-frame intervals is equal to or greater than the threshold value θstill between the frame image at the similar section end Nb and the processing target frame image In is determined by the conditional expression S (Nb + (i-1) × 4, Nb + i × 4) ≧ θstill, 1 ≦ i ≦ (n-Nb) / 4 (3). When the formula (3) is satisfied, the portion between the frame image INb at the end of the similar section and the processing target frame image In is regarded as a long-term similar section candidate, and the procedure proceeds to step 505. When the expression (3) is not satisfied, the candidate for the long-term similar section cannot be determined, and therefore, the procedure proceeds to step 406 of FIG.

The candidates obtained in step 504 include "a portion where similar images continue" and "a portion where a gradual image change occurs" as shown in FIG. In step 505,
First, the first frame image INb of the long-term similar section candidate,
Whether or not the degree of similarity with other frame images is equal to or greater than the threshold value θratio, conditional expressions S (Nb, Nb + i × 4) ≧ θratio, 1 ≦ i ≦ (n-Nb) / 4 ...・ Check using (4). Next, when the expression (4) is satisfied,
The time length of this candidate is the minimum time length Tst of the long-term similar section.
Whether or not it is longer than ill is examined using the conditional expression n−Nb ≧ Tstill (5). However, the minimum time length Tst in equation (5)
ill is set when executing the procedure 401 in FIG.

When the expressions (4) and (5) are satisfied at the same time, the candidates obtained in the procedure 504 are regarded as the long-term similarity section, and then the procedure 405 described above is terminated.

If the expression (4) is satisfied and the expression (5) is not satisfied, the candidates obtained in step 504 are the long-term similar sections unless the images after the processing target frame image In are processed. I can't judge whether it is a part. Therefore, the procedure 405 is terminated without doing anything.

If the expression (4) is not satisfied, this candidate is regarded as a "portion where a gradual image change occurs". Further, since the expression (2) is established, the frame number (n-4)
The following is considered as a long-term similar section candidate. At the similar section end Nb that represents the leading frame number of the long-term similar section candidate, (n
-4) is substituted.

In this embodiment, the calculation method of the similarity between frame images is determined so that there is no correlation with the motion that can be subjectively ignored by the user who views the summary video. However, since the subjectivity of the user varies depending on the user, the calculation method of the degree of similarity may be determined by emphasizing the movement of the object noticed by the user.

Further, in this embodiment, the "portion where similar images continue" was obtained using the equations (3) and (4). That is, the similarity between images in time series and the similarity based on the first frame image of the similar section are used. However, the same effect can be obtained by using a method that uses the last frame image of the similar section as a reference. Therefore, by using a method other than equations (3) and (4), the "portion where similar images continue" is used. You may calculate. Alternatively, the “portion of similar images” may be calculated using only equation (4).

After the procedure 405 in FIG. 4 is completed, the procedure proceeds to procedure 406. Step 406 is a process of detecting “a part of the image including an object moving at high speed”. A specific operation in step 406 will be described with reference to FIG.

The procedure 801 is the frame image In to be processed.
And the image In-1 immediately before that is used to detect pixels on an object that moves at high speed. The principle of this processing is shown in FIG.
Will be explained.

As shown in FIG. 9, the frame image In-1,
When the object on the position p (where p is a vector) moves at high speed between In and In, it is assumed that the original object does not already exist within the distance Lmin. At this time, due to the movement of the object, the brightness f (p, n-1) of the pixel at the position p on In-1.
, And the luminance f (p + d, n) of pixels within a circle of radius Lmin centered on the same position p on In (where d is a vector having a length of Lmin or less), It becomes equal to or greater than the preset threshold value θw1, and | f (p, n-1) -f (p + d, n) | ≧ θw1, | d · d | ≦ Lmin × Lmin ··· (6) holds. However, in Expression (6), | a | represents the absolute value of the scalar amount a, and (d · d) is the vector d.
Represents the dot product of.

Further, noise is removed by detecting only an object larger than a circle having a radius of Rmin. Assuming that the same luminance region is the same object, as shown in FIG.
Conditional expression | f (where the brightness difference with the brightness f (p + D, n−1) of the pixel at position p on 1 above (where D is a vector having a length of Rmin or less) is less than the threshold θw1 p, n-1) -f (p + D, n-1) | <θw1, | D ・ D | ≤ Rmin × Rmin ・ ・ (7) Whether or not expression (6) is satisfied at position p Find out.

Based on the above, step 801 in FIG.
Then, after detecting a pixel p that simultaneously satisfies the expressions (6) and (7), this pixel is regarded as a "pixel on an object moving at high speed" and registered as a high-speed operation pixel.

In order to simplify the processing, there are cases where pixels are thinned out, and the average of pixels of 8 pixels × 8 lines is regarded as one pixel. In this way, when the contents of a plurality of pixels are represented by one pixel, the area per pixel becomes large like a mosaic. In this case, as shown in FIG. 10, pixels satisfying only equation (6) may be regarded as high-speed pixels.

As a method of detecting an object moving at high speed, for example, a method using a motion vector can be considered.
The motion vector may be used to detect the fast-moving pixel as in the following procedure.

[Example of Procedure for Detecting High-Speed Moving Object Using Motion Vector] Procedure 1) Divide an image into a plurality of blocks, and then use two time-series images to perform motion for each block Find the vector.

Procedure 2) A block in which the magnitude of the motion vector is equal to or larger than a fixed value is obtained. Procedure 3) The pixels included in the block obtained in Procedure 2 are regarded as high-speed operation pixels.

Further, in the equations (6) and (7), the pixel to be compared with the pixel p is the pixel included in the specific circle. However, as shown in FIG. 11, the pixel included in the quadrangle including the pixel p is used. May be

It is also possible to assume a region of interest to the user in advance and detect a high-speed moving pixel from the pixels included in the assumed region. For example, as shown in FIG. 12, the fast moving pixel may be detected on the assumption that the user always pays attention to the vicinity of the center of the screen.

In step 802, the total number of high-speed operation pixels included in the frame image In-1 is checked. Then, when the total number of high-speed operation pixels becomes a value larger than the threshold value θm set for removing noise, a high-speed operation section candidate is provided between this frame image In-1 and the next frame image In. To consider.

As a result of the subjective evaluation of the fast-forward video, even if there is an object moving at a high speed, it is subjectively ignored if it moves only for a moment. Therefore, in step 804, from the high-speed operation section candidates, the section that becomes a candidate due to a momentary motion is excluded. There are several possible methods for realizing this processing, but here, one method will be described.

In the present embodiment, since the video of the quadruple speed fast-forward reproduction is used as a reference, the video is grouped into four frames and referred to as a section unit. In step 803, it is determined whether or not a high-speed operation section candidate is detected in step 802. When it is determined that "detected", the section unit including this high-speed operation section candidate is UN (N is an integer of 1 or more)
Will be written, and the process proceeds to step 804. If it is determined that “there is no detection”, there is no high-speed operation section candidate, and the procedure 406 is terminated.

Step 804 is a process for determining whether the section unit including the high-speed operation section candidate continues for a preset lower limit value Nmove or more. Further, it is checked whether or not all the section units between the section unit UN and the section unit UN- (Nmove-1) * 4 which is (Nmove-1) units before the section unit include the high speed operation section candidate. Next, when it is determined that “a candidate for the high-speed operation section is included”, the area between the frame images IN-Nmove * 4 and IN is regarded as the high-speed operation section.

After the step 804 is finished, the procedure 406 described above is finished. Then, after the end of step 406 of FIG. 4, the process proceeds to step 407.

Step 407 is a process for determining the video reproduction speed for the section shift summary video. First, the reproduction speed in the high speed operation section is determined to be double speed. Next, from the long-time similar sections, a section that does not include a high-speed operation section in the middle is taken out, and the reproduction speed thereof is determined to be 8 times speed. Further, the remaining section in which the reproduction speed is not determined is regarded as the standard section, and the reproduction speed is determined to be 4 times speed.

In step 407, the reproduction speeds of the high-speed operation section, the standard section, and the long-time similar section are set to 2 times speed, 4 times speed, and 8 times speed, respectively. If it is slower and the reproduction speed in the similar section for a long time is higher than that in the standard section, another value may be set. Further, the display speed may be used to determine the reproduction speed, as in the case of reproducing the entire long-duration similar section in 1 second.

When the video is played back according to the playback speed determined in step 407, the display time becomes less than the preset shot length lower limit value Nshot for some shots. Therefore, in step 408, the reproduction speed is corrected.

In step 408, first, for each shot SHk obtained in step 404, the time NSHk (k
Is an integer greater than or equal to 1). Next, the shot length lower limit value N
Check if it is less than shot using conditional expression NSHk <Nshot ・ ・ ・ ・ ・ ・ ・ ・ (8). When the expression (8) is satisfied, the shot SHk is regarded as a short-time long shot, and the reproduction speed is determined so that the shot display time becomes the shot-length lower limit value Nshot.

In this embodiment, the shot length lower limit value is fixed, but the shot length lower limit value may be changed according to the details such as the fineness of the texture.

After the step 408 is completed, the procedure advances to the step 409.
Step 409 realizes a video summarization method of reproducing time-sequential shots one after another while deciding the reproduction speed so that the boundary of the shot and the rhythm of the preset cycle have a correlation with each other, and evoking this “rhythm”. It is a procedure for doing. First, the playback speed is determined so that the summary video has a rhythm with a preset cycle at the shot boundaries. Next, the playback speed is modified so that the user can grasp the content of each shot while maintaining the rhythm. Note that step 409
The summary video created in Section 1 will be called the rhythm presentation summary video.

A concrete operation in step 409 is shown in FIG.
Will be described using. In order to give a rhythm of a preset cycle to the shot boundaries, for example, the playback speed may be determined so that the display time of each shot is constant.
Various methods are conceivable for evoking the rhythm, such as a method of smoothing sound at the boundaries of shots and a method of displaying a still image at the boundaries of shots.

In this embodiment, as in the example shown in FIG.
The first frame image of the shot is displayed for the still time Tstl, and then the entire shot is fast-forwarded for the fast-forward time Tlen. At this time, a rhythm with an interval of (Tstl + Tlen) is generated. For example, if Tstl is set to 0.2 seconds and Tlen is set to 0.8 seconds, a rhythm of 1 second interval is generated.

In step 1301 of FIG. 13, the number LSHk of frames (k is an integer of 1 or more) of the shot SHk to be processed is calculated.

In step 1302, the processing target shot SHk
The display time of the fast forward portion of the
The reproduction speed V (k) is determined by V (k) = LSHk / Tlen (9) so that len is obtained.

In step 1303, the speed upper limit value Vmax which is a reference for determining the reproduction speed of the processing target shot SHk so that the user can grasp the contents of the processing target shot SHk.
Determine (k). Various methods of determining the speed upper limit value Vmax (k) can be considered. For example, a method of averaging the reproduction speeds determined in step 407 in the shot SHk to be processed and substituting this value for the speed upper limit value Vmax (k) can be considered.
Also, a method of considering a preset fixed value as Vmax (k) can be considered. Furthermore, the following method using the long-term similar section, the standard section, and the high-speed operation section determined in step 407 can be considered.

[Method for Determining Velocity Upper Limit Value Using Long-Time Similar Section, Standard Section, and High-Speed Operation Section] Procedure 1) A shot including only the standard section in a shot that includes a long-time similar section and does not include a high-speed operation section The speed upper limit value Vmax (k) is set to a larger value as compared with. For example, it is set to 8 times speed.

Procedure 2) For a shot that includes a high-speed motion section and does not include a long-term similar section, the speed upper limit value Vmax (k) is set to a smaller value than a shot having only a standard section. For example, it is set to double speed.

Procedure 3) The upper speed limit value Vmax (k) is set to the same value for the shot including both the high speed operation section and the long time similar section and the shot for the standard section only.
For example, it is set to 4 times speed.

In step 1304, the reproduction speed V (k) determined by the equation (9) is equal to the speed upper limit value determined in step 1303.
Whether or not Vmax (k) or more is checked by using the conditional expression V (k) ≧ Vmax (k) ... (10). If the formula (10) is satisfied, the processing target shot SHk is displayed in order to let the user understand the contents.
Correct the playback speed by extending the display time of the fast forward part of. This correction processing is executed in procedure 1305.

If the equation (10) is not satisfied, the process shown in FIG.
After the reproduction speed of the fast-forward portion of the shot SHk is determined to be V (k) like the shot SH1 portion of the above, the procedure 1309 described above is ended.

As described above, the procedure 1305 is a process for making the user understand the contents of the shot, and extends the shot display time and corrects the reproduction speed.

In order to give a rhythm to the shot boundary,
Set the shot display time to "an integral multiple of other shots". However, in order to reproduce each shot one after another in a short time, the display time of the fast-forward portion of each shot is made as short as possible. Therefore, in procedure 1305, the display time of the fast-forward portion of the shot satisfying the expression (10) is set to 2 × Tlen, which is twice the display time of other shots.

Next, a method of determining the reproduction speed will be described. The reproduction speed of the first half of the shot satisfying the expression (10) is corrected to be equal to or lower than the speed upper limit value Vmax (k) determined in step 1303. That is, the shot SH in FIG.
As shown in 2, set the playback speed VP (k) in the first half to the speed upper limit value V
Set a value equal to or less than max (k), and set the playback speed V in the latter half so that the display time for the entire fast-forward portion is set to 2 x Tlen.
Determine R (k). However, the playback speed VR in the second half
(K) may exceed Vmax (k). Also, in order to maintain the rhythm, a stationary part is inserted between the first half and the second half of the fast-forward part.

The playback speed VP (k) in the first half of the fast-forward portion is

[0101]

[Equation 2] Can be determined by Also, the playback speed in the second half of the fast-forward portion
VR (k) is

[0102]

(Equation 3) Can be determined by After calculating equations (11) and (12),
The procedure 409 described above is ended.

If some shots whose contents cannot be grasped may occur, the procedure 409 may be terminated after the procedure 1302 is terminated. Further, when it is not necessary to predict the change in shot, the reproduction speed V (k) may be determined without using the equation (9).

After the procedure 409 of FIG. 4 is completed, the procedure proceeds to the procedure 410. From steps 410 to 412, similar shots are integrated and regarded as a shot group by using the correlation between shots in time series, and a plurality of time series frame images (for example, one second) are extracted from one shot group. This is a procedure for realizing a video summarization method of sequentially reproducing partial moving images in time series after selecting such partial moving images. First, similar shots are integrated to create a shot group using correlation between shots in time series. Next, one partial moving image is selected from each shot group and processed so as to clearly show the characteristics of the shot group. For example, when shots including similar objects are integrated to create a shot group, this similar object is a feature of the shot group. Further, when the shots having almost the same time length are integrated, this time length becomes a feature of the shot group.

When reproducing the summary video, the processed partial moving images are reproduced one after another. In addition, without performing processing,
You may directly reproduce the selected partial moving image.

The summary video created in steps 410 to 412 will be referred to as a partial selection summary video.

Step 410 is a process for creating a shot group. The specific operation in step 410 will be described with reference to FIG.

Since shots are units of content, frame images in one shot have one common feature. For example, in a shot of tracking a person, the person appears in all frame images. Therefore, the content of the shot can be represented by using a part of the frame images in the shot. In this embodiment, in order to simplify the process,
The number of frame images Nrep representing the content is set in advance, and the content of the shot is represented by using Nrep frame images in the shot (hereinafter, referred to as representative spatiotemporal image).

At step 1601, an image representative of the shot SHk is determined and regarded as the representative spatiotemporal image Ik, j, (k is an integer of 1 or more, j is an integer of 1 or more and Nrep or less). However, as a method of determining the representative spatiotemporal image, various methods such as a method of selecting the Nrep sheets at the head portion and a method of selecting images at fixed time intervals can be considered. If the processing amount is not limited, all frame images in the shot may be regarded as the representative spatiotemporal image.

After the step 1601, the shots of similar contents in time series are integrated using the representative spatiotemporal image and regarded as a shot group. Examples of shots of similar content include shots of similar backgrounds and shots of similar subjects. In this embodiment, the shots are integrated by paying attention to the color and movement of the object on the screen.

According to the analysis by the authors, in the representative spatiotemporal images of shots having similar contents, there is a color common to each representative spatiotemporal image, and objects having the common color make common movements. It is difficult to check whether two objects appearing on the screen have a common motion, but a change due to the motion of the object can be easily detected by using a pixel change region, a motion vector, and the like described later. Therefore, the representative spatiotemporal image is processed and divided into an area (hereinafter, referred to as a moving area) affected by a change due to the movement of an object on the screen and a remaining area (hereinafter, referred to as a stationary area). At this time, shots having similar contents satisfy the following common color ratio condition.

Common color ratio condition: When a color forming a still area and a color forming a moving area of a representative spatiotemporal image are called a static constituent color and a moving constituent color, respectively, a still constituent color common to the representative spatiotemporal image. , There are moving constituent colors, and pixels having these colors occupy a proportion of the threshold θ shot or more in each representative spatiotemporal image.

An example of similar shots satisfying the common color ratio condition is shown below. Example 1: A shot with a similar background, such as shots taken at the same location. Representative spatiotemporal image I of two shots SHk-1 and SHk of the poolside landscape shown in FIG.
In k-1, 1, Ik, 1, Ik-1, the proportion of static constituent color A in 1 = 60% Ik-1, the proportion of static constituent color B in 1 = 20% Ik-1, in 1 Proportion of static constituent color C = 15% Ik-1, proportion of static constituent color D in 1 = 5% Ik, proportion of static constituent color A in 1 = 80% Proportion of static constituent color C in Ik, 1 = 15% Ik, the proportion of the stationary constituent color E in 1 = 5%, and the sum of the proportions of the common stationary constituent colors A and C in Ik-1, 1 = 7
5% The sum of the ratios of Ik of common stationary constituent colors A and C and 1 = 95
%, The common static component color occupies 75% or more of the entire screen.

Example 2: A shot of a similar subject, such as when shots of the same subject continue. Representative spatiotemporal image I of two shots SHk-1 and SHk of vehicle tracking shown in FIG.
In k-1, 1, Ik, 1, Ik-1, the proportion of static constituent color V in 1 = 40% Ik-1, the proportion of static constituent color W in 1 = 40% Ik-1, in 1 Ratio of static constituent color X = 10% Ik-1, ratio of static constituent color Y in 1 = 10% Ik, ratio of static constituent color Z in 1 = 40% Ratio of static constituent color W in Ik, 1 = 35% Ik, the proportion of static constituent color X in 1 = 10% Ik, the proportion of still constituent color Y in 1 = 15%, and Ik-1, 1 of common moving constituent colors W, X, Y Sum of proportions =
60% Sum of ratios of common dynamic constituent colors W, X, Y in Ik, 1 = 6
Since it is 0%, the common dynamic constituent color occupies 60% or more of the entire screen.

In step 1602 of FIG. 16, a moving area is detected from the representative spatiotemporal image and the representative space-time image is divided into a moving area and a still area. Using the motion vector, the motion area becomes as shown in FIG. Further, when the object does not move between the two frame images, the brightness of the pixels at the same position is almost equal, so that if the next pixel change area is regarded as the moving area, the moving area becomes as shown in FIG. .

Pixel change region: The frame image Ik, j in the representative spatiotemporal image and the frame image one frame time later.
A set of "pixels where the absolute value of the brightness difference between pixels at the same position is greater than or equal to the threshold value θW1".

Step 1603 is a process for obtaining a static constituent color and a moving constituent color. Hereafter, the red component, green component, and
The procedure will be described by referring to the blue components as R, G, and B, respectively. First, in the stationary area of the representative space-time image of the shot SHk, a histogram HS (c, SHk) of 512 colors c of 8 gradations of RGB is obtained. Similarly, the histogram HM (c, SHk) of the moving area is obtained. Next, in order to exclude colors with a small number of pixels as noise, set the minimum number of constituent colors θH, and use the following formulas for static constituent colors HSVk and moving constituent colors HMVk (k is an integer of 1 or more). ).

HSVk = {c | HS (c, SHk)> θH} (13) HMVk = {c | HM (c, SHk)> θH} (14) In the following, the top shot of the shot group is described as SHtop (top is an integer of 1 or more). Steps 1604 to 16
Up to 06, the processing is performed by examining time-series shots after this shot SHtop to see which shots satisfy the common color ratio condition.

In step 1604, shots SHtop-SHtop
The static component color and the dynamic component color that are common between + m are referred to as the static common color HSCtop, m and the dynamic common color HMCtop, m (top, m is an integer of 1 or more), and the following formula is used. calculate.

[0120]

[Equation 4] In step 1605, the "static common color HSCtop, m and moving common color HM
The ratio of the pixel having Ctop, m to the representative space-time image of the shot SHk "is called the image common color ratio AMC (k, top, m), and is calculated using the following formula.

[0121]

(Equation 5) In step 1606, the expressions AMC (k, top, m) ≧ θshot, top ≦ k ≦ top + m (18) representing the establishment of the common color ratio condition are all k Shot SHtop
~ SHtop + m is regarded as satisfying the common color ratio condition. In order to check up to which shots satisfy the common color ratio condition, if the expression (18) is satisfied for all ks, 1 is added to m and the process returns to step 1604. Otherwise, proceed to step 1607.

In this embodiment, the end of the shot group is obtained using the image common color ratio between two time-series shots. First, in step 1607, the image common color ratios AMC (k, k, 1) and AMC (k + 1, k, 1) defined by equation (17) (k is an integer from top to top + m) are calculated. To do. The former AMC (k, k, 1) represents “the ratio of the static common color and the dynamic common color of the two shots SHk and SHk + 1 to the representative spatiotemporal image of the previous shot SHk”. The latter AMC (k + 1, k, 1) is "two shots
SHk, SHk + 1 static common colors and dynamic common colors are shot later SH
“K + 1 occupies the representative spatiotemporal image”.

In step 1608, the minimum value SS (k) = min (AMC (k, k, 1), AMC (k + 1, k, 1)) of the image common color ratio is ... -Calculate (19) and regard it as the priority for the integration of shots SHk and SHk + 1.

At step 1609, the shots SHtop to SHtop + m-1 which satisfy the common color ratio condition and the next shot SHto.
Find the end of the shot group from p + m. Among the integration priorities SS (top) to SS (top + m-1) between these shots, the expression SS (kmin) ≤ SS (k), top ≤ k that represents the lowest priority. ≤top + m-1 ····························· (20) to find the SS (kmin), as shown in the example of FIG. Shot SHkmin is considered to be the end of the shot group. (The second shot in FIGS. 21A and 21B) And the procedure 16 described above
Finish 10

Various methods of obtaining the shot group can be considered. For example, in the case of time-series shots, if the time lengths are almost equal, the contents are similar and can be regarded as one shot group. Therefore, the shot group may be obtained using a method different from that of this embodiment.

After step 1606, the process proceeds to step 1607 and the end of the shot SHk +
The procedure 410 may be terminated by regarding m-1 as the end of the shot group.

After the step 410 of FIG. 4 is completed, the procedure advances to step 411. In step 411, one partial moving image is selected from each shot group. However, there are various methods of selecting the partial moving image, such as the head portion of the head shot of the shot group and the head portion of the shot representing the contents of the shot group. As for the method of setting the time length of the partial moving image, various methods such as a fixed length method and a method of changing the time length depending on the content of the sound can be considered.

Although the partial moving image is selected in step 411, one still image may be selected from each shot group. In this case, when the selected still images are displayed one after another, a summary video is displayed. Further, the selected still images may be reduced and displayed in a list. The list display result can be handled in the same manner as the summary video.

After the procedure 411 is completed, the procedure proceeds to the procedure 412.
Step 412 is processing for clearly indicating the characteristics of the shot group, and the partial moving image selected in step 411 is processed.

In this embodiment, the shots included in the shot group satisfy the common color ratio condition. That is, each shot forming the shot group has the static common color and the dynamic common color obtained in step 1607.

Since the static common color and the dynamic common color represent the colors of the objects that commonly exist in the shot groups, they can be regarded as the characteristics of the shot groups. For example, a shot of a similar background indicates the background color, and a shot of a similar subject indicates the subject color. Therefore, in step 412, the pixels of the stationary common color and the pixels of the moving common color are displayed as usual, and the other pixels are displayed with half the luminance. Note that any processing method may be used as long as the static common color and the dynamic common color are emphasized, for example, the luminance of pixels excluding the static common color and the dynamic common color is set to zero.

The image processing method is determined according to the shot integration method, and there are various variations. For example, in the case of integrating shots having almost the same time length, a processing method in which the average time length of shots is displayed below the partial moving image as shown in FIG. 22 can be considered.

After the step 412 is completed, the procedure returns to the step 402.
If it is not necessary to clearly specify the characteristics of the shot group, the procedure 412 may be skipped and the procedure may return to the procedure 402 after the procedure 411 ends.

The above is one embodiment of the video summarization process of the first computer 5 in FIG. 1, and is a detailed description of the procedure 201 in FIG.

After the video summarization process is completed, the procedure 202 in FIG.
To execute. In step 202, the first computer 5 in FIG. 1 controls the video disk device 1 and the VTR 2 to reproduce the video to be processed and the video summarized in step 201. The reproduced video is MPEG in the video compression device 4 of FIG.
It is compressed by the method and stored in the file server 6.

However, in the procedure 201 of the present embodiment, since the method of reproducing the summarized video as shown in FIG. 3 is created without creating the summarized video, the video disc device is operated according to the method of reproducing the summarized video. By controlling 1 or VTR2, reproduction of the summary video is executed.

It is not always necessary to compress in the MPEG system, but it may be compressed in another compression system such as the JPEG system. Further, as the storage method of the summary video, as described above, not only the method of directly compressing the video but also various other methods can be considered.

The summary video reproduction process of the second computer 7 of FIG. 1, which is a concrete operation of the procedure 203 in FIG. 2, will be described below.

FIG. 23 is a flowchart of an embodiment of the summary video reproduction process of the second computer 7 in FIG.

In step 2301, a video display method is selected. However, many options are possible. For example, if the user wants to see the details of the video in detail, “standard speed playback of the video to be processed” may be selected. When the video is displayed, the video that the user wants to watch is played back at a normal speed.

When it is desired to grasp the contents centering on the motion of the subject, the "section shift summary video" created by the processing of steps 405 to 408 in FIG. 4 may be selected. When the image is displayed, the entire image is reproduced while the reproduction speed is subjectively within the allowable range.

There are individual differences in subjectivity. Therefore, the user may select the reproduction speed determination method according to his / her subjectivity. Specifically, instead of executing the following three processes, only a part of them may be executed, or a process of “relatively slowing the playback speed in a part of a video with a fine texture” may be added. Good.

-Procedure 405 for relatively increasing the reproduction speed in a long-time similar section-Procedure 406 for relatively decreasing the reproduction speed in a high-speed operation section-Reproduction with a short short shot Procedure 408 for making the speed relatively slower In addition, if the user wants to know the temporal flow of the contents included in the video, the “rhythm presentation summary video” created by the procedure 409 in FIG. 4 may be selected. . At the time of image display, shots that have the same content are collected one after another.

When it is desired to see the portions having different contents as little as possible, the "partial selection summary video" created by the processing from step 410 to step 412 in FIG. 4 may be selected. At the time of image display, a shot group in which shots having similar contents in time series are collected is played one after another.

In step 2302 of FIG. 23, step 2301
According to the selection result according to, the required video is called from the file server 6 in FIG. 1 and the video is reproduced. However, when only the frame number information for creating the summary video is stored in the file server 6, the processing target video is reproduced while being summarized in accordance with the frame number information. In addition, the user may specify "from which frame to which frame to play" before playing the video, or by only specifying "from which frame to play" and watch the video at the desired time. Playback may be interrupted with.

Note that the flow of the video summarization process shown in FIG. 4 may be decided according to the option of step 2301. For example, when the option of step 2301 does not include “rhythm presentation summary video”, step 409 of FIG. 4 does not need to be executed. Similarly, if the "section shift summary video" is not included, it is not necessary to perform steps 405 to 408 in FIG. 4, and if the "partial selection summary video" is not included, the procedure in FIG. It is not necessary to perform steps 410 to 412.

Further, in the present embodiment, an example in which the summarized video and the video to be processed are played back on the second computer 7 in FIG. 1 is shown, but it is played back on another computer connected to the file server 6. You may. For example, the reproduction may be performed on the first computer 5 in FIG. 1, or if two or more computers are connected to the file server 6, the reproduction may be performed on all the computers.

As described above, according to the section shift summary video of the present embodiment, a time section in which similar images continue for a certain time or more is detected and regarded as a long-time similar section, and the long-time similar section is relatively set. By playing back fast, the video changes one after another in a short time, so that the "subjectively slow playback speed" disappears. Also, by detecting a time section that includes an object that moves at high speed and considering it as a high-speed operation section, and making the playback speed in the high-speed operation section relatively slow, there is no object that moves at high speed. There is less "fast part". Furthermore, by dividing the video into multiple shots by content and setting a lower limit on the display time length of each shot, each content is displayed for a certain period of time or more,
The “subjectively high playback speed” disappears. Thus, subjectively, by reproducing the entire image while keeping the reproduction speed within the allowable range, user fatigue is reduced as compared with the conventional method. In addition, it is possible to grasp the contents centering on the motion of the subject.

Further, in the rhythm presentation summary video of this embodiment, after setting the upper limit to the display time length of each shot, the reproduction speed is determined so as to satisfy at least one of the following two conditions, and the rhythm is set. Recalling the time-sequential shots one after another while recalling.

Rhythm condition: A shot boundary and a rhythm of a preset cycle have a correlation.

Contents condition: When the speed upper limit value is set so as to guarantee that "a portion reproduced at a speed equal to or lower than the speed upper limit value which is the criterion for determining the reproduction speed can always grasp the contents", each shot The playback speed of at least a part of is below the upper speed limit.

By summarizing the video so as to satisfy the rhythm condition, it is possible to predict a change in shots, so that all shots can be viewed without being overlooked.

By summarizing the video so as to satisfy the content condition, the user can grasp the content of at least a part of all shots. A shot is a collection of parts with the same contents, so you can understand the contents of all shots.

However, when the rhythm condition is not satisfied, the change of the shot cannot be predicted, so that the shot is overlooked. If the content condition is not satisfied, the content of some shots cannot be grasped. It is desirable to satisfy both the rhythm condition and the content condition at the same time.

Further, in the partial selection summary video of this embodiment, similar shots are integrated by using the correlation between shots in time series to create a shot group, and then a partial moving image selected from each shot group is created. Are played one after another. By using this summary video, it is possible to efficiently see different content parts.

[0156]

As described above, according to the present invention, firstly, a time section in which similar images continue for a certain time or more is detected and regarded as a long-time similar section, and the long-time similar section is reproduced relatively quickly. By doing so, you can change the image one after another in a short time,
The “subjectively slow playback speed” can be eliminated. In addition, by detecting the time interval of the image containing a fast-moving object and considering it as the high-speed motion interval, and by making the playback speed of the high-speed motion interval relatively slow, the fast-moving object disappears, and It is possible to reduce the “high speed portion”. Furthermore, by grouping shots of the same content, and setting a lower limit on the display time length of each shot, each content can be displayed for a certain period of time or longer, and the "subjectively high playback speed" can be eliminated. it can. Thus, subjectively, by reproducing the entire image while keeping the reproduction speed within the allowable range, user fatigue is reduced as compared with the conventional method. In addition, it is possible to grasp the contents centering on the motion of the subject.

Secondly, after setting an upper limit on the display time length of each shot, the playback speed is determined so as to satisfy at least one of the following two conditions, and the rhythm is recollected in time series. Replay shots one after another.

Rhythm condition: the shot boundary and the rhythm of a preset cycle have a correlation.

Content condition: Each shot when the speed upper limit value is set so as to guarantee that "a part reproduced at a speed equal to or lower than the speed upper limit value which is a criterion for determining the reproduction speed can always grasp the content". The playback speed of at least a part of is below the upper speed limit.

By summarizing the video so as to satisfy the rhythm condition, the change in shots can be predicted, so that all shots can be viewed without being overlooked.

By summarizing the video so as to satisfy the content condition, the user can grasp the content of at least a part of all shots. A shot is a collection of parts with the same contents, so you can understand the contents of all shots.

However, when the rhythm condition is not satisfied, the change of the shot cannot be predicted, so that the shot is overlooked. If the content condition is not satisfied, the content of some shots cannot be grasped. It is desirable to satisfy both the rhythm condition and the content condition at the same time.

Thirdly, similar shots are integrated to create shot groups by using the correlation between shots in time series, and then partial moving images selected from each shot group are reproduced one after another. By using this summary video, it is possible to efficiently see different content parts.

[Brief description of drawings]

FIG. 1 is an overall system diagram of a video summarizing device according to an embodiment of the present invention.

FIG. 2 is a flowchart of the operation of the video summarizing device in the embodiment.

FIG. 3 is a conceptual diagram of representation of a summary video in a file format in the embodiment.

FIG. 4 is a flowchart of video summarization processing in the same embodiment.

FIG. 5 is a flowchart of a long-term similar section detection process in the embodiment.

FIG. 6 is a diagram showing an example of creating a partial area in the embodiment.

FIG. 7 is a diagram showing a portion where a video image changes gradually in the embodiment.

FIG. 8 is a flowchart of a high speed operation section detection process in the same embodiment.

FIG. 9 is a diagram showing a case where an object on the screen moves at high speed in the embodiment.

FIG. 10 is a diagram showing comparison of pixels using only the formula (6) in the example.

FIG. 11 is a diagram showing a comparison of pixels at position p in equations (6) and (7) in the same embodiment.

FIG. 12 is a diagram showing a region in which high-speed operation pixel detection is performed in the same embodiment.

FIG. 13 is a flowchart of a rhythm presentation summary process in the same embodiment.

FIG. 14 is a conceptual diagram of a reproducing method satisfying a rhythm condition in the embodiment.

FIG. 15 is a conceptual diagram of a method of reproducing a rhythm presentation summary video in the embodiment.

FIG. 16 is a flowchart of shot integration processing in the same embodiment.

FIG. 17 is a conceptual diagram of a shot of a similar background in the example.

FIG. 18 is a conceptual diagram of a shot of a similar subject in the embodiment.

FIG. 19 is a conceptual diagram of a moving area in the example.

FIG. 20 is a conceptual diagram of a moving area in the example.

FIG. 21 is a conceptual diagram of tail determination processing in the same embodiment.

FIG. 22 is a conceptual diagram of a method for processing a partial moving image in the same example.

FIG. 23 is a flowchart of a summary video reproduction process in the embodiment.

[Explanation of symbols]

 1 Video Disk Unit 2 VTR 3 Frame Memory 4 Video Compressor 5 Computer 6 File Server 7 Computer

Claims (29)

[Claims] 1. A similarity between frame images of videos to be summarized is calculated, and a long-term similar section, which is a time section in which similar images continue for a predetermined time or more, is detected, and the long-term similar section is detected. The video summarization method for reproducing the video data faster than the time interval other than the long time similar interval. 2. The two frame images are each divided into a plurality of partial areas, and the two are divided by using the similarity of the partial areas.
The video summarization method according to claim 1, wherein the degree of similarity between the frame images is calculated. 3. A high-speed operation section, which is a time section in which an object on the screen moves faster than other time sections, is detected from a frame image of a video to be summarized, and the high-speed operation section is other than the high-speed operation section. A video summarization method that plays back slower than the time interval. 4. The video summarization method according to claim 3, wherein the high-speed motion section is detected in a predetermined area of the frame image. 5. A time-series frame image IM (M is a natural number of 1 or more) is sampled from a video to be summarized,
Between a pixel PN in a specific frame image IN (N is a natural number greater than or equal to 1 and less than M) and a pixel around the pixel PN + 1 corresponding to the pixel PN in the next frame image IN + 1. 4. The pixel PN for which the brightness difference exceeds a preset threshold value .theta.W1 is calculated for each frame image unit, and the high-speed operation section is calculated using the pixel PN. 4. The video summarization method described in 4. 6. A time-series frame image IM (M is a natural number of 1 or more) is sampled from a video to be summarized,
A first luminance difference, which is a luminance difference between the pixel PN in the specific frame image IN (N is a natural number of 1 or more and less than M) and the pixels around the pixel PN, is calculated, and the pixel PN is calculated.
And a second luminance difference which is a luminance difference between a pixel around the pixel PN + 1 corresponding to the pixel PN in the next frame image IN + 1 and the second luminance difference is calculated for each frame image. The pixel PN at which the brightness difference of 1 is less than the preset threshold value θW1 and the second brightness difference exceeds the threshold value θW1, and the high-speed operation section is calculated using the pixel PN. The video summarizing method according to claim 3 or 4. 7. The video summarization method according to claim 3, wherein the high-speed motion section is obtained by using a motion vector. 8. A frame image of a video to be summarized is divided into a plurality of shots for each content, and a lower limit is set for a display time length of the shot according to the content of the shot. The video summarization method described in any one of. 9. A frame image of a video to be summarized is divided into a plurality of shots for each content, and a reproduction speed is determined so that a boundary between the shots and a rhythm of a preset cycle have a correlation, A video summarizing method of sequentially reproducing the shots while recalling. 10. The video summarizing method according to claim 9, wherein a rhythm is recalled by using a sound. 11. Reproduction is interrupted at intervals of a preset time Tlen, a frame image displayed at that time is displayed for a preset time Tstl, and a rhythm having (Tlen + Tstl) as a cycle is generated, 10. The video summarizing method according to claim 9, wherein the reproduction speed is determined so that the rhythm has a correlation with a shot boundary. 12. A frame image of a video to be summarized is divided into a plurality of shots for each content, a speed upper limit value that is a criterion for determining a playback speed is determined according to the content of the video, and at least a part of the shots is obtained. Is reproduced at a speed equal to or lower than the speed upper limit value, the reproduction speed of the shot is determined so that the display time length of the shot becomes equal to or lower than a time length upper limit value which is a preset threshold value, and the shots are sequentially reproduced. Video summarization method. 13. A frame image of a video to be summarized is grouped for each content and divided into a plurality of shots, a speed upper limit value which is a reference for determining a playback speed is determined according to the content of the video, and the entire shot is shot. Preset reference time length Trh
When the reproduction speed required to display y is equal to or less than the speed upper limit value, the reproduction speed of the shot is determined so that the display time length of the shot becomes the reference time length Trhy, and the entire shot is displayed. When the reproduction speed necessary for displaying with the reference time length Trhy exceeds the speed upper limit value, the display time length of the shot is changed to the reference time length T.
Set to twice rhy and determine the playback speed of the shots so that the playback speed during the first half of the display time length, Trhy, is less than or equal to the speed upper limit value. Method. 14. A similarity between frame images is calculated, a long-time similar section, which is a time section in which similar contents continue for a predetermined time or more, is detected, and in a shot including the long-time similar section, the long-range similar section is detected. 14. The video summarization method according to claim 12, wherein the upper speed limit value is set to a value larger than that of a shot that does not include a time-similar section. 15. The video summarization method according to claim 14, wherein each of the two frame images is divided into a plurality of partial regions, and the similarity between the two frame images is calculated using the similarity of the partial regions. 16. A high-speed motion section, which is a time section in which an object on the screen moves faster than other time sections, is detected from a frame image of a video to be summarized, and the shot including the high-speed operation section has the high-speed operation section. 14. The video summarization method according to claim 12, wherein the upper speed limit is set to a value smaller than that of a shot that does not include a motion section. 17. The video summarizing method according to claim 16, wherein the high-speed motion section is detected in a predetermined area of the frame image. 18. A pixel PN in a specific frame image IN (N is a natural number of 1 or more and less than M) by sampling a time-series frame image IM (M is a natural number of 1 or more) from a video to be summarized. And a luminance difference between a pixel around the pixel PN + 1 corresponding to the pixel PN in the next frame image IN + 1 is calculated, and the luminance difference is preset in each frame image unit. The pixel PN that exceeds the value θW1
18. The video summarization method according to claim 16, wherein the high speed operation section is calculated using the pixel PN. 19. A time-series frame image IM (M is a natural number greater than or equal to 1) is sampled from a video to be summarized, and a pixel PN in a specific frame image IN (N is a natural number greater than or equal to 1 and less than M) is obtained. , A first brightness difference which is a brightness difference between the pixel PN and the surrounding pixels, and calculates the pixel PN
And a second luminance difference which is a luminance difference between a pixel around the pixel PN + 1 corresponding to the pixel PN in the next frame image IN + 1 and the second luminance difference is calculated for each frame image. The pixel PN at which the brightness difference of 1 is less than the preset threshold value θW1 and the second brightness difference exceeds the threshold value θW1, and the high-speed operation section is calculated using the pixel PN. The video summarization method according to claim 16 or 17. 20. The video summarizing method according to claim 16, wherein the high-speed motion section is obtained by using a motion vector. 21. A frame image of a video to be summarized is divided into a plurality of shots, and the similar shots are integrated by using a correlation between the shots in a time series to be regarded as a shot group. A video summarizing method of selecting a plurality of partial moving images which are time-series frame images from a group and sequentially reproducing the partial moving images. 22. The video summarizing method according to claim 21, wherein a common feature is obtained from all shots included in the shot group, the partial moving image is processed based on the feature, and the processed partial moving images are sequentially reproduced. . 23. A common feature is obtained from all shots included in a shot group, a partial moving image is processed based on the feature, and a top frame image of the processed partial moving image is reduced and displayed on the screen. 23. The video summarizing method according to claim 22, wherein a list is displayed. 24. The video summarizing method according to claim 21, wherein shots in time series in which a similar background and a similar subject are photographed are detected and set as shot groups. 25. A representative spatiotemporal image, which is a frame image selected from shots, is processed, and the representative spatiotemporal image is divided into a moving region which is a region affected by a change due to motion and a region other than the moving region. The video summarization method according to any one of claims 21 to 24, wherein a color histogram is created for each static area, and a correlation between shots is obtained using the color histogram. 26. A partial moving image in a shot group using a common portion of moving constituent colors which are colors appearing in a moving area and a common portion of still constituent colors which are colors appearing in a still area. 26. The video summarizing method according to claim 25, wherein 27. The video summarizing method according to claim 21, wherein shots having substantially the same time length and continuous in time are detected and regarded as a shot group. 28. The method according to claim 8, wherein a portion taken continuously in time by the video camera is regarded as a shot.
7. The video summarization method described in any one of 7. 29. The video summarizing method according to claim 8, wherein one scene of the scenario is regarded as a shot.