WO2004047109A1

WO2004047109A1 - Video abstracting

Info

Publication number: WO2004047109A1
Application number: PCT/IB2003/004797
Authority: WO
Inventors: Francesco Pessolano; Mauro Barbieri
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2002-11-19
Filing date: 2003-10-27
Publication date: 2004-06-03
Also published as: AU2003274479A1

Abstract

A method of creating a collection of relevant video segments (201,203) by selecting respective portions (222-234, 242-254) from a video stream which corresponds to a program is disclosed. The collection of relevant video segments (201,203) can be applied as video trailer or as video abstract. Hence the duration of the collection of relevant video segments is relatively short compared with the duration of the program. The method comprises detecting a first non-relevant video segment (e.g. 202) in the video stream, which corresponds to a commercial-break in the program; and selecting a first one of the respective portions from the video stream, which is, in the time-domain, located in a neighborhood of the first non-relevant video segment, as a first one of the relevant video segments.

Description

Video abstracting

The invention relates to a method of creating a collection of relevant video segments by selecting respective portions from a video stream which corresponds to a program, a first duration of the collection of relevant video segments being relatively short compared with a second duration of the program. The invention further relates to a video segment compilation unit for creating a collection of relevant video segments by selecting respective portions from a video stream which corresponds to a program, a first duration of the collection of relevant video segments being relatively short compared with a second duration of the program.

The invention further relates to a video storage system comprising: - a receiving unit for receiving a video stream;

- storage means for storage of the video stream and for storage of a collection of relevant video segments being selected from the video stream; and

- a video segment compilation unit for creating the collection of relevant video segments, as described above. The amount of audio-video information that can be accessed and consumed in people's living rooms has been ever increasing. This trend may be further accelerated due to the convergence of both technology and functionality provided by future television receivers and personal computers. To select the audio-video information that is of interest, tools are needed to help users extract relevant audio-video information and to effectively navigate through the large amount of available audio-video information. To allow users to get a quick overview of the recorded audio-video information, and to decide whether to view an entire recorded program, an interesting feature is the automatic generation of video trailers. When a program has been or is recorded, the recorded program is analyzed in order to select relevant video segments from the video stream. By afterwards displaying the relevant video segments the user is provided with a nice overview of the recorded program.

An embodiment of the method of the kind described in the opening paragraph is known from the article "Video Abstracting", by R. Lienhart, et al., in Communications of the ACM, 40(12), pages 55-62, 1997. This article discloses that video data may be modeled in four layers. At the lowest level, it consists of a set of frames; at the next higher level, frames are grouped into shots or continuous camera recordings, and consecutive shots are aggregated into scenes based on story-telling coherence. All scenes together make the video. The concept of a clip is described as a frame sequence being selected to be an element of the abstract; a video abstract thus consists of a collection of clips. The known method comprises three steps: segmentation and analysis of the video content; clip selection and clip assembly. The goal of the analysis step is to detect special events such as close-ups of the main actors, gunfire, explosions and text. A disadvantage of the known method is that it is relatively complex and not robust.

It is an object of the invention to provide method of the kind described in the opening paragraph which is relatively easy. This object of the invention is achieved in that the method comprises:

- detecting a first non-relevant video segment in the video stream, which corresponds to a commercial-break in the program;

- selecting a first one of the portions from the video stream, which is, in the time-domain, located in a neighborhood of the first non-relevant video segment; and - associating the first one of the portions with a first one of the relevant video segments.

In general, broadcasters or other types of content providers insert commercial- breaks, also called advertisement breaks or commercial blocks, at those portions in the program where there is a peak in the viewer's attention. This editing technique is known as "cliffhanger". By selecting from the video stream those portions which precede or alternatively succeed the commercial-breaks a collection of relevant video segments, i.e. clips, is created. The commercial-breaks are non-relevant video segments. There are several commercial-break detection techniques known in the art which are based on video, audio or audio-video content analysis. The detection is e.g. based on detection of black-frames or uni- color frames as disclosed in WO 01/91123. Other techniques are based on cut-detection, logo-detection, i.e. logo present or not, format detection, i.e. letterbox or not, sub-title detection, i.e. subtitles or no subtitles. See for instance the article "Improving digital video commercial detectors with genetic algorithms", by D. Schaffer et al., in Proceedings of the Genetic & Evolutionary Computation Conference 2002, July 2002, New York. Also MPEG features are applied to detect commercial-breaks. See for instance the article "Real-time commercial detection using MPEG features", by N. Dimitrova et al, in Proceedings of IPMU 2002, July 2002, Annecy. Commercial-break detectors based on audio are e.g. disclosed in the patent applications with patent attorney docket numbers PHNL020407 (Filed May,2,2002, WO IB02/01639) and PHNL010339 (Filed May,l 1,2001, EP01201730.7). In both cases the amount of audio signal power is applied to detect commercials.

In an embodiment of the method according to the invention the first one of the respective portions precedes the first non-relevant video segment in the video stream. By selecting from the video stream those portions which precede the commercial-breaks a video trailer, i.e. teaser, is created. Because it is the interest of broadcasters to keep the viewers interested in the rest of the program, the segment immediately preceding a commercial-break usually represents interesting events and does not disclose key information, e.g. the name of the murderer in a detective film. h another embodiment of the method according to the invention the first one of the respective portions succeeds the first non-relevant video segment in the video stream. By selecting from the video stream those portions which succeed the commercial-breaks a video abstract is created which e.g. can be used as follows. If you miss an episode of your favorite television series, the video abstract can be used to become briefly informed what happened in the missed episode, hi that case, you don't have to find time to view the missed episode completely before you watch the next episode.

The length, i.e. duration, of the selected video segments might be equal to a predetermined value. But preferably the duration is controllable by a user. Optionally the duration of the video segments is related to the duration of the program or to the number of detected commercial-breaks in the program. In an embodiment of the method according to the invention the collection of relevant video segments is extended by adding further portions of the video stream. This adding of further portions is advantageous e.g. in the case that the number of commercial- breaks per time is below a predetermined threshold. These further portions of the video stream can be selected from the video stream on basis of time distances between commercial- breaks. Alternatively these further portions of the video stream are selected on basis of content analysis of the video stream, e.g. as disclosed in the article "Video Abstracting" by R. Lienhart, et al., in Communications of the ACM, 40(12), pages 55-62, 1997.

In an embodiment of the method according to the invention the order of the selected portions is changed. By viewing the selected portions in the correct order, i.e. corresponding to the order as in the original program, there is a probability that too much of the story is revealed. By changing the order this probability is reduced.

It is another object of the invention to provide a video segment compilation unit of the kind described in the opening paragraph which is arranged to create a collection of relevant video segments in a relatively easy way.

This object of the invention is achieved in that video segment compilation unit comprises:

- a commercial-break detector for detecting a first non-relevant video segment in the video stream, which corresponds to a commercial-break in the program; - a video segment selector for selecting a first one of the portions from the video stream, which is, in the time-domain, located in a neighborhood of the first non- relevant video segment; and

- an associating unit for associating the first one of the portions with a first one of the relevant video segments. It is another object of the invention to provide a video storage system of the kind described in the opening paragraph which is arranged to create a collection of relevant video segments in a relatively easy way.

This object of the invention is achieved in that the video segment compilation unit of the video storage system, comprises: - a commercial-break detector for detecting a first non-relevant video segment in the video stream, which corresponds to a commercial-break in the program;

- a video segment selector for selecting a first one of the portions from the video stream, which is, in the time-domain, located in a neighborhood of the first non- relevant video segment; and - an associating unit for associating the first one of the portions with a first one of the relevant video segments.

In an embodiment of the video storage system according to the invention the storage means comprises a hard-disk. In another embodiment of the video storage system according to the invention the storage means is arranged to store the video stream on a removable memory device, i.e. removable storage medium, like an optical-disk. A video segment compilation unit in accordance with the invention could be included, for example, in a television set, a computer, a video recorder (VCR), a DVD recorder, a set-top box, satellite- tuner or other apparatus in the field of consumer electronics. Modifications of the video segment compilation unit and variations thereof may correspond to modifications and variations thereof of the video storage system described.

These and other aspects of the method, of the video segment compilation unit and of the video storage system according to the invention will become apparent from and will be elucidated with respect to the implementations and embodiments described hereinafter and with reference to the accompanying drawings, wherein:

Fig. 1 schematically shows an embodiment of a recording and reproducing apparatus according to the invention;

Fig. 2 A schematically shows the creation of a video trailer on basis of a program, according to the invention; and

Fig. 2B schematically shows the creation of a video summary on basis of a program, according to the invention. Same reference numerals are used to denote similar parts throughout the figures.

A program might be a television program as broadcast by a television station, i.e. television broadcaster. Typically the television program will be watched by means of television sets. However a program might also be provided by another type of content provider, e.g. by means of the Internet. In that case the program might be watched by other types of equipment than television sets. Alternatively the program is not broadcast but exchanged by means of removable media, like optical-disks or cassette tapes. In this disclosure examples are described in which the program is a television program. It will be clear that the invention has a broader scope.

A television signal comprises picture information, sound information and additional information, such as for example teletext information. The television signal transmits a television program. The television program can comprise a movie or film, an episode of a series, a captured reproduction of a theater performance, a documentary or a sports program. These types of information of the television program may be interrupted by a plurality of units of commercial-break information and announcement information. Each commercial-break usually comprises a plurality of commercial messages. A commercial message usually has a length or commercial message duration of 10, 15, 20, 30, 40 or 60 seconds. Fig. 1 schematically shows an embodiment of a recording and reproducing apparatus 100 according to the invention. This recording and reproducing apparatus 100 is a hard-disk based video storage system. The recording and reproducing apparatus 100 is adapted to record a television signal FS contained in the received signal TS and to reproduce a recorded television signal AFS. The received signal TS may be a broadcast signal received via an antenna or cable but may also be a signal from a storage device like a VCR (Video Cassette Recorder) or Digital Versatile Disk (DVD). The received signal TS is provided by means of the input connector 110. The reproduced television signal AFS is provided at the output connector 112 and can be displayed by means of a display device, e.g. comprised by a television set.

The recording and reproducing apparatus 100 includes:

- a receiving unit 102 for receiving the signal TS. This receiving unit 102, e.g. tuner, is arranged to select the television signal FS of a television station. This television signal FS represents a video stream which corresponds to a television program 200; - A recording and reproducing means 106 for storage of the video stream as provided by the receiving unit 102. The recording and reproducing means 106 include a signal processing stage for processing the television signal FS to be recorded and for processing the reproduced television signal AFS, as is commonly known. This processing stage might include data compression. The recording and reproducing means 106 include a hard-disk as recording medium for the recording of the processed television signal FS.

- An exchange unit 104 for adaptation of stored information to a reproduced television signal AFS and for transmission of this a reproduced television signal AFS via the output connector 112, e.g. to a television set. The adaptation might include modulation on a carrier of the television signal FS representing the video stream. The stored information comprises the video stream as provided by the receiving unit 102 and a collection of relevant video segments; and

- A video segment compilation unit 108 for creating such a collection of relevant video segments by selecting respective portions from the video stream which corresponds to the television program. The purpose of this video segment compilation unit 108 is to create a video trailer or alternatively a video abstract of the video stream. Hence the duration of the collection of relevant video segments is relatively short compared with the duration of the television program. E.g. a television program takes about 1 or 2 hours and the duration of the collection of relevant video segments is in the range of seconds to minutes. That means e.g. from 10 seconds to 2 minutes. As a consequence each of the relevant video segments lasts only a few seconds. On user request the duration of the relevant video segments to be selected might be shorter or longer. It is not required that all relevant video segments have the same length. The creation of the collection of relevant video segments can be performed during the recording of the video stream or after the recording has finished. In the former case the video stream is provided by means of connection 114 and in the latter case the video stream is provided by means of connection 116.

The video segment compilation unit 108 comprises:

- a commercial-break detector 118 for detecting non-relevant video segments in the video stream, which correspond to respective commercial-breaks in the television program;

- a video segment selector 120 for selecting portions from the video stream, which are, in the time-domain, located in a neighborhood of the non-relevant video segments. That means that portions from the video stream are selected which precede or succeed the respective commercial-breaks. These portions can be adjacent to the commercial-breaks but preferably there is a gap or delay, i.e. time difference between the end of a commercial-break and the beginning of a selected portion or vice versa, in the range of a few seconds. The size of the time difference is configurable by the user; and

- an associating unit (122) for associating the selected portions (222-234, 242- 254) with the relevant video segments. The collection of relevant video segments can be stored as a number of copies of the respective portions of the original video stream. But preferably only a set of pointers is stored. The pointers indicate start or stop locations within the video stream corresponding to begin or end, respectively of the selected portions of the video stream. The collection of relevant video segments, as video data or as pointers, can be stored in the same memory device as applied for the storage of the original video stream or in a separate memory device. It will be clear that in the case of a recording and reproducing apparatus which is based on a removable storage medium it is preferred that both video stream and collection of relevant video segments are stored on the same storage medium.

The applied commercial-break detector 118 includes detection means adapted to detect black- frame information in the picture information. Black-frame information is present in the picture information when the luminance signal of the television signal has an amplitude value below a black level threshold value during a complete field. When the detection means detect black-frame information they supply black-frame time information representative of the time relationship or position of the detected black-frame information in the television signal. Black-frame time information supplied by the detection means is stored in memory. The commercial-break detector 118 further includes analysis means adapted to analyze whether the time difference between adjacent units of frame picture time iiiformation corresponds to a predetermined preset time difference. Thus, it is checked whether black- frame information is followed by further black-frame information within 35 seconds or whether black-frame information is followed by the subsequent black-frame information within a preset time difference of 43 seconds to 50 seconds. These preset time differences are based on the fact that commercials usually have a length of 30 seconds, 45 seconds or 60 seconds. It should be noted that alternative commercial-break detectors, as cited above, might be applied too. It is to be noted that a commercial-break is very often preceded and followed by a few scenes of another television film to be broadcast with the aid of a television signal in the near future by way of announcement information. It will be clear that these scenes should not be included in the collection of relevant video segments under construction. It is therefore desirable that this announcement information which may precede and follow a commercial- break in the television signal is skipped during creation of the collection. These scenes are usually characterized by black-frame information including a preceding fade-out and by a high rate of change of scenes. In accordance with the measures as defined for the commercial-break detector 118 it is checked whether a very high rate of change of scenes can be detected adjacent a detected commercial-break, h other words the commercial-break detector 118 is capable of detecting the presence of announcement information. The video segment selector 120 is controlled on basis of this detection. That might be by adapting the above mentioned size of the time difference between commercial-break and relevant video segment to be selected.

The commercial-break detector 118 and the video segment selector 120 may be implemented using one processor. Normally, these functions are performed under control of a software program product. During execution, normally the software program product is loaded into a memory, like a RAM, and executed from there. The program may be loaded from a background memory, like a ROM, hard disk, or magnetically and/or optical storage, or may be loaded via a network like Internet. Optionally an application specific integrated circuit provides the disclosed functionality.

Fig. 2 A schematically shows the creation of a video trailer 201 on basis of a television program 200, according to the invention. The television program 200 comprises a number of commercial-breaks 202-214. In general, these commercial-breaks 202-214 are not distributed equidistantly in the time-domain, but located at exciting moments of the television program. Very often the number of commercial-breaks in the second half of a television program is higher than the number of commercial-breaks in the first half of the television program. In many cases a commercial-break is located such that for the viewer a "question" arises just before the commercial-break starts and that the "question" is answered after the commercial-break. By selecting relatively small portions 222-234 of the video stream which precede the respective commercial-breaks, fragments of relatively important scenes of the television program are gathered. However the "answers" on the "questions" are not revealed. Hence, the collection of relevant video segments that precede the respective commercial- breaks is very well suitable as a video trailer. Typically the selected portions 222-234 of the video stream have a duration of a few seconds [1-10 seconds]. The duration of the selected video segments might be equal to a predetermined value. But preferably the duration is controllable by a user. Optionally the duration of the video segments is related to the duration of the television program. E.g. if the duration of the television program is long then the duration of the selected video segments might be relatively short. Optionally the duration of the video segments is related to the number of detected commercial-breaks in the television program. E.g. if the number of commercial-breaks is limited then the duration of the selected video segments might be relatively long.

Fig. 2B schematically shows the creation of a video summary 203 on basis of a television program 200, according to the invention. As said above, in many cases a commercial-break is located such that for the viewer a "question" arises just before the commercial-break starts and that the "question" is answered after the commercial-break. By selecting relatively small portions 242-254 of the video stream which succeed the respective commercial-breaks, fragments of relatively important scenes of the television program are gathered. Typically the "answers" on, most probably implicit, "questions" are revealed after the commercial-breaks. Hence, the collection of relevant video segments that succeed the respective commercial-breaks is very well suitable as a video abstract, because the highlights are disclosed. What is specified, in connection with Fig. 2 A, for the duration of video segments for the creation of a video trailer, substantially holds for the duration of video segments for the creation of a video abstract. However, typically the duration of a video segment for a video abstract might be longer than the duration of a video segment for a video trailer.

Optionally a collection of relevant video segments comprises both portions which precede commercial-breaks and portions which succeed commercial-breaks. These type of collections are typically useful as video abstract. It should be noted that the above-mentioned embodiments illustrate rather than limit the invention and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be constructed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements and by means of a suitable programmed computer. In the unit claims enumerating several means, several of these means can be embodied by one and the same item of hardware.

Claims

CLAIMS:

1. A method of creating a collection of relevant video segments by selecting respective portions from a video stream which corresponds to a program, a first duration of the collection of relevant video segments being relatively short compared with a second duration of the program, characterized in: - detecting a first non-relevant video segment in the video stream, which corresponds to a commercial-break in the program;

- selecting a first one of the portions from the video stream, which is, in the time-domain, located in a neighborhood of the first non-relevant video segment ; and

- associating the first one of the portions with a first one of the relevant video segments.

2. A method as claimed in claim 1, characterized in that the first one of the respective portions precedes the first non-relevant video segment in the video stream.

3. A method as claimed in claim 1, characterized in that the first one of the respective portions succeeds the first non-relevant video segment in the video stream.

4. A method as claimed in claim 1, characterized in that a third duration of the first one of the relevant video segments is related to the second duration of the program.

5. A method as claimed in claim 1, characterized in that a third duration of the first one of the relevant video segments is controllable by a user.

6. A method as claimed in claim 1, characterized in that a third duration of the first one of the relevant video segments is related to the number of commercial-breaks in the program.

7. A method as claimed in claim 1, characterized in that the collection of relevant video segments is extended by adding further portions of the video stream.

8. A method as claimed in claim 2, characterized in comprising changing the order of the selected portions.

9. A video segment compilation unit for creating a collection of relevant video segments by selecting respective portions from a video stream which corresponds to a program, a first duration of the collection of relevant video segments being relatively short compared with a second duration of the program, characterized in comprising:

- a commercial-break detector for detecting a first non-relevant video segment in the video stream, which corresponds to a commercial-break in the program;

- a video segment selector for selecting a first one of the portions from the video stream, which is, in the time-domain, located in a neighborhood of the first non- relevant video segment ; and

- an associating unit for associating the first one of the portions with a first one of the relevant video segments.

10. A video storage system comprising:

- a receiving unit for receiving a video stream;

- storage means for storage of the video stream and for storage of a collection of relevant video segments being selected from the video stream ; and

- a video segment compilation unit for creating the collection of relevant video segments, as claimed in claim 9.

11. A video storage system as claimed in claim 10, characterized in that the storage means comprises a hard-disk.

12. A video storage system as claimed in claim 10, characterized in that the storage means is arranged to store the video stream on a removable memory device.

13. A video storage system as claimed in claim 12, characterized in that the removable memory device comprises an optical-disk.