ES2399625T3 - Procedure and device for creating and viewing custom multimedia segments - Google Patents

Abstract

A procedure carried out by a multimedia device, the procedure comprising: receiving a stream (700A) of content in the multimedia device, in which the stream (700A) of content comprises a video stream, an audio stream and a stream of intraband data; while a video stream is presented to a user, in response to receiving a user indication deselecting a location (704, 706) within the content stream (700A), identify a sequence of a plurality of values (701) of calculated key generated from a portion of the intra-band data flow that precedes the selected location (704, 706), determine, for each calculated key value of the sequence, one or more hourly values that reflect a time in which it was generated the calculated key value, and storing, in a storage device, the sequence of key values (701) calculated in association with the determined hourly values and metadata that specify an action to be performed after recognition of the sequence of values (701) of key calculated in an intra-band data stream of a content stream (700A).

Description

Procedure and device to create and display personalized multimedia segments.

FIELD OF THE INVENTION

The present invention relates to the recognition of intra-band data in a multimedia content stream and the location of points within the multimedia content stream in a multimedia device.

BACKGROUND

The approaches described in this section are approaches that could be followed, but not necessarily approaches that have been previously conceived or followed. Therefore, unless otherwise indicated, it should not be assumed that none of the approaches described in this section entitle to its classification as a prior art merely thanks to its inclusion in this section.

Viewers can record TV programs broadcast using a cassette recorder (VCR). Therefore, a user of a VCR can record a particular TV program at the time it is broadcast and play the same recorded TV program at a later time. To achieve this, a VCR transforms the electrical signals that represent a TV program into magnetic signals and stores the magnetic signals on a magnetic tape. The magnetic tape is usually in the form of a cassette tape inserted into the VCR by the VCR user. When a VCR user decides to play the TV program, the process is reversed, and the VCR transforms the magnetic signals stored on the cassette tape into electrical signals and sends the electrical signals to a television.

With the development of digital technology, VCRs are being replaced by digital video recorders (DVRs). A DVR records TV programs broadcast for later playback by transforming electrical signals from the TV program into digital information and storing the digital information on a hard disk. When a user plays the recorded TV program, the DVR converts the digital information back into analog signals and sends the signals to the TV, which presents the TV program to the viewer. In addition, with the advent of digital televisions, the DVR can receive digital and analog signals and send digital signals directly to a digital television, which presents the TV program in digital format. Many content providers now use digital cable and satellite technology to broadcast digital programming. Digital programming is received by cable television receivers and / or satellite transceivers that are used to present digital programming to a user on a display screen. Thus, DVRs have the ability to receive digital programming signals from cable television receivers or satellite transceivers to record and present, as well as being incorporated into such devices.

DVRs have become increasingly popular and are widely used to record TV shows. More importantly, the use of DVRs to record TV programs has made content providers look for different means of presenting content to DRV users and enriching the experience of video programs. Personalized broadcast content has been absent in the content provider and DVR communities. US Patent No. 7,889,964, owned by the Applicant, describes a procedure for identifying the content of a broadcast using the area of subtitles for the deaf. US Patent No. 6,233,389, owned by the Applicant, also describes a procedure for recognizing specific words in the subtitle data for the deaf.

A technique to recognize differentiated events within an unmodified video program is desirable.

US 2005/0226601 A 1 discloses a technique for synchronizing an effect with a multimedia presentation. A "value obtained + time stamp" table includes pairs of values in which a value obtained, such as a CRC value, is associated for one or more segments of a recording, with a time stamp for such segment of the recording. An effect creator, programmer, content provider or other user creates a table of "effect signals + time point" that associates effects with locations in a content stream. During the multimedia presentation of the recording, at least one processor determines a time stamp input for a segment of the multimedia presentation, consults the time stamp input in the "effect signals + time point" table and triggers the effect which is associated with a time point when an interpolated time signal is sufficiently equal to the trigger time of the specified effect.

US 2004/0040042 A 1 discloses a technique for synchronizing the improvement of content with a video program using the subtitles for the deaf. An analyzer analyzes a sequence of subtitle execution for the deaf to identify a set of unique key phrases. A producer associates complementary data with the key phrases in the sequence of execution of the subtitles for the deaf that correspond to the desired program points. A design computer could calculate key values calculated from the key phrases in a data file to reduce the size of the data file. when the program is played, a viewer calculation unit monitors the sequence of subtitle execution for the deaf to

Detect the key phrases. Upon detection of a particular key phrase, the viewer's calculation unit

retrieves the complementary data associated with the particular key phrase. The present invention is defined by the independent claims. The dependent claims are they deal with optional features of some embodiments of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not limitation, in the figures of the attached drawings, in the that similar reference numbers refer to similar elements and in which: FIG. 1 is a block diagram illustrating a system for data recognition and synchronization

intraband;

FIG. 2 is a block diagram illustrating a general overview of components of a device for the recognition and synchronization of intra-band data; FIG. 3A is a flow chart that represents a procedure for generating sequence data of values of

calculated key;

FIG. 3B is a block diagram illustrating the procedure for generating sequence data of values of calculated key; FIG. 3C is a block diagram illustrating the procedure for generating sequence data of values of

key calculated using various subtitle schemes for the deaf; FIG. 3D is a block diagram that illustrates an icon that appears on a display screen; FIG. 4 is a flow chart representing a procedure for the recognition and synchronization of

subtitles for the deaf on a multimedia device;

FIG. 5A is a block diagram that illustrates a state machine to find pattern matches intraband data; FIG. 5B is a block diagram illustrating sequence data of calculated key values; FIG. 6 is a block diagram illustrating a procedure for synchronization with intraband data; FIG. 7 is a block diagram illustrating a procedure for creating and viewing multimedia segments within

a video program according to an embodiment; and FIG. 8 is a block diagram of a system in which embodiments of the invention can be implemented. DETAILED DESCRIPTION In the following description, numerous specific details are set forth for explanatory purposes to provide a thorough understanding of the present invention. However, it will be apparent that the present invention can be practiced without these specific details. In other cases, well-known structures and devices are shown in block diagram to avoid unnecessarily obscuring the present invention. Examples are described in this document according to the following scheme:

1.0

Overview

2.0

Structural overview

3.0

Approach to recognize intra-band data and synchronize with it

3.1 Subtitle data for the deaf

3.1.1. Generation of sequence data of calculated key values

3.1.2. Key sequence metadata calculated

3.1.3. Recognition of key sequences calculated on a multimedia device

3.1.4. Synchronization with subtitle data for the deaf

3.2.

Enhanced Television signaling data

4.0.

Focus to create and view custom multimedia segments

4.1. Selection of multimedia program segments by the user

4.2. Creation of user-initiated event identification data for multimedia programs

4.3 Creating an identified content of multimedia programs

4.4.

Sharing of event identification data for multimedia programs

5.0.

Implementation Mechanisms

1.0 SET OVERVIEW

The needs identified in the foregoing Background, and other needs and other objects that will become apparent from the following description, are satisfied in the present invention, which comprises, in one aspect, an intraband data recognition and synchronization system.

As mentioned above, the use of DVRs to record TV programs has caused content providers to seek different means of presenting content to DVR users and enriching the experience of video programs. For example, content providers who wish to present additional marketing or promotional content within video programs may cause particular icons, messages or interactive elements to be presented to the user in particular parts of the video program.

As described in US Patent No. 7,934,170, entitled "Method and Apparatus for Displaying Branded Video Tags", owned by the Applicant, a DVR may present portions of video program with commercial content presenting icons or related content during the issuance of the video program Thus, if the content provider wishes to promote a video program of early appearance, an interactive icon may be presented during the presentation of an advertisement detailing a video program of early appearance. While watching the ad, viewers can interact with the icon using a remote control to make the DVR automatically record the video program of early appearance that is being promoted.

For many of these additional features to work, a DVR must be synchronized with the flow of multimedia content to present additional content data in due course. Normally, the additional content is directly embedded in the multimedia content stream before the content is broadcast. In this way, the DVR processes the additional content along with the audio and video data within the multimedia content stream. By directly embedding the content in the multimedia content stream, the content provider is guaranteed that the additional data is presented to the user precisely at the place where the content provider requested it to be submitted. In addition, the multimedia content stream may be embedded with additional information not visible to the user to assist the DVR in synchronization. The above features are described in additional detail in US Patent No. 7,889,964, entitled "ClosedCaption Tagging System" and in US Patent No. 6,215,526, entitled "Analog Video Tagging and Encoding System", each of which is owned From applicant.

An example allows DVRs and other multimedia devices to synchronize with intra-band data within multimedia content streams. Intraband data is used to signal additional data on the same channel or "band" as used to send data, in this case audio and video data. An advantage of using intraband data for synchronization is that intraband data is normally intact within the content streams. Hence, since they are normally intact and unaltered, intraband data is a reliable source for synchronization with a multimedia content stream.

Examples of intra-band data include subtitle data for the deaf and Enhanced Television (ETV) signaling data. Subtitle streams for the deaf contain subtitle data for the deaf (usually, instructional data and subtitle text for the deaf) associated with video programs such as commercials, television shows, feature films and other multimedia content to present to a viewer. ETV signaling data defines instruction and control signals for interactive applications that are synchronized with ETV video programs.

When a multimedia device receives a stream of multimedia content that contains intra-band video and data programs, the device processes the intra-band data. The multimedia content stream may be broadcast on the air by a content provider or previously recorded by the multimedia device. During recording or playback of video programs, the multimedia device attempts to recognize intraband data patterns. Intraband data patterns can be provided from another device or other service provider as event identification data. Event identification data identify 4

intra-band data patterns that are associated with particular events in the flow of multimedia content. The event identification data can be stored in a storage device and can be accessed when intraband data is processed.

The multimedia device attempts to recognize intra-band data patterns using a correlation algorithm. According to one example, the correlation algorithm is implemented in a finite state machine. While listening for intra-band data within a multimedia content stream, the multimedia device uses the state machine to determine if the intra-band data matches a particular pattern of the intra-band data.

According to one example, event identification data, which is stored locally on the multimedia device, also includes metadata. Metadata identifies one or more actions to take in response to the pattern identification by listening to intra-band data from a multimedia content stream. If a match occurs, the multimedia device processes the metadata associated with the pattern and takes the appropriate action specified in the metadata. For example, the multimedia device may present an icon to a user on a display screen. Metadata can instruct the multimedia device regarding any necessary response if a user reacts to the icon. In addition, in response to the recognition of an intra-band data pattern, the multimedia device may store video segments associated with the pattern in a storage device. The multimedia device may also use the metadata associated with the recognized pattern to determine where a video segment begins or ends, or where a particular event occurs within the video program. In this way, a procedure for recognition and synchronization with intraband data is enabled.

In another aspect, a procedure for creating and viewing custom multimedia segments is provided. Users are provided on multimedia devices with a procedure by which they can select events

or portions within a video program for storage or playback. In a first multimedia device, a user selects one or more locations within a video program using a control interface such as a remote control. A multimedia device receives signals from the control interface and determines which locations were selected in the video program. The multimedia device then calculates an intra-band data pattern relative to the selected location (s) and stores the intra-band data pattern as event identification data in the same manner as described above.

Event identification data relating to the selected location (s) can be provided to another multimedia device. Using the event identification data, the other multimedia device attempts to identify intra-band data patterns associated with the video program to recognize the location (s) within the video program.

According to one example, the selected video program location (s) represent an event or a video segment. In addition, according to another embodiment, when the video segment is recognized, video identifications can be placed in the appropriate places within the multimedia content stream so that a user can jump directly to the site during video program playback.

In addition, in another example, once video identifiers representing the selected locations of the video program have been generated, subtitle data for the deaf can be extracted from the content stream and the content stream can be provided to a portable device. The portable device uses video identifiers to indicate the location or particular segments within the video program. Alternatively, instead of putting video identifiers in the multimedia content stream, an unedited version of the content stream may be provided to the multimedia device along with data representing the selected location (s). Data representing the selected location (s) may be provided separately from the multimedia content stream. A portable device can use the data to identify interesting locations within a video program and present the locations to a user. Alternatively, the portable device may use the data representing the selected location (s) in any appropriate manner.

In other aspects, the invention encompasses an apparatus and a computer-readable medium.

2.0 STRUCTURAL VISION OF ASSEMBLY

Throughout this description, several terms will be used to refer to particular concepts whose definitions are provided herein. Specifically, "intraband data" is signaling data sent by the same channel or the same "band" that is used to send audio and / or video data. Collectively, intraband data is contained in an "intraband data flow". According to one embodiment, the intra-band data stream may include subtitle data for the deaf or Enhanced Television (ETV) signaling data in a multimedia content stream. "Intraband data patterns" includes any intraband data that appears in a particular segment of the intraband data flow. An intra-band data pattern identifies a sequence of intra-band data selected from a portion of the intra-band data flow. For example, the intra-band data pattern may be a collection of subtitle data for the deaf or ETV signaling data that encompass a particular segment of the intra-band data flow. "Event identification data" is data that identifies one or more intraband data patterns. In addition, event identification data may also include additional data such as metadata describing events within the multimedia content stream, as well as instructional data for taking action in response to the recognition of intra-band data patterns. Event identification data can be provided to multimedia devices, which use the event identification data to recognize intra-band data in a multimedia content stream and locate points within the multimedia content stream.

With reference to FIG. 1, according to an example, a system for recognition and synchronization with intraband data is shown. The system contains the DVR 102 and 104, each of which is coupled in communication with the network 105 through any appropriate communications interface, such as an Ethernet or wireless communication port. In addition, the system includes server 106A, content provider 1068, personal computer101A and portable device 1018.

The personal computer 101A can be a personal calculating device, such as a desktop or portable computer, and is also coupled to the network 105 through any appropriate interface. The personal computer 101A is also connected to the portable device 1018. The portable device 1018 is a handheld calculation device, a mobile phone, a portable media player or any other portable device capable of displaying multimedia content. Through the network 105, the DVRs 102 and 104, the personal computer 101A and the portable device 1018 each communicate with the server 106A and the content provider 1068.

The network 105 can be implemented by any means or mechanism that allows the exchange of data between devices in the communications system. Examples of network 105 include, without limitation, a network such as a local area network (LAN), a wide area network 0f'JAN), Ethernet or Internet, or one or more terrestrial, satellite or wireless links. Alternatively, or in addition, any number of devices connected to the network 105 can also be connected directly to each other via a communications link.

In one example, DVR 102 and 104 communicate with server 106A, which provides program guide data, graphic resources (such as fonts, images, etc.), service information, software, advertisements, event identification data. and other forms of data that allow DVR 102 and 104 to operate independently of server 106A to meet the interests of viewers.

The content provider 1068 can also provide multimedia program content directly to DVRs 102 and 104 through air broadcasts, satellite transmissions or coaxial cable. The multimedia program content includes content such as feature films, television shows, variety shows, talk shows, advertisements, etc., for presentation to DVR users. In addition, content provider 1068 also provides additional data to server 106A, including promotional data, icons, electronic page data and other information for the server 106A to interpret and provide to DVR 102 Y

104. According to another example, a content provider 1068 also provides multimedia program content to server 106A for processing. Server 106A processes multimedia program content, creates event identification data and provides event identification data to DVRs and other devices.

Multimedia devices use event identification data to recognize and synchronize with multimedia content streams. With reference now to FIG. 2, a block diagram illustrating the components of a device for recognition and synchronization with intraband data according to an example is shown. In particular, FIG. 2 represents the components of a DVR 200. Note that, for example, reference will be made to a DVR throughout the description. However, the invention is not limited in that regard, and the process of recognition and synchronization with intra-band data can be implemented in any type of multimedia device or computer system.

The DVR 200 generally comprises a plurality of components, signified by the input module 202A, for receiving a multimedia program content from a content provider 1068 or a server 106A. According to one example, the broadcast signals representing a multimedia content stream are sent to the DVR 200 from the content provider 1068. The DVR 200 receives the broadcast signals that represent a flow of multimedia content from an antenna, a cable TV system, a satellite receiver or any other transmission system via the input module 202A. The input module 202A may comprise a plurality of tuning modules that allow a plurality of signals to be received and recorded simultaneously. In addition, the 202A input module takes TV input streams in a multitude of forms, for example a broadcast on the National Television Standards Committee (NTSC) or PAL system, and digital forms such as the Digital Satellite System (DSS) ), Digital Broadcasting Services (D8S) or the Advanced Television Standards Committee (ATSC). D8S, DSS and ATSC are based on standards called Group 2 of Experts in Motion Pictures (MPEG2) and Transport of MPEG 2 Transport. MPEG 2 Transport is a standard for formatting the digital data stream from a TV source transmitter so that the TV receiver can disassemble the input stream to find programs in the multiplexed signal.

The DVR 200 also includes a recorder module 204, which is functional for recording one or more portions of a multimedia content stream received from the input module 202A and storing the multimedia content stream as a digital data stream in the device 214 of storage. The recording module 204 records the incoming stream of multimedia content by storing the digital data stream in at least one storage component, such as the storage device 214, which is designed to retain segments of the digital data stream. According to one embodiment, the storage device 214 may be a hard disk drive located in the DVR 200. Alternatively, the storage device 214 may be any type of volatile and / or non-volatile storage medium, such as flash memory, and It can also consist of removable storage media, such as a DVD-ROM disc. Note that the DVR 200 may contain multiple storage devices, storage media or removable storage media of the type described above in any combination thereof.

During playback, the digital data stream is converted into an analog signal and then modulated into an RF carrier by means of the output module 2028 through which the signal is delivered to a standard television. The 2028 output module can alternatively deliver a digital signal to a television or video monitor. For example, the DVR 200 can use a Digital Visual Interface (DVI) port to send digital signals to a TV via a DVI cable. Alternatively, the 2028 output module can deliver analog and / or digital signals internally to a TV without the use of any cables. For example, the DVR 200 may be part of a television, so that communication between the DVR 200 and the display screen is done internally through a system bus located on the television.

To record and store multimedia content streams, as well as other types of data, the DVR 200 includes a storage device 214. The storage device 214 can be a hard disk, a memory or any other form of storage located in the DVR 200. The storage device 214 can be used to store not only program material or multimedia content streams, but also additional data for the use by the DVR 200. For example, event identification data may be stored in the storage device 214 for future recovery. Although the storage device 214 appears as a module within the DVR 200, the storage device 214 may alternatively be located outside the DVR 200, and the DVR 200 can communicate with the storage using an interface such as the communications interface 206.

The DVR 200 also includes a communications interface 206, through which the DVR 200 communicates with the network 105 via Ethernet, wireless network, modem or other communications standards. In other examples, the communication interface 206 may be any type of communications port, such as a coaxial cable, infrared, fiber optic or serial port. According to one example, data is communicated from the server 106A over the network 105 to the DVR 200 using an Ethernet cable connected to the communication interface 206. Using the communications interface 206, the DVR 200 can communicate with a PC 101A or the portable device 1018 through a

LAN

The 202A input module receives multimedia content streams for recording and playback for DVR users. When multimedia content streams are received in the input module 202A, they are recorded by the recorder module 204 and processed for presentation to the user via the output module 2028. Processing of the multimedia content stream for viewing includes sending the multimedia content stream to the storage device 214 and then to the output module 2028 for viewing. In fact, the DVR records portions of the multimedia content streams during live playback, allowing a user to pause and rewind a live broadcast without recording the entire video program. The processing of the flow of multimedia content is further described in more detail in US Patent No. 6,233,389, entitled "Multimedia Time Warping System", owned by the Applicant.

During operation of the DVR 200, the analysis module 210 listens for intra-band data, such as subtitle data for the deaf or ETV signaling data. While the multimedia content stream is analyzed, the analysis module 210 monitors the portion of the intraband data stream. Alternatively, the analysis module 210 can monitor intraband data at any other time, such as in the reproduction of the multimedia content stream.

According to one example, the multimedia content stream is an MPEG transport stream and the intraband data is interspersed with video audio data as a private data event. Since intraband data is a discrete event within the content stream, it is easily recognizable by a multimedia device such as the DVR 200. In addition, in the case where the original broadcast is in an analog format, the DVR 200 contains a plurality of components necessary to digitize multimedia content streams in the input module 202A, so that the multimedia content stream is in the form of an MPEG transport stream when processed by the analysis module 210. Intraband data is extracted while video and audio are placed in a packaged elementary stream (PES), as further described in patent 6,233,389. The intraband data is then combined with the audio and video for processing by the analysis module 210 and for presentation by means of the output module 2028.

To synchronize with multimedia content streams, the DVR 200 also contains a pattern recognition state machine 212. In one example, the pattern recognition state machine 212 represents an algorithm for the recognition of intra-band data patterns within the multimedia content stream. the pattern recognition state machine 212 processes the event identification data, which includes known intra-band data patterns, in an attempt to identify the intra-band data with known intra-band data patterns.

According to one example, the pattern recognition state machine 212 is a state-based algorithm that attempts to identify intra-band data from the analysis module 210 with intra-band data patterns in the event identification data stored in the storage device 214 . If a match occurs, the pattern recognition state machine 212 reports the match to the DVR 200, which inspects the metadata associated with the recognized intra-band data pattern. Based on the metadata, the DVR 200 can take any number of actions in synchronization with the flow of multimedia content.

Note that although the components of the DVR 200 have been described in relation to a DVR, the above modules may be integrated into a TV system, so that the modules are housed in a television capable of performing the functions of each module of the DVR 200. In addition, in other examples, the modules described above may be located among any number of devices capable of performing the described functions. Thus, through the operation of the described components, the DVR 200 can recognize intra-band data and synchronize actions and events with the intra-band data.

3.0 APPROACH TO RECOGNIZE INTRABAND DATA AND SYNCHRONIZE WITH THE SAME

As mentioned above, intraband data can take many forms. Two such forms include subtitle data for the deaf and ETV signaling data. Note, however, that the invention is not limited to subtitle data for the deaf or ETV signaling, and that it can be applied to other types of intra-band data.

3.1 Subtitle data for the deaf

One type of intra-band data within a multimedia content stream is subtitle data for the deaf. Subtitle data for the deaf are data that represent subtitle text for the deaf for presentation to a user, as well as subtitle control for the deaf in conjunction with video programming. Multimedia devices can recognize subtitle data for the deaf and synchronize actions and events with the subtitle data for the deaf in various ways using the processes as described herein.

A multimedia device can use subtitle data patterns for the deaf to recognize and synchronize with multimedia content streams. The types of data patterns available in subtitle data for the deaf are numerous. For example, there may be differentiated data patterns within the subtitle text for the deaf itself, the control data for the subtitles for the deaf, as well as any other event defined by the subtitle data for the deaf. By recognizing differentiated patterns within subtitle data for the deaf, a DVR can identify events within the multimedia content stream.

One way to recognize patterns within the subtitle data for the deaf is by calculating calculated key values that represent the text of the subtitles for the deaf and identifying patterns of the sequences of calculated key values. Thus, according to an example, in a multimedia device, such as a DVR or a server, the subtitle data for the deaf is analyzed and calculated key values corresponding to the subtitle data for the deaf are generated. The calculated key values are then compiled into sequences of calculated key values with particular video programs or segments, and are also combined with metadata that defines instruction and control information for processing in multimedia devices.

These calculated key sequences and these metadata are provided to multimedia devices such as DVR in the form of sequence data of calculated key values. Multimedia devices use the sequence data of calculated key values to recognize and synchronize with subtitle data for the deaf. The multimedia device uses an identification algorithm to sequentially compare the generated calculated key values from the subtitle data for the deaf with multiple calculated key sequences that the multimedia device has stored locally. According to one example, the identification algorithm is implemented by means of a state machine that processes the generated values of the calculated key and informs whether or not a match has occurred in a sequence of calculated keys identified by the sequence data of key values. calculated

3.1.1 Generation of sequence data of calculated key values

According to one example, to recognize subtitle data for the deaf and synchronize actions and events with the subtitle data for the deaf in a computer system, the calculated key value sequence data is generated and provided to devices such as DVR 102 and 104 Each DVR 102 and 104 also receives one or more multimedia content streams that contain video programs for presentation to a user. As defined herein, a video program may be a feature film, a variety show, an interview program, a music video, a telecomedy program, commercials, a live television broadcast or any other multimedia content. reproduced for a user of a DVR.

Subtitle data for the deaf, according to one example, may be contained in a separate stream of data associated with the stream of multimedia content. For example, if a program is in digital form, subtitle data for the deaf are communicated within the MPEG2 stream as part of a private data channel. However, when communicating in an analog format, subtitle data for the deaf are modulated, instead, in the Vertical Suppression Interval (V81) of the analog TV signal in several standard ways; for example, the North American Teletext Broadcast Standard (NA8TS) can be used to modulate information on lines 10 to 20 of an NTSC signal, while the FCC orders the use of line 21 for subtitles for the deaf (SS) and Extended Data Services (EDS). However, analog subtitle signals for the deaf are decoded by the DVR input module and are passed to the other components of the DVR as if they were distributed through a private MPEG2 data channel. Therefore, as described above, the analysis module 210 of the DVR 200 can process the subtitle data for the deaf that were originally transmitted in analog or digital format. More information on the location of subtitle data for the deaf in multimedia content streams can be found in the US application with serial number 09 / 665.921, entitled "Closed-Caption Tagging System", owned by the Applicant.

In another example, subtitle data for the deaf can be contained in any detectable form within the multimedia content stream. Server 106A and DVR 200 can read and analyze subtitle data for the deaf.

As part of the process of recognizing subtitle data for the deaf and synchronizing actions and events with the subtitle data for the deaf in a multimedia content stream, each DVR 102 and 104 also receives event identification data. According to one example, the event identification data is sequence data of calculated key values. Sequence data of calculated key values may come from various sources such as server 106A. For example, server 106A can process a program content in advance by calculating calculated key values and determining calculated key sequences for use by multimedia devices. Program content may be provided by content provider 1068 to server 106A before it is first broadcast to multimedia devices. The server 106A can provide calculated key sequences to the multimedia devices before the multimedia devices receive the content of the associated program. Alternatively, the sequence data may come from another computing device, such as another DVR.

Therefore, with reference now to FIG. 3A, a flowchart depicting a procedure for generating data from a sequence of key values calculated according to an example is shown. In step 302, server 106A receives data representing one or more video programs from content provider 1068. The data may be in the form of multimedia content stream or may be individual data from video programs separate from the multimedia content stream. Together with the video program data, the content provider 1068 also supplies the server 106A with subtitle data for the deaf associated with the video program. In one example, the subtitle data for the deaf may be embedded within a multimedia content stream containing the video program provided by the content provider 1068. In addition, the content provider 1068 may provide metadata containing event information associated with video programs.

Server 106A defines information that specifies events or actions on which the multimedia device operates while presenting a video program to a user. This information may include references to graphic icons or additional content to show the user. The content or icons would be present on the multimedia device

or they would be obtainable by the device through the Internet, the network or another DVR. In addition, the information may present data representing certain events within the video program. For example, if a sports broadcast were divided into separate times and a break, the content provider 1068 could provide event information to the server 106A indicating where the break occurs in the flow of multimedia content. Using this information, server 106A can generate sequence data of calculated key values to allow a DVR to automatically identify the beginning and end of the break within the video program. Since the rest has been identified, a DVR user can request the DVR to automatically skip the rest when viewing the recorded broadcast.

In addition, according to an example, the server 106A can obtain subtitle data for the deaf for video programs from any multimedia device, such as a DVR. For example, DVR 102 may have previously recorded a video program containing subtitle data for the deaf. The DVR 102 can send the recorded video program to server 106A over network 105, including subtitle data for deaf people associated with the video program for processing by server 106A.

In step 304, the server 106A analyzes the subtitle data for the deaf of the video program by converting it into calculated key values, which are used to create sequence data of calculated key values. Server 106A can use several different procedures to create calculated key values. The procedure used to create calculated key values for subtitle data for the deaf may vary according to the procedure used by the subtitle stream for the deaf to display the subtitle text for the deaf to a viewer of the video program.

Thus, server 106A tracks which subtitle mode for the deaf is being used to display the subtitle data for the deaf. For example, according to one example, the subtitle text for the deaf can be processed and displayed to a user using the sudden appearance mode, the ascending appearance mode or the background appearance mode, as further described herein. document. Depending on the mode used, a calculated key value represents the result of applying a calculated key function to several characters in a line of text. A calculated key function H is a transformation that takes an input m of variable size and returns a fixed-size string, which is called the calculated key value h; that is, h = H (m). The variable size input for subtitle data for the deaf will depend on the subtitle procedure for the deaf that is used, as will be further described herein. In addition, according to one example, the number of characters that can be put on a subtitle data line for the deaf is limited.

However, the analyzer 312 can be configured to accept any number of characters in a line of text to accommodate different systems. Thus, as described in the following, the analyzer 312 determines when to generate calculated key values in response to the detection of differentiated events within the subtitle data for the deaf. Differentiated events within the subtitle data for the deaf can be instructions such as a "car return" instruction. A "car return" instruction can be used to define the end or beginning of a line of text. Thus, for example, when a line of text has been completed, a "car return" will cause the cursor position to be placed on the next line.

The calculated key function is useful for creating a standardized value of calculated key of varying text sizes. According to one example, the characters may represent hexadecimal numbers similar to those defined by the US Standard Code for Information Exchange (ASCII). In addition, some subtitle implementations for the deaf may use a modified ASCII standard for the presentation of alternative symbols, such as musical notes during audio in a video program.

According to one example, the analysis module 210 applies the calculated key algorithm to the hexadecimal representations of the subtitle data for the deaf to produce calculated key values. In addition, the analysis module 210 adjusts the procedure for calculating key values calculated depending on the subtitle procedure for the deaf being used.

With reference to FIG. 38, a block diagram illustrating the procedure for generating sequence data of key values calculated according to an example is shown. Frames 310A-310D represent frames of a video program within a stream of multimedia content received on server 106A. According to one example, video programs are played at a rate of thirty frames per second. Each frame 310A-310D is reproduced at a particular time according to the speed of the frames. For example, frame 310A occurs at the time =

1:00 and plot 3108 occurs at the time = 1: 05. Hence, assuming a frame rate of thirty frames per second, frame 3108 occurs at frame number 151, or 150 frames after frame 310A.

Note that frames 310A-310D are only four different frames between many frames within the multimedia content stream, and many frames may exist between each of frames 310A and 310D. In frames 310A-310D, calculated key values are generated for the A-D text, respectively. The A-D text is text contained within a subtitle stream for the deaf, and frames 310A-310D represent the moment at which the calculated key values for the text are generated. Note, however, that the text of subtitles for the deaf usually appears longer than one frame at a time. According to one example, the transmission rate for subtitle data for the deaf is two bytes per video frame, and the transmission or reception of a full line of text takes several frames. Text 8, for example, can be displayed in multiple frames that precede frame 3108. However, frame 3108 is the position within the flow of multimedia content in which a calculated key value is generated for text 8.

In addition, since each frame 310A-310D is displayed at a particular time, metadata can be created that contain relative time data that represents an instant within the video program or the multimedia content stream in which the frame is being displayed. thus, frame 310A is displaying text A at 1: 00, while frame 3108 is displaying text 8 at 1: 05. Analyzer 312 reads the A-D text of each frame 310A-310D to generate the calculated key values 314A-314D and the metadata associated with those calculated key values, respectively. However, note that it is not necessary to create metadata every time a calculated key value is generated.

In another example, parser 312 can analyze additional frames that do not display any text. However, since these frames do not contain any subtitle text for the deaf, the generation of a calculated key value will not be triggered. Rather, as described in further detail herein, the analyzer 312 does not generate a calculated key value until a trigger event has occurred. Depending on the subtitle procedure for the deaf being used, the trigger event will vary from case to case. For example, according to one example, the trigger event is the identification of a "carriage return" instruction within the subtitle data for the deaf. When the "carriage return" instruction within the subtitle data for the deaf is detected, the analyzer 312 generates a calculated key value representing the one or more lines of subtitle text for the deaf that span several seconds within the program of video.

Table 320 represents an enumeration of calculated key values generated by the video program analyzer 312 represented by frames 310A-310D. Table 320 includes a Value column of the key cah; ulada, a Time column and a Time Offset column. The Value of the calculated key column represents the calculated key values generated by the analyzer 312 for frames 310A-310D. The Time column represents the relative time at which each key value calculated according to the due process of subtitles for the deaf was generated. The hourly value may be relative to the entire multimedia content stream, to the particular 310A-310D frames that contain the video program or to the real-time clock of the DVR.

When the analyzer 312 generates a calculated key value for any frame, the analyzer 312 interrogates a timer module 209 of the DVR 200 to determine the moment at which the calculated key value was generated. While the 312 analyzer uses the same relative time unit to generate the time in the Time column, the DVR 200 can accurately track any time variation between the generated values of the calculated key and the data of the sequence of calculated key values. The Time Offset column indicates the difference between the hourly data associated with a calculated key value and the hourly data associated with a subsequent calculated key value. For example, it was documented that the calculated key value "3059" has no time offset value. In this case, it is determined that the calculated key value "3059" is the beginning of a sequence of calculated keys, so no time offset value is required. However, for the calculated key value "2Z55", a time offset value of: 05 seconds is documented, indicating that the calculated key value "2Z55" is generated approximately: 05 seconds after the generated calculated key value "3059". This information can be used to locate the beginning of a video program or, alternatively, any particular event or location within the video program on a DVR. The process of using the time offset information to synchronize a computer system with a video program is discussed in more detail below in section 3.3.

Thus, the analyzer 312 analyzes the subtitle data for the deaf within the video program. With reference again to FIG. 3A, once the analyzer 312 has analyzed the subtitle data for the deaf of a video program by converting it into calculated key values, the server 106A goes on to generate calculated key sequences associated with the video program in step 306. When the analyzer 312 has completed the generation of a particular number of key values calculated for a video program or a segment of a video program, the analyzer 312 creates sequence data of key values calculated from the calculated key values generated.

The number of key values calculated in a sequence of calculated keys, as well as the amount of time represented by the calculated key values can be arbitrarily set by the analyzer 312, the server 106A or by any other device or user that wish control the creation of sequence data of calculated key values. For example, if the content provider 1068 provided a particular video program for analysis to the server 106A, the content provider 1068 may also indicate that, during a particular portion of the video program that for five to ten seconds, a on-screen icon requesting user intervention. FIG. 3D is a diagram illustrating the appearance of such an icon on a display screen according to an example. According to FIG. 3D, the display screen 400 shows an icon 404. The icon 404 contains a dialog requesting the intervention 404A of the user. In this example, the content provider 1068 asks the server 106A for the 404 icon to appear on the display screen 400 during a particular portion of a video program. Here, the analyzer 312 can be configured to generate calculated key values that span at least 10 seconds of the selected portion of the video program. This ensures that calculated key values are generated for at least the selected portion of the video program, giving a computer system sufficient time to recognize the complete sequence of calculated key values and to positively identify the program or the video portion. In addition, to compensate for any delay or margin of error, the analyzer 312 can be configured to generate calculated key values that cover more than the selected portion of the video program.

Alternatively, the length of any sequence data of key values calculated can be determined by the uniqueness of each calculated key value generated within the sequence of calculated keys. For example, in some situations, the same subtitle data for the deaf can result in the same calculated key value. If two video programs contain very similar subtitle data for the deaf, such as two ads with very little subtitle text for the deaf, they can start by generating the same key values calculated for the first portion of the program. However, the longer the analyzer generates 312 calculated key values for a particular video program, the more likely the calculated key sequence is unique. Therefore, the analyzer 312 can be configured to select any number of calculated key values to include them in a calculated key sequence to control the accuracy of the coincidence of sequence data of key values calculated with the video programs. . In addition, when the analyzer 312 generates the calculated key values, the server 106A can associate metadata with the sequence data of calculated key values. Associated metadata may indicate that an icon 400 is displayed requesting user intervention, as set forth in the preceding example.

When calculated key values are generated, the analyzer 312 includes hourly data associated with the time at which each calculated key value was generated. However, the same subtitle data for the deaf or even the same calculated key value can be generated at different times according to the subtitle procedure for the deaf used.

Thus, with reference to FIG. 3C, different subtitles procedure for the deaf are shown according to an example. Respectively, a user is shown subtitle data for deaf people in 340 mode of sudden appearance, in 342 mode of ascending appearance or in 346 mode of appearance in the background. FIG. 3C shows an area of incoming SS data of a subtitle stream for the deaf and a display area that represents the actual output or display of the subtitle text for the deaf for a user.

3.1.1.1 Sudden appearance mode

In the 340 mode of sudden appearance, according to one embodiment, between one and four lines of text appear on the screen all at once, they remain there for a period of time and then are replaced by the following subtitle text lines for the deaf.

In sudden appearance mode 340, incoming subtitle data for the deaf are placed in the subsequent buffer 3508 and displayed with a single "dump buffers" instruction once the full text for a particular frame or series of frames are ready to be displayed. In this mode, a calculated key value is generated for the text that is being displayed when the "dump buffers" instruction is issued. When the "dump buffers" instruction is issued, the text in the subsequent buffer 3508 is transferred to the previous buffer 350A, which represents the text being displayed on the display device 360.

In the sudden appearance mode, there are only two well-defined events that change the visible text on the display device 360 for the user. These events include (1) the dump of the buffers, so that the content of the subsequent buffer 3508 is placed in the previous buffer 350A, and (2) the deletion of the text shown on the display screen 360. Generating calculated key values in the sudden appearance mode is merely a matter of keeping track of the composite text in the subsequent buffer and creating a calculated key value for the text in the subsequent buffer 3508 once they have been processed. dump instruction or an erased instruction from the displayed buffer.

Thus, in the sudden appearance mode, the analyzer 312 will keep track of the text that is being composed in the subsequent buffer 3508. While monitoring the subsequent buffer 3508, the analyzer 312 continues to update a calculated key value corresponding to the value total of any text accumulated in the back buffer 3508. Initially, the back buffer 3508 is empty. While the text is being composed by entering it on each line, the analyzer 312 waits until a "cursor placement" instruction is issued. A cursor placement instruction can be used to place the text cursor on a new line. Based on the cursor placement instruction, the analyzer 312 can determine that a line is complete and calculates a key value calculated for the particular line.

At the end, the previous buffer 350A is emptied and the subsequent buffer 3508 is turned over to the previous one by issuing a dump instruction of the buffers. Note that the rear buffer 3508 can contain multiple lines of text. To generate a single key value calculated for multiple lines of text, a cumulative key value calculated for all lines of text that appears in the previous buffer 350A is generated. The calculated cumulative key value is created once the subsequent buffer 3508 is dumped to the previous buffer 350A. Thus, while text is being compiled in the subsequent buffer 3508, the analyzer 312 tracks the individual key values calculated for each line of text as described above. When the previous buffer 350A is turned over, the calculated key values for each line are combined with each other to create the calculated cumulative key value.

According to one example, when the analyzer 312 generates the calculated key value, the analyzer 312 also retrieves the hourly data associated with the multimedia content stream from the timer module 209. The timer module 209 can indicate the timing of the relative content stream of form relative to the beginning of the flow of multimedia content, one hour of calendar clock, the time since the last key value calculated

or the time relative to the beginning of a video program within the multimedia content stream. Hence, with reference again to FIG. 38, for example, the analyzer 312 determines that the text A of the frame 310A has been dumped to the previous buffer 350A at the time = 1: 00 and that it has a corresponding calculated key value of "3059".

3.1.1.2 Ascending appearance mode Another way of presenting subtitle text for the deaf is the ascending appearance mode 342. In 342 mode of ascending appearance, which is more common in live broadcasts, such as news or talk shows, text is typed as it is posted and appears on the TV as soon as it is received. Once a "carriage return" instruction is received, the text goes up. A carriage return instruction is an instruction that causes the cursor to return to the beginning of the line. Since the cursor has returned to the beginning of a new line, a carriage return can be used to indicate that a new line is being composed. According to one example, text can initially be composed in the display device 360 such that the frame X thus includes line 1 of text, while the incoming data 350C of SS contains text that is subsequently composed, such as lines 2- 4 of text. When line 2 of text is ready to be displayed, line 2 of text "ascends" to the screen, causing line 1 of text to move up in frame Y of display device 360. In one example, the Y plot only allows two lines of text to be presented at a given time. Thus, in any subsequent frame, when line 3 of text is ready to be displayed, line 1 of text would be deleted and line 2 of text would move up, occupying the second row of text on display device 360. Alternatively, any number of subtitle lines for the deaf can be provisioned in ascending appearance mode.

According to one example, to generate key values calculated in the ascending appearance mode, the analyzer 312 monitors the text lines and listens for a carriage return instruction. Once the analyzer 312 determines that such an event has occurred, the analyzer 312 generates a key value calculated for the corresponding text. Thus, a calculated key value is generated when a line of text is completed, and subtitle data for the deaf contain a distinct event visible to the user that marks the end of the line. Instead of creating a cumulative key value calculated for all lines of text displayed on the screen, the ascending appearance mode generates and documents a key value calculated for each line of visible text as it appears on the screen.

For example, according to one example, parser 312 can determine that a line of text is complete when the text ascends a line; for example, line 1 of text in frame Y. Analyzer 312 can detect that a line of text is complete when it detects a "carriage return" instruction within incoming data 350 of SS. Hence, with reference again to FIG. 38, for example, analyzer 312 determines that a line of text corresponding to text 8 is complete after detecting a "carriage return" instruction in frame 3108 and calculates a calculated key value of "2Z55" at the time = 1 : 05.

3.1.1.3 Background appearance mode

Another subtitle mode for the deaf is 344 mode of background appearance. The background 344 mode of appearance in the background is similar to the sudden appearance mode 340, except that the incoming SS data 3500 is composed directly in a previous buffer and, since the content of the previous buffer is fed directly to the device Viewing, a viewer can see the text as it is composed. In one example, to generate calculated key values, the analyzer 312 waits until the first event visible to the user occurs after an entire line of text is completed. Thus, in background mode 344, a procedure for generating calculated key values includes the accumulation of a line of text as it is being displayed and waiting for the first event visible to the user after the line is complete to generate the calculated key value for that line. The first event visible to the user could be the start of another line, the deletion of the screen or an instruction to "dump buffers" to show the new text in the sudden appearance mode.

3.1.2 Calculated key sequence metadata

Regardless of the type of subtitle mode for the deaf being used, once the analyzer 312 generates the calculated key values associated with a video program, the calculated key values are placed in table 320 along with any additional data , such as the time and time offset data for each key value calculated. The cumulative data in table 320 thus represents sequence data of values generated by a device such as server 106A.

Once the data of the calculated key value sequence has been generated, additional metadata can be associated with the calculated key sequence. According to one example, additional metadata may include instructional data and control data to take some action upon recognition of the sequence of calculated keys.

Thus, with reference again to FIG. 38, table 330 includes columns representing calculated key sequences and metadata associated with the calculated key sequences. According to one example, the sequence of calculated keys 3059, 2Z55, A6E3 and 9Y95 includes metadata describing an EVENT 1 event. EVENT 1 can be an event within the multimedia content flow that occurs at a particular time. For example, EVENT 1 can define a particular location within a video program, such as a break during a sporting event. The metadata that defines EVENT 1 may also include additional information, such as a description of the video program and subsequent actions to take after recognizing the event. For example, EVENT 1 may include identification information to insert an identification into the program flow at the time of EVENT 1. The identifier is operative to allow a user to go directly to EVENT 1 or, alternatively, to skip the EVENT 1 during playback. The functionality of the identifications in the multimedia content stream is described in greater detail in the following, and is also described in the US application with serial number 09 / 665.921, entitled "Closed Caption Tagging System", owned by the Applicant.

With reference still to table 330, metadata can also specify particular actions, such as ACCESSION 1, associated with the sequence of calculated keys A6E3, 9Y95, E4E9, and 9F4U. According to one example, ACCESSION 1 can describe a particular action that the DVR must take after recognizing the sequence of calculated keys. For example, ACCESS 1 may include presenting an icon to a user of the DVR. The metadata that defines ACCESS 1 can include time information related to the duration of the action, as well as information related to the placement of graphics and icons on the display device connected to the DVR. Note that the action specified by the metadata can take place at a different time from the moment of recognition of the sequence of calculated keys. For example, metadata may indicate that the action should take place at a time that precedes the recognition of the sequence of calculated keys. In this way, if the program is reproduced from the recorded copy, the DVR will scan ahead, carrying out the calculated key comparisons, and know how to perform the action in due time.

In another example, ACCESSION 1 may include instructional data for the DVR to process. Instruction data can cause the DVR to record and store one or more portions of the multimedia content stream. For example, during a live broadcast of a particular video program, if the DVR recognizes the calculated key sequence of C, D, E, FY processes the instruction data associated with the calculated key sequence, the instruction data will make The DVR starts recording the program immediately. In other examples, instructional data may cause the DVR to start recording the live broadcast at any time after recognition of the sequence of calculated keys.

In addition, the metadata associated with any sequence of calculated keys is not limited to the specified events or actions, but may be any data that can be understood and processed by a DVR or a device that receives the metadata.

3.1.3 Recognition of key sequences calculated on a multimedia device

When the server 106A has accumulated sequence data of calculated key values, server 106A sends the sequence data of calculated key values to a DVR or other device that uses the sequence data of calculated key values to recognize data from. subtitles for the deaf and to synchronize with video programs using the data of subtitles for the deaf. With reference now to FIG. 4, a flowchart is shown that schematizes the process of recognizing subtitle data for the deaf and synchronizing actions and events with the subtitle data for the deaf. In step 402, the DVR 102 receives the sequence data of calculated key values represented by table 330 in FIG. 38 from server 106A. In one example, in what is known as a "pull" transaction, the DVR 102 receives the calculated key value sequence data after initiating a request for the calculated key value sequence data to the server 106A. Alternatively, in what is known as a "push" transaction, the server 106A can automatically send the calculated key value sequence data to the DVR 102 without any request from the DVR.

102. When the DVR 102 receives the calculated key value sequence data, it stores the calculated key value sequence data in the storage device 214.

In step 404, DVR 102 receives a stream of multimedia content for presentation to a user. According to an example, the multimedia content stream may be received from the input module 202A and represent multimedia content that is currently being broadcast by the content provider 1068. Alternatively, the multimedia content stream may be reproduced from the storage device 214 of the DVR 102. In addition, in other embodiments, the DVR 102 may receive a stream of multimedia content via broadcast or over the Internet without presenting the stream of multimedia content to the user. . Hence the DVR 102 can receive content from the content provider 1068 and store the content on the storage device 214.

While the multimedia content stream is being recorded or played back, in step 406, the DVR 102 begins analyzing the subtitle data for deaf people associated with the multimedia content stream.

The analysis module 210 analyzes the subtitle data for the deaf in the same manner described above with reference to the server 106A in FIG. 38. Thus, the analysis module 210 analyzes the subtitle data for the deaf by transforming them into multiple key values calculated according to the procedure used for the subtitles for the deaf (ie, sudden onset, ascending onset or background appearance). While generating the calculated key values, the analysis module 210 sends the calculated key values and the time stamps associated with each calculated key value to a pattern recognition state machine 212, which is used to compare and identify the calculated key values with any of the calculated key value sequence data downloaded, stored or received by the DVR 102 from the server 106A or any other source.

In step 408, the pattern recognition state machine 212 is used to monitor the calculated key values generated by the analysis module 210 and compare the calculated key values with the sequence data of calculated key values received in the step 402. According to one example, the pattern recognition state machine 212 is a state machine constructed according to the calculated key value sequence data that was received in step 402.

Looking at an example, suppose the pattern recognition state machine 212 adopts the structure of a tree that contains knots and branches. Each node of the tree can represent a particular calculated key value in a sequence of calculated key values, and each "branch" of the tree represents the complete sequence of calculated keys identified by the sequence data of calculated key values. This structure is an additional reference object in FIG. 5A, set forth below. the pattern recognition state machine 212 attempts to match the calculated key values generated by crossing the tree nodes. At any given node, if a match occurs, the pattern recognition state machine 212 continues to the next node. If the previous node represents the end of a sequence of calculated key values, a match has occurred, and the pattern recognition state machine 212 may indicate that a sequence of calculated key values has been identified, which, in turn instead, it causes the OVR 200 to take the appropriate action, as described in greater detail herein.

Accordingly, in step 510, the pattern recognition state machine 212 determines whether any calculated key value generated by the analysis module 210 matches the calculated key values contained within the sequence data of calculated key values . As long as the calculated key values match and the time difference between the calculated key value events is within a threshold of the expected calculated key sequence, a positive match will be reported.

Referring now to Fig. 5A, according to an example, data of a sequence of key values calculated in the form of a pattern recognition state machine is shown. The sequence data of calculated key values are represented by the state machine 510, which contains nodes representing individual calculated key values within the calculated key value sequences. According to one example, the state machine 510 contains a START 502 node representing the initial state of the state machine.

According to an example, the nodes of the state machine 510 are indexed according to the first calculated key value of any calculated key sequence identified in the sequence data of calculated key values. Or after the state machine 510 is initialized, it can first determine whether that calculated key value of the first node 504 matches a calculated key value generated by the analysis module 210. If no match occurs, the state machine 510 continues to cross the index 501, subsequently checking the next node 520. Note that, although it is rare, different sequences of calculated key values can start with the same calculated key value. Therefore, node 504, which represents calculated key sequences that begin with the calculated key value "3059", may contain two distinct branches representing different sequences of calculated key values: (1) 3059, 2Z55, A6E3 and 9Y95, or

(2) 3059, 2Z55, E4E9 and 9F4U.

According to one example, the state machine 500 is a lightweight component of the OVR 200, which means that the state machine 500 is designed to occupy minimal system resources during the operation of the OVR 200. Therefore, even in the case of that the state machine 500 must cross tens of thousands of nodes to determine whether a sequence of calculated key values has been recognized, few system resources are required to operate the state machine 500. Oe here that the users of the OVR do not appreciate any delay or degradation in the reproduction of multimedia content during the operation of the state machine 500.

According to one example, the efficiency of the state machine 500 is further increased by the use of index 501. By indexing the first key value calculated within each sequence of keys calculated at an index 501, the state machine 500 can quickly discard a large number of key sequences calculated for a comparison without taking up many system resources.

To initiate the processing of calculated key values generated from the analysis module 210, the state machine 500 is initialized and the checking of the nodes located at index 501 begins. According to an example, the first calculated key value generated by the Analysis module 210 is "3059", corresponding to text A in frame 310A of FIG. 38. Accordingly, the state machine 500 is started, it will check the first node 504 representing the calculated key value "3059" and find a match, giving a positive output according to this example, any match causes the state machine to proceed to the right, while any error causes the state machine to cross to a lower level. However, in other examples, the state machine 500 may be structured in any manner suitable for checking calculated key values.

However, suppose that the analysis module 210 initially reported a calculated key value of "2Z55". Instead of finding a match in the first node 504, the state machine 510 would report a negative output and continue down to the next node 520 in index 501, which represents the calculated key value "2Z55". Alternatively, if no match is found in a particular node, the state machine 510 may also proceed to the ERROR node 515, which causes the state machine to restart and start again at the node 502 of START.

Therefore, assuming that a match occurred in the first node of index 501, the state machine 212 then waits for the next generated key value calculated from the analysis module 210. In one example, the next generated key value calculated is "2Z55". Consequently, the state machine 510 compares "2Z55" with the value in the next node 506. Since the value in the next node is also "2Z55", the state machine 510 will find a match and proceed to node 508. Without However, note that if no match was found in node 506, the state machine 500 would not proceed to the ERROR state, but instead try to find a match for the key value calculated in node 507, which represents "E4E9". In this way, node 506 represents a fork in the state machine 500, so that two sequences of calculated key values can begin with the calculated key values of "3059" and "2Z55", respectively.

In response to the receipt of calculated key values generated by the analysis module 210, the state machine 500 sequentially traverses each node represented in FIG. 5 until no match occurs

or until the end of a sequence of calculated keys has been identified. In addition, while searching for calculated key value matches with each key value calculated in the sequence of calculated key values, the state machine 500 can also compare the time lag between the calculated key value generated by the analysis module 210 and the key value calculated within the sequence of calculated keys. If both calculated key values match and match the time offset values of both calculated key values, the status machine 500 can determine that a positive match has occurred. In another example, the state machine 500 may determine that a particular calculated key value has coincided with the time lag of a calculated key value within a sequence of calculated keys allowing "compensation" or margin of error. According to this specific example, the state machine 500 determines that a match has occurred for the sequence of calculated keys of 3059, 2Z55, A6E3 and 9Y95 after receiving the respective calculated key values generated by the analyzer 312 in FIG. 38.

When no match is found, the status machine 500 is reset to the HOME node 502 and reset the match search process when the next calculated key value is received from the analysis module 210.

Note that throughout the operation of the DVR 200, the state machine 500 is continuously trying to find a match of the calculated key values with the calculated key sequences. When a positive match is found for a complete sequence of calculated keys, in step 412, the state machine 500 reports a full match, causing the DVR 200 to inspect the metadata associated with the sequence data of calculated key values.

In other examples, the state machine 500 uses a table of calculated keys to try to find matches of calculated key values with sequence data of calculated key values. A table of calculated keys is a data structure composed of an array (or a table in which the data to be searched) is stored and a correspondence function. A table of calculated values is commonly used in search algorithms and is a data structure that is well known in the art.

In addition, in another example, the DVR 200 contains a plurality of state machines. By having more than one state machine that work at the same time, multiple patterns can be identified within the same sequence of calculated key values. For example, suppose the calculated key values of "X6Y6, A4Z4 and 81C1" were generated from a multimedia content stream. Let's also assume that there are two known patterns, "X6Y6, A4Z4" and "A4Z4, 81C1". If a single state machine were running on the DVR 200, only one of those patterns would be recognized. However, since the DVR 200 can concurrently run more than one state machine, the DVR 200 can recognize both "X6Y6, A4Z4" and "A4Z4, 81C1" patterns from the calculated key values of "X6Y6, A4Z4 and 81C1 ". Therefore, in other examples, the DVR 200 may contain a plurality of state machines, and each state machine may represent different intra-band data patterns.

With reference to FIG. 58, a sequence of key values calculated according to an example is shown. The calculated key value sequence data contains the sequence data 520A and the 5208 metadata. The sequence data 520A corresponds to the data representing the calculated key value sequence, as well as the approximate time difference between the incidences of the calculated key values. Thus, as described above, the state machine 500 crosses the nodes, as depicted in FIG. 5A, until no match is found at a particular level or if the end of a sequence of calculated keys has been found. According to one example, the pattern recognition state machine 212 determines that the sequence of the calculated key values 3059, 2Z55, A6E3 and 9Y95 has coincided with the generation of the same calculated key values from the base analysis module 210 to the time difference between each calculated key value generated.

3.1.4 Synchronization with subtitle data for the deaf 16

When a DVR determines that a pattern match has occurred, it uses the metadata described above to synchronize with the flow of multimedia content. In particular, the DVR uses metadata to determine the relative location of events or video programs within the multimedia content stream. Thus, when the DVR 102 determines that a particular sequence of calculated key values has been generated, the DVR 200 will consult the 5208 metadata to determine the action to be taken. Metadata 5208 may contain information such as instruction and control information, program timing, content information, display instructions and graphic information. In addition, metadata 5208 may also contain instruction data 522.

Once the pattern recognition state machines 212 have determined that the calculated key values from the multimedia content stream match a particular sequence of calculated keys, then, in step 412, the DVR inspects the sequence metadata of calculated keys associated with the sequence of calculated keys and responds accordingly. Depending on the content of the metadata, a DVR such as DVR 102 can respond in many different ways to the recognition of a sequence of calculated key values. For example, metadata may contain information that allows the DVR to recognize events within a video program, to synchronize with the video program or with the entire multimedia content stream, to present information to a user or to have it Record a future program. In addition, in other examples, the metadata may contain information that allows a DVR to perform any action related to the flow of multimedia content, the video program or the operation of the DVR, for example skipping advertisements and other content.

Thus, with reference now to FIG. 6, a block diagram is shown illustrating a procedure for synchronizing actions and events with intra-band data according to an example. Table 606 represents sequence data of calculated key values, while table 608 represents metadata associated with the particular sequence of calculated keys. In this particular example, table 606 contains the sequence of calculated keys 3D59, 2Z55, A6E3 and 9Y95, as well as the time and time offset values for each key value calculated. As described above, this information is received from server 106A or any other source, and is found in DVR 102 in the form of pattern recognition machines 212. Content 600 represents a stream of multimedia content that contains one or more video programs. While playing or recording media stream 600, the DVR indicates its playback location within content stream 600 using cursor 604, which moves over a program buffer bar representing the program material, as described in US Patent No. 6,850,691 and in US Application No. 10/915,990, both entitled "Multimedia Progress Indication System" and owned by the Applicant. In this particular example, the cursor 604 is currently at the 1: 12 hour of the multimedia content stream or the video program. While playing the content stream 600 using an analysis module, the DVR generates the calculated key value "3D59" at the instant: 00, the calculated key value "2Z55" at the instant: 05, the calculated key value " A6E3 "at the instant: 10 and the calculated key value" 9Y95 "at the instant: 12, which is at that time at the location indicated by cursor 604.

Once the calculated key values and the corresponding hourly data have been generated, the DVR determines that the calculated key values match the sequence data of calculated key values found in table 606. According to one example, the DVR can positively match the calculated key values with the sequence data of calculated key values comparing not only the calculated key values themselves, but also the time lags for each calculated key value.

After determining a match, the DVR now knows where it is located within the program. Specifically, the DVR determines that the cursor 604 is located 12 seconds after the start of the specified start of the program according to the sequence data of calculated key values. Next, the DVR examines the metadata information found in Table 608 to determine the appropriate action to be taken in response to the sequence data of calculated key values. According to one embodiment, table 608 indicates that EVENT 1 takes place at the instant: 1 O. By examining the current location of cursor 604, DVR 102 can calculate that EVENT 1 takes place: 02 seconds before the current cursor location . If EVENT 1 was defined to be showing an icon to the user for 10 seconds, DVR 102 can immediately display the icon for eight seconds, since it knows that it has already passed the beginning of EVENT 1 in two seconds. If the program being played is a recording or the user is playing a live broadcast but its playback location lags behind the broadcast itself, the DVR 102 can scan ahead of the current playback location and calculate Key values calculated ahead of the current playback location. This allows DVR 102 to be more precise in performing any action associated with an event.

In addition, after inspecting table 608, DVR 102 can determine that EVENT 2 will take place at the instant: 40 of the video program. After examining the current location within the video program, DVR 102 determines that EVENT 2 takes place: 28 seconds after the last key calculated. Since DVR 102 can determine the precise location of EVENTS 1 and 2 within the multimedia content stream, DVR 102 can be programmed to react to those events in any way. For example, according to one example, DVR 102 may be instructed to display an icon to a user during EVENT 2. In another example, DVR 102 may be programmed to start recording the flow of multimedia content in the EVENT

2. In this way, the information that identifies any EVENT 1 or 2 can be used to identify a differentiated event within the video program 600. In addition, in other embodiments, table 608 may define other events or other actions that a DVR should take after recognition of a sequence of calculated keys. For example, after recognizing the sequence of calculated keys 3D59, 2Z55, A6E3 and 9Y95, table 608 may indicate that the DVR should immediately show the user an interactive icon on a display device.

In addition, once DVR 102 recognizes EVENTS 1 AND 2 within the video program, DVR 102 can insert video identifiers into the multimedia content stream at EVENT 1 Y2 locations. According to an example, video identifiers are data objects embedded in a multimedia content stream that contains video and audio data, such as an MPEG-2 stream. After video identifiers are placed in the multimedia content stream, the multimedia content stream is stored for later playback. When played back, the DVR 102 processes the multimedia content stream and encounters the video identifiers in the exact location of EVENTS 1 AND 2. Thus, once video identifiers have been placed in the content stream multimedia, the DVR no longer needs to recognize subtitle data for the deaf to synchronize with the flow of multimedia content, because events or actions have been defined by video identifiers, which are automatically processed along with the video and audio data during playback The video identifiers are presented in greater detail in the US application with serial number 09 / 665.921, entitled "Closed-Caption Tagging System".

Although the above process has been described in relation to DVR 102, the procedure can be performed by any other device capable of performing such steps, such as DVR 104, PC 101A or portable device 101 B.

3.2 Enhanced television signaling data

Another type of intra-band data that can be used to recognize intra-band data patterns is Enhanced Television (ETV) signaling data. ETV allows the use of interactive applications that are distributed "intra-band" with other video services in a multimedia content stream. To synchronize intra-band applications with video programs, ETV uses instructional signals coordinated with the timing of video programs. More information is available regarding signaling capabilities for intra-band data transmission in the OpenCable Enhanced TV Application Messaging (ETV-AM-102) specification, which specifies signaling capabilities for intra-band data transmission synchronized with the time line of A video service

According to one example, multimedia devices with ETV features include an ETV agent to receive and process signals from ETV instructions. ETV instruction signals make multimedia devices discover, download, launch and terminate ETV applications at particular times. Thus, according to one example, a multimedia device can recognize ETV signaling data patterns that define the start

or the termination of applications within a multimedia content stream of ETV. This can be implemented using the analysis module 210 to identify and extract the ETV signaling data from the multimedia content stream. In addition, ETV instruction signals may include additional information, such as information describing the content or programming of ETV. In addition, the ETV instruction signals are associated with time data to process the ETV application in due course. Using this time data, a multimedia device can identify a relative position within a multimedia content stream. Therefore, a multimedia device may interpret ETV signaling data to determine that (a) an ETV event is occurring and (b) that the ETV event is synchronized with a particular portion or segment of the video program.

Thus, similar to the sequence recognition of key values calculated in the subtitle data for the deaf, the pattern recognition state machine 212 can be used to recognize patterns of signals from ETV instructions and time information. Since ETV applications are already synchronized with the accompanying video program, a multimedia device can synchronize with a multimedia content stream recognizing ETV signaling data patterns. As with the subtitle data for the deaf, once the multimedia device recognizes a pattern, the multimedia device can take similar actions to those described above with respect to the synchronization of actions and events with the subtitle data for the deaf.

4.0 FOCUS TO CREATE AND SEE CUSTOM MULTIMEDIA SEGMENTS

4.1 Selection of multimedia program segments by the user

According to one embodiment, the video programs in a multimedia content stream are displayed to a DVR user in a display device such as a television screen or a computer monitor. In other embodiments, the video programs may be presented on a display device such as a portable multimedia viewing device. A DVR user can control the playback of the video program using a control interface such as a remote control.

A user can mark sections of a multimedia program or place points of interest related to the content within the multimedia program. For example, a user may want to mark the best plays of a recording of a football match. Once the user marks the moves, he can send the resulting sequences of calculated keys to a friend's DVR. In the US application with serial number 10 / 220,558, entitled "System and Method for Internet Access to a Personal Television Service" and in the US application with serial number 10 / 741,596, entitled "Secure Multimedia Transfer System" a procedure is described Data transfer between several DVRs. The friend's DVR can then present the user's points of interest to the friend when the friend reproduces his recorded copy of the match. The friend's DVR presents the points of interest along with the program buffer bar, shown, for example, on your display device.

The friend's DVR processes the recorded match beforehand before starting to play the match by calculating calculated key values of the recorded match and comparing the calculated key values with the received sequences of calculated key. This allows the DVR to locate the points of interest in the recorded match and present the points of interest to the friend. The friend can then jump to points of interest using his remote control.

DVR users can distribute their own sets of program points of interest to other users. Users can also attach metadata to each point of interest that can make the DVR present text to the viewer; for example, "Isn't this a great action scene?" The user can also attach metadata to a point of interest that tells the DVR to skip x seconds in the program from that point of interest or to present x seconds of the program before jumping to the next point of interest. This allows users to create their own condensed versions of a program that they can distribute to their friends, family, classmates, students, an interest group, etc.

In addition, you can instruct a DVR to reorder segments of a multimedia program or a complete stream of content. One purpose to do so is when the broadcaster has encoded segments of the multimedia content stream. Consequently, unauthorized viewers will not be able to clearly see the flow of multimedia content. However, for authorized viewers, the DVR puts the segments in the correct order. In addition, users may wish to reorder segments of the multimedia content stream to create their own customized multimedia programs. Users can then store personalized multimedia programs or distribute personalized multimedia programs to their friends and other DVR users. Custom multimedia programs may include a compilation of segments retrieved from different multimedia content streams, or from segments that have previously been stored in the DVR's storage device.

Using the procedure to recognize and synchronize a computer system with intra-band data, as described above, the DVR can perform these functions with ease.

For example, when a user wishes to save or indicate the location of a particular content within the video program, the DVR user can press a button on the control interface to indicate it. The control interface sends a signal to the DVR, which interprets the signal as a user selection. The DVR can then determine the time within the program in which the user sent the signal from the control inteñaz and calculate an intra-band data pattern leading to that point.

4.2 Creation of user-initiated event identification data for multimedia programs

With reference to FIG. 7, a block diagram illustrating a procedure for creating and viewing multimedia segments within a video program according to an embodiment is shown. When a 700A stream of multimedia content containing intra-band data such as subtitle data 702 for the deaf (SS) is being recorded

or by playing on a DVR or other device, time information is associated in the form of time stamps associated with intraband data. Specifically, as described above, the flow of multimedia content, including intra-band data, is packaged in differentiated frames of audio, video and data. When packaged, each frame is given a relative time stamp. These timestamps can be used to track the time of intra-band data within the multimedia content stream or video program.

When a user views a segment of a video program that he would like to save or that he would like to take note of, the user has the DVR 102 mark the selected segment a location within the video program. According to one embodiment, a graphical user interface is displayed on a display screen that includes a bar 7008 of program buffer and a cursor 708. The cursor indicates the current playback location of the program. When a user wishes to select a location within the video program, the user waits until the cursor has reached the desired location and, using a control interface such as a remote control, selects the desired location. When the user presses a button on the remote control, a signal is sent from the remote control to the DVR, causing the DVR to present the user with a drop-down menu. The drop-down menu may contain options for the user to select in relation to the desired location in the video program. According to one embodiment, the options available to the user include inclusion in favorites, skipping the location, attaching a note or any other action that may be carried out in relation to a video program.

According to one embodiment, while the video program is playing, a DVR user selects location 704 using the control interface. When the DVR receives the signal from the control interface, it recognizes that a user selection has occurred and records the location of the selection, which occurs in the instant: 10. At this point, the DVR creates event identification data. As described above, event identification data may contain intra-band data patterns or sequences, as well as metadata to identify actions or events within the multimedia content stream.

Again, one type of intra-band data is subtitle data for the deaf. According to one embodiment, to create event identification data, the DVR searches for sufficient subtitle data for the deaf before location 704 to create a unique sequence of calculated key values. Table 710 represents sequence data of calculated key values of video segment 700. Sequence data of calculated key values may indicate the calculated key value, the time of the calculated key value and a time lag for each calculated key value. According to one embodiment, the time lag indicates the difference between time between the generation of the calculated key value and the previous key value calculated in the video program.

Thus, table 710 identifies the calculated key values "W5W4" and "M5M5" that precede location 704. The DVR records the time at which each calculated key value was generated according to the particular subtitle procedure for the deaf used for show the subtitle text for the deaf and the time lag of the calculated key values. Here, the calculated key value "W5W4" occurs in the instant: 06 from the start of the program and is the first key calculated in the sequence with a time offset of zero. The calculated key value "M5M5" occurs at the instant: 08 from the start of the program and is two seconds after the calculated key value "W5W4". When a DVR recognizes the sequence of calculated keys "W5W4, M5M5", the DVR can identify that an action is required prior to the actual reproduction of location 704 and carry out the appropriate action, such as the presentation of an icon. Metadata may indicate that the icon is displayed two seconds after recognition of the sequence of calculated keys "W5W4, M5M5".

In addition, the user can also select a second location 706 that represents the end of a desired portion of a video segment that occurs instantly: 20. According to one embodiment, the DVR analyzes the SS data between the instants: 10 and: 20, generates three calculated key values "Z1B1, E5Y5 and A2R2" It records the time at which each calculated key value was generated. Hence, in this particular example, the calculated key value "Z1 B1" occurs first (here, at the instant: 12), that the calculated key value "E5Y5" occurs two seconds later and that the calculated key value "A2R2" occurs four seconds after "E5Y5". Collectively, the calculated key values "Z1 B1, E5Y5 YA2R2" represent the sequence of keys calculated for the second location 706. The metadata associated with the program will indicate this.

Thus, in table 710, the calculated key value "Z1B1" occurs at the instant zero, since it is the first key calculated in the sequence, the calculated key "E5Y5" occurs two seconds after the calculated key value "Z1B1" and the calculated key value "A2R2" occurs four seconds after the calculated key value "E5Y5". Alternatively, the time lag can be relative to any portion of the video program, of the multimedia content stream, of the location in the program or video stream or of the video segment.

When a DVR reproduces the 700B stream of multimedia content using table 710 as sequence data of calculated key values for its state machine, it analyzes the SS data 702 and generates calculated key values. Once the DVR recognizes the calculated key value "A2R2" according to the procedure described above to recognize calculated key sequences, the DVR can immediately carry out an action, such as causing the presentation of an icon in a display screen connected to the DVR. In this case, when the DVR recognizes the calculated key sequences found in Table 710, the DVR can determine the exact start and end times of the video segment between locations 704 and 706. Given this information, the metadata associated with the Sequence data of calculated key values can instruct the DVR to skip the program segment between locations 704 and 706 or can instruct the DVR, for example, to store the program segment between location 704 and 706.

The DVR can recognize video segments relative to any multimedia content stream using the time stamps in the sequence data of calculated key values. Hence, even if a video program was broadcast or played at two different times in different locations, the same video segments could be recognized using the time stamps associated with the calculated key values. More specifically, time stamps are related to the generation of calculated key values and not to a local DVR time or a broadcast time. Therefore, assuming that most of the programming is identical when played in different locations or at different times, the video segments can be recognized with great precision using the timestamps of the calculated key values. Note that although the above discussion focuses on subtitle data for the deaf, the invention is not limited to such an embodiment and that a DVR can create event identification data using any type of intra-band data, such as ETV signaling data.

4.3 Creating an identified content of multimedia programs

After recognizing events or segments within a video program, the DVR 102 can generate identifiers to correspond to the segment or events and insert those identifiers into the multimedia content stream in the appropriate locations. Each identifier may contain instruction and control information that a device that processes the identifier translates and with which it acts. For example, according to one embodiment, if a user wishes to skip all content between locations 704 and 706, after recognizing the sequence of calculated keys associated with the video segment, the DVR can simply create an identifier at location 704, causing the DVR automatically jumps 10 seconds to location 706, thereby skipping the video segment.

In addition, the identifier data is processed as part of the multimedia content stream and, therefore, is automatically synchronized with the video program playback. With the identifier data, a device that receives the flow of multimedia content can automatically jump to the selected location. Alternatively, the identifier may include information that identifies the content related to the location within the video program. For example, an identifier corresponding to the location 704 in the multimedia content stream 700A may include information that identifies the location as a break within a sports program. In addition, location 706 may be the end of the transmission. Thus, a DVR can present a user interface for a user to indicate that the rest within the sports program takes place at location 704 or at the instant: 10 and that it ends at location 706 or at the instant: 20. Using this information, the user can make the DVR jump behind location 704 or proceed directly to location 706.

In addition, while identifiers are inserted into the multimedia content stream to correspond to particular events or segments, the DVR 102 can remove intra-band data from the multimedia content stream. For example, according to one embodiment, the multimedia content stream is an MPEG2 transport stream that includes separate audio, video and data inputs. The original intra-band data is located in the data portion of the multimedia content stream. When an identifier is generated, the identifier is inserted into the data portion of the multimedia stream. In addition, once the intraband data has been deleted, the PC 101, for example, can transfer the multimedia content stream to the portable device 101 B. Since identifiers are placed in the multimedia content stream and the stream has been removed of subtitles for the deaf, it is not necessary for the portable device 101 B to display subtitle data for the deaf. Instead, identifiers within the multimedia content stream are automatically processed in synchronization with the multimedia content stream while the multimedia content stream is being displayed to a user. Additional details on the generation and insertion of identifiers in a multimedia content stream are presented in US application No. 09 / 665,921, entitled "Closed Caption Tagging System", owned by the Applicant.

In another embodiment, a multimedia device, such as DVR 102 or PC 101, can simply provide the portable device 101 B with a list of locations of interest. Thus, instead of inserting identifiers in the multimedia content stream, the content stream remains untouched and the portable device 101 B can identify events within a video program using the list of locations of interest. The list of locations of interest can be provided separately from the flow of multimedia content to the portable device101B. According to one embodiment, the list of locations of interest includes hourly data that identifies the relative time at which an event occurs. In addition, the list of locations of interest may also include additional data, such as data describing each event. The portable device 101 B may decide to use the data from the list of locations of interest in any appropriate manner.

According to one embodiment, the list of locations of interest is generated by a multimedia device in response to user intervention. For example, a user can select one or more locations within a video program using a control interface. The multimedia device receives signals from the control interface and determines where the selected locations are within the video program. The multimedia device can make such determination by examining the time data associated with the flow of multimedia content. The multimedia device then stores the data that identifies each selected location with its associated time data in the list of locations of interest and provides the list to the portable device 101 B.

In addition, in another embodiment, a multimedia device may generate a list of locations of interest after recognizing events or segments within a video program using intra-band data patterns. For example, a multimedia device may process a complete stream of multimedia content before providing it to a portable device 101 B. When the multimedia device recognizes an intra-band data pattern, it examines the metadata associated with the intra-band data pattern and identifies events within a video program, including the hourly data associated with each event. Using the time data, the multimedia device generates a list of locations of interest that lists each event and the time at which the event takes place with respect to the video program or the flow of multimedia content. The list of locations of interest is then provided to the portable device101B, which can use the list to identify the locations within the video program.

4.4 Sharing of event identification data for multimedia programs

As noted above, users of a DVR can transfer event identification data from their DVR to other DVRs. This can be achieved through a direct transfer via a network or Internet to another DVR. Once a DVR receives event identification data, the DVR user can be notified that a custom program index has been received and can also be notified of the program with which it is related. The user can then choose to play the program using the index.

In addition, a DVR service can act as a distribution point for event identification data. Event identification data associated with a program can be uploaded to server 106A from the DVR. When the server 106A receives the event identification data from the DVR, it stores the event identification data on a local storage database device. The server 106A can provide event identification data created in multiple DVRs for download via the network 105. When another DVR 104 wants to retrieve event identification data associated with a particular program, the server 106A sends the identification data of events to DVR 104 via network 105. Therefore, a system is enabled to create and view custom video segments.

5.0 IMPLEMENTATION MECHANISMS

Figure 8 is a block diagram illustrating a computer system 800 in which an embodiment of the invention can be implemented. The ordering system 800 includes an 802 bus or other communication mechanism for communicating information, and a processor 804 coupled with the 802 bus for information processing. System 800 also includes a main memory 806, such as a random access memory (RAM) or other dynamic storage device, coupled to the 802 bus to store information and instructions to be executed by the processor 804. The main memory 806 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 804. The computer system 800 also includes a read-only memory 808 (ROM) or other static storage device coupled to the 802 bus for storing static information and instructions for the processor 804. A storage device 810, such as a magnetic disk or an optical disk, is provided and is coupled to the 802 bus to store information and instructions.

The computer system 800 may be coupled, via the 802 bus, to a display device 812, such as a cathode ray tube (CRT), to display information to a user of the computer. An input device 814, which includes alphanumeric and other keys, is coupled to the 802 bus to communicate information and instruction selections to the processor 804. Another type of user input device is the cursor control 816, such as a mouse. , a control ball or cursor direction keys for communicating address information and instruction selections to the processor 804 and for controlling the movement of the cursor on the display device 812. Typically, this input device has two degrees of freedom on two axes, a first axis (for example, x) and a second axis (for example, y), which allows the device to specify positions on a plane.

The invention relates to the use of a computer system 800 to implement the techniques described herein. According to an embodiment of the invention, these techniques are performed by the computer system 800 in response to the processor 804 executing one or more sequences of one or more instructions contained in the main memory 806. Such instructions can be read to the main memory 806 from another machine-readable medium, such as storage device 810. Execution of instruction sequences contained in main memory 806 causes processor 804 to carry out the process steps described herein. In alternative embodiments, a wired circuitry may be used instead of or in combination with software instructions to implement the invention. Thus, the embodiments of the invention are not limited to any specific combination of hardware and software circuitry.

As used herein, the term "machine-readable medium" refers to any medium that participates in providing data that causes a machine to operate in a specific manner. In an embodiment implemented using computer system 800, various machine-readable means are involved, for example to provide instructions to processor 804 for execution. Such a medium can take many forms, including, without limitation, non-volatile media, volatile media and transmission media. Non-volatile media includes, for example, optical or magnetic discs, such as the storage device 810. Volatile media includes dynamic memory, such as main memory 806. Transmission media includes coaxial cables, copper wire and optical fiber, including the wires that comprise the bus.

802. The transmission media may also take the form of acoustic or light waves, such as those generated during data communications by radio or infrared waves. All such media must be tangible to allow the instructions carried by the media to be detected by a physical mechanism that reads the instructions by entering them into a machine.

Common forms of machine-readable media include, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape or any other magnetic media, a CD-ROM, any other optical media, punch cards, paper tape, any other physical medium with hole patterns, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other chip or memory cartridge, a carrier wave, as will be described later in this document, or any other means of which it may read a computer

Various forms of machine-readable media may be involved in carrying one or more sequences of one or more instructions to the processor 804 for execution. For example, the instructions may initially be ported on a magnetic disk of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions via a telephone line using a modem. A local modem of a computer system 800 can receive the data over the telephone line and use an infrared transmitter to convert the data into an infrared signal. An infrared detector can receive the data carried on the infrared signal and an appropriate circuitry can put the data on the 802 bus. The 802 bus carries the data to main memory 806, from which processor 804 retrieves it and executes the instructions. The instructions received by main memory 806 can optionally be stored in the storage device 810, either before or after execution by the processor 804.

The computer system 800 also includes a communications interface 818 coupled to the 802 bus. The communications interface 818 provides a two-way data communications link to a network link 820 that is connected to a local network 822. For example, the interface 818 communications can be an integrated services digital network (ISDN) card or a modem to provide a data communications connection with a corresponding type of telephone line. As a further example, the communications interface 818 may be a local area network (LAN) card to provide a data communications connection with a compatible LAN. Wireless links can also be implemented. In any such implementation, the communications interface 818 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

Normally, network link 820 provides data communication, through one or more networks, with other data devices. For example, the network link 820 may provide a connection through the local network 822 with a central computer 820 or with data equipment operated by an Internet service provider 826 (ISP). In turn, ISP 826 provides data communications services through the global data packet communications network now commonly referred to as "Internet" 828. Local network 822 and Internet 828 use both electrical, electromagnetic or optical signals that they carry digital data streams The signals through the various networks and the signals on the network link 820 and through the communications interface 818, which carry the digital data to and from the computer system 800, are exemplary forms of carrier waves that carry information.

The computer system 800 can send messages and receive data, including program code, through the

or networks, network link 820 and communication interface 818. In the Internet example, a server 830 could transmit a requested code for an application program through Intemet 828, ISP 826, local network 822 and communications interface 818.

The received code can be executed by the processor 804 when it is received, and / or stored in the storage device 810 or other non-volatile storage for later execution. In this way, the computer system 800 can obtain application code in the form of a carrier wave.

In the foregoing specification, embodiments of the present invention have been described with reference to numerous specific details that may vary from one implementation to another. Thus, the only and exclusive indicator of what the invention is, and what the Applicant intends to be the invention, is the set of claims that this application discloses, in the specific way in which such claims are disclosed to know, including any subsequent correction. Any definition expressly formulated herein of terms contained in such claims will govern the meaning of such terms as used in the claims. Therefore, no limitation, any element, property, characteristic, advantage or attribute that is not expressly listed in a claim should limit the scope of such claim in any way. Memory and drawings must, therefore, be considered in an illustrative, non-restrictive sense.

Claims (17)

1. A procedure carried out by a multimedia device, the procedure comprising: receiving a stream (700A) of content in the multimedia device, in which the stream (700A) of content comprises a video stream, an audio stream and an intraband data stream; while a video stream is presented to a user, in response to receiving a user indication of selecting a location (704, 706) within the content stream (700A), identify a sequence of a plurality of calculated key values (701) generated from a portion of the intraband data flow that precede the selected location (704, 706), determine, for each calculated key value of the sequence, one or more hourly values that reflect a time at which the calculated key value was generated, and storing, in a storage device, the sequence of key values (701) calculated in association with the determined hourly values and metadata that specify an action to be carried out after recognition of the sequence of key values (701) calculated in an intra-band data stream of a content stream (700A).

2.

The method of Claim 1 wherein the intraband data flow received in the multimedia device comprises data (702) of subtitles for the deaf (SS).

3.

The method of Claim 1, further comprising:

while a video stream is presented to a user, before receiving the indication, display on a display screen operatively coupled to the multimedia device a graphical user interface comprising a bar (7008) of program buffer and a slider (708), indicating the slider (708 ) a current playback location of the content stream (700A), and in response to the reception of the indication, present a menu on the display screen through which the user can specify the action.

Four.

The method of Claim 1 wherein the intraband data flow received in the multimedia device comprises Enhanced Television (ETV) signaling data.

5.

The method of Claim 1 wherein the determination includes determining two hourly values for each key value calculated in the sequence, one of the two hourly values reflecting a time offset from a given hourly value for a first key value calculated. of the sequence.

6.

The method of Claim 1, further comprising:

while a video stream is presented to a user, in response to receiving a second user indication of selecting a second location (704, 706) within the content stream (700A), identify a second sequence of a plurality of calculated key values (701) generated from a second portion of the intraband data flow that precede the second selected location (704, 706), AND storing, in the storage device, the second sequence of key values (701) calculated in association with the metadata.

7.

The method of Claim 6 wherein the metadata indicates that a program segment of a recognized content stream (700A) should be skipped based on the sequence of calculated key values (701) and the second sequence of values (701) of calculated key.

8.

The method of Claim 6 wherein the metadata indicates that a program segment of a recognized content stream (700A) should be stored based on the sequence of calculated key values (701) and the second sequence of values (701) of calculated key.

9.

The method of Claim 6, further comprising:

determine, for each key value calculated in the second sequence, one or more hourly values that reflect a time at which the corresponding key value calculated in the second sequence was generated, and 24 storing, in the storage device, the second sequence of key values (701) calculated in association with the second determined hourly values.

10.

The method of Claim 1 wherein the content stream (700A) is an MPEG-2 transport stream.

eleven.

The method of Claim 1, further comprising:

5 copy the sequence of values (701) of the calculated key and the metadata through a data network to another multimedia device.

12.

The method of Claim 1, further comprising: loading the sequence of calculated key values (701) and metadata through a data network on a server.

13.

The method of Claim 1 wherein the metadata indicates text and / or an icon that must

10 occur after recognition of the sequence of key values (701) calculated in an intra-band data stream of a content stream (700A). 14. The method of Claim 1 wherein the indication is provided by the user by means of a remote control device. 15. The method of Claim 1 wherein the metadata indicates one hour after which the sequence of key values (701) calculated to perform the action is recognized.

16.

An apparatus for creating and displaying personalized multimedia streams, the apparatus being adapted to carry out the procedure of any one of Claims 1-15.

17.

A computer readable storage medium that contains one or more sequences of instructions for creating and displaying personalized multimedia streams, instructions that, when executed by one or more

20 processors (804), make the one or more processors (804) carry out the process steps of any one of Claims 1-15. FIG. L .t.om 106A SERVER PROVIDER one\) Or> 101A COMPUTER STAFF (PC) .llUEl PORTABLE DEVICE 1M..DVR ~ DVR

202 INPUT MODULE

202B OUTPUT MODULE

FIG. 2 204 MODULE TAPE RECORDER ~ ANALYSIS MODULE 209 TIMER MODULE 206 COMMUNICATIONS INTERFACE 214 STORAGE 212 PATTERN RECOGNITION STATE MACHINES FIG. 3A 302 RECEIVE VIDEO PROGRAM FROM THE CONTENT PROVIDER 304 CONVERT SUBTITLE DATA FOR DEAFTS ASSOCIATED WITH THE VIDEO PROGRAM IN CALCULATED KEY VALUES ~ 306 GENERATE CALCULATED KEY SEQUENCES ASSOCIATED WITH THE VIDEO PROGRAM , 308 SEND THE DEVICES OF THE CALCULATED KEY SEQUENCE TO THE DEVICES FRAME 1 Hour = 1: 00 Text A FIG.3B 31O.B..TRAMA 151 Time = 1: 05 Text B FRAME 301 Hour = 1: 10 Text C PLOT 361 Hour = 1: 12 Text D Table .a.2lf Table ~

~ calculated alorclave

Hour Lag time

3059

tOO -

2Z55

1:05 : 05

ASE3

1:10 : 05

9Y95

1:12 : 02

Calculated Key Sequence

Metadata

3D59,2Z55, A6E3.9Y95

EVENT 1

A6E3,9Y95. E4E9,9F4U

ACCESSION 1

FIG. 3C Display device Sudden Appearance ASCENDING APPEARANCE Display device SS incoming data Plot X Line 2 of text Line 3 of text Line 1 of text Line 4 of text

---

_._ --- .......... •. • ......_ ------- _.. ---- ......-... ...-_ ..... --- _...-.....-... --- _..-......-....----

Line 3 of text Line 4 of text 4 APPEARANCE ON SECOND PLANE Display device SS incoming data .3.5! LQ +++++ xto Line 3 of text Line 2 of te ..... Line 4 of text FIG.3D 400 404A or) N FIG. 4 402 RECEIVE SEQUENCE DATA OF THE CALCULATED KEY 1404 RECEIVE THE CONTAINER CURRENT 406 ANALYZE THE SS CURRENT INSIDE OF THE CURRENT OF CONTENT AND GENERATE CALCULATED KEY VALUES 408 COMPARE CALCULATED KEY VALUES IN THE CURRENT OF SS WITH DATA RECEIVED OF THE CALCULATED KEY SEQUENCE yes ID ~ INSPECT THE CALCULATED KEY SEQUENCE METADATES FIG.5A STATE MACHINE 500 INDEX 501 • w w ._ .., __., · ~. : • ......... "...... • '------,., ---_...... FIG.5B 52M 520B DATA METADATA SEQUENCE F ~

KEY VALUE rJLCWDA

HOUR TIME SCROLL

3059

: 00 START

2Z55

: 05 +: 05

A6E3

: 10 +: 05

9r95

: 12 +: 02

FIG. 6 16OI_ ~ 13059 2Z55 A6E3 9Y95 · C. · ~ · ~~ ___ "·" · 1 I I 1 x ~ I 2:00 1:12 (J C11 6028 Program time: 0: 05: tu: 12: 40 ~. •• -p ---. ---.--.--.-.-._.---'--" -•.••--.·_···one 606 608 ~ ~ FROM LAST I EVENT HOUR CALCULATED KEY

-

: 02

: 10 EVENT 1 +: 28 : 40 EVENT 2 FIG. 7 ~ 10UA 702 DAT ~ SDE SS! { one !; . .\............ "one do! cJu r- ~~: 00 'iD:: 20 • t •, t t [70 I Vakxes dave cabidal X51A YF11 W5W4 ~ C ·; ~ ÜB ·· _ · r - ;;; - 1 : 00 :: 06: 08 TO .. •• I

-

_ .. ~ .._ --- _.--- ~ --_.

~ WASH CALCULATE ~ TIME OFFLINE : 0 : 06 .W5W4 : 02 : 08 M5M5 .. BRAND : 10 : 0 : 12 ZIB! : 14 : 02 E5Y5 : 18 : 04 A2R2 .. :twenty BRAND

-------------

_one...-..

FIG. 8 DEVICE . DISPLAY 1 / DEVICE INPUT w ..... .a1.4 CURSOR CONTROL lil6 one"" V ' '\ ¡ /.1"1 -----------------------, .6M PROCESSOR I COMMUNICATIONS INTERFACE L -____ 1llB._ 1ill.C !: COMPUTER CENTRAL .82!