WO2008103364A1 - Systèmes et procédés pour envoyer, recevoir et traiter des signets multimedias - Google Patents

Systèmes et procédés pour envoyer, recevoir et traiter des signets multimedias Download PDF

Info

Publication number
WO2008103364A1
WO2008103364A1 PCT/US2008/002200 US2008002200W WO2008103364A1 WO 2008103364 A1 WO2008103364 A1 WO 2008103364A1 US 2008002200 W US2008002200 W US 2008002200W WO 2008103364 A1 WO2008103364 A1 WO 2008103364A1
Authority
WO
WIPO (PCT)
Prior art keywords
bookmark
stream
message
content
information
Prior art date
Application number
PCT/US2008/002200
Other languages
English (en)
Inventor
Sanghoon Sull
William J. Raduchel
Myounghoon Kim
Original Assignee
Vmark, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Vmark, Inc. filed Critical Vmark, Inc.
Publication of WO2008103364A1 publication Critical patent/WO2008103364A1/fr

Links

Classifications

    • G—PHYSICS
    • G11—INFORMATION STORAGE
    • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • G11B27/19—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier
    • G11B27/28—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording
    • G—PHYSICS
    • G11—INFORMATION STORAGE
    • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
    • G11B27/102—Programmed access in sequence to addressed parts of tracks of operating record carriers
    • G11B27/105—Programmed access in sequence to addressed parts of tracks of operating record carriers of operating discs

Definitions

  • the following disclosure is directed generally to the field of multimedia content distribution, and more particularly to the saving, sending, and processing of bookmarks identifying particular positions in the multimedia content.
  • Finding such content has become more difficult as the amount of content has increased.
  • a user who once accessed particular multimedia content needs or desires to access the content again at a later time, possibly at or from a different place.
  • a user who watches multimedia content on DTV might want to send a multimedia bookmark from DTV to his/her mobile device through a network to continue to view the content later from that exact same spot in the content, whether on the same original device or on an alternative device such as on a personal computer.
  • a user who once viewed multimedia content at work may want to continue to view the content at home.
  • a second method is to locate such content on the Internet and send a link to such content.
  • the two primary problems are finding the content on the Internet and secondly the user has to then find the particular scene within the content being referenced.
  • bookmarks mark a document or video from the beginning.
  • Conventional bookmarks mark a document such as a static web page for later retrieval by saving a link (address) to the document.
  • Internet browsers support a bookmark facility by saving an address called a Uniform Resource Identifier (URI) to a particular file.
  • URI Uniform Resource Identifier
  • Internet Explorer manufactured by the Microsoft Corporation of Redmond, Washington, uses the term "favorite" to describe a similar concept.
  • Conventional bookmarks store only the information related to the location of a file, such as the directory name with a file name or a Universal Resource Identifier (URI).
  • URI Uniform Resource Identifier
  • the files referred to by conventional bookmarks are treated in the same way regardless of the data formats for storing the content.
  • a simple link is used for multimedia content.
  • a URI is used to link to a multimedia content file through the Internet.
  • the multimedia content associated with the bookmark is always played from the beginning.
  • Access to the same position of a multimedia content indicated by a bookmark is not trivial.
  • VOD video on demand
  • users can easily access the same position of a multimedia content later by using both URI and media time because users are watching the same AV stream which resides in a server managed by VOD service providers.
  • multimedia content is accessible to users in the form of different AV streams such as in a variety of bit rates, encoding/compression methods such as MPEG-2, H.264 and proprietary encoding solution from Microsoft, Inc., delivery mediums, service providers, and others. Accessing multimedia content at the exact same point across different devices can present numerous problems.
  • the current invention is a simple and elegant solution to the problem of bookmarking particular locations in a multimedia stream.
  • a cellular phone or other mobile device to send a text message or e-mail, or by using a computer to send an e-mail, HTTP Get and Post methods or instant message
  • the invention allows for the marking of an exact scene in any AV stream being watched.
  • the invention further allows the user to fine tune the bookmark of the exact scene to save for later personal use, share with friends or on a social network, or to share with the general public by posting on a bulletin board.
  • a bookmark relevant to a first AV stream can be mapped or transformed into a bookmark relevant to a second AV stream having content that is the same or comparable to the content of the first AV stream.
  • accessing multimedia content at the exact same point across different devices can be accomplished using the invention.
  • FIG. IA illustrates a system for sending, receiving and processing a multimedia bookmark between different types of client devices, according to an embodiment of the present disclosure.
  • FIG. IB illustrates an exemplary system for sending, receiving and processing a multimedia bookmark between two different types of client devices, e.g., DTV and PC, according to an embodiment of the present disclosure.
  • FIG. 1C is an exemplary block diagram for the media localization server for managing the information on the contents, according to an embodiment of the present disclosure.
  • FIG. 2 illustrates an exemplary timestamp relationship between a variety of AV streams which correspond to the same content, according to an embodiment of the present disclosure.
  • FIG. 3A is an exemplary flowchart illustrating a scheme for generating a multimedia bookmark according to an embodiment of the present disclosure.
  • FIG. 3B is an exemplary flowchart illustrating a scheme for creating and sending a bookmark by using a cellular phone text message according to the present disclosure
  • FIG. 4 illustrates an exemplary scheme of generating the AV content information for the multimedia bookmark generated by a user, according to an embodiment of the present disclosure.
  • FIG. 5A is an exemplary flowchart illustrating a scheme for processing a received multimedia bookmark, according to an embodiment of the present disclosure.
  • FIG. 5B is an exemplary flowchart illustrating a method for processing a multimedia bookmark at the client devices, according to an embodiment of the present disclosure.
  • FIG. 5C is an exemplary flowchart illustrating a method for processing a multimedia bookmark at the bookmark server, according to an embodiment of the present disclosure.
  • FIG. 6 is an exemplary flowchart illustrating a process performed in the media localization server, according to an embodiment of the present disclosure.
  • FIG. 7 illustrates an exemplary AV pattern matching process for locating a frame accurate bookmarked position, according to an embodiment of the present disclosure.
  • the present disclosure is directed to a process of saving, sending, receiving, and processing audiovisual bookmarks between a variety of consumer electronic devices to access the same position of a multimedia content regardless of the source, format, and bit rate, among others.
  • the disclosed method is especially useful in enabling the viewing of content on different devices, such as starting to watch an Internet video on your work computer and then to complete your viewing of the same video at the exact spot you left off on your home television.
  • the disclosed method is also especially useful in allowing for the ease of sharing of live TV content within a social network. Instead of calling friends to tell them about some exact scene viewed, a viewer can use a cellular phone or computer or other such devices to reference an exact scene and share such exact scene among friends and a social network without having to take an additional step to find such content on the Internet.
  • ATSC ATSC Advanced Television Systems Committee, Inc.
  • ATSC is an international, non-profit organization developing voluntary standards for digital television.
  • countries such as U.S. and Korea adopted ATSC for digital broadcasting.
  • a more extensive explanation of ATSC may be found at "ATSC Standard A/53C with Amendment No. 1 : ATSC Digital Television Standard, Rev. C," (see World Wide Web at atsc.org). More description may be found in "Data Broadcasting: Understanding the ATSC Data Broadcast Standard” (McGraw- Hill Professional, April 2001) by Richard S. Chernock, Regis J. Crinon, Michael A. Dolan, Jr., John R.
  • AVC Advanced Video Coding (H.264) is the newest video coding standard of the ITU-T Video Coding Experts Group and the ISO/IEC Moving Picture Experts Group.
  • An explanation of AVC may be found at "Overview of the H.264/ AVC video coding standard", Wiegand, T., Sullivan, G.J., Bjntegaard, G., Luthra, A., Circuits and Systems for Video Technology, IEEE Transactions on , Volume: 13 , Issue: 7 , July 2003, Pages:560 - 576; another may be found at "ISO/IEC 14496-10: Information technology - Coding of audio-visual objects - Part 10: Advanced Video Coding" (see World Wide Web at iso.org); Yet another description is found in “H.264 and MPEG-4 Video Compression” (Wiley) by Iain E.G.
  • MPEG-I and MPEG-2 are alternatives or adjunct to AVC and are considered or adopted for digital video compression.
  • Communication Network Any local or wide area network (wired, wireless) including Internet, Wi-Fi and Bluetooth.
  • DMB Digital Multimedia Broadcasting (DMB), first commercialized in
  • DTV Digital Television is an alternative audio-visual display device augmenting or replacing current analog television (TV) characterized by receipt of digital, rather than analog, signals representing audio, video and/or related information.
  • Video display devices include Cathode Ray Tube (CRT), Liquid Crystal Display (LCD), Plasma and various projection systems.
  • Digital Television is more fully described at "Digital Television: MPEG-I, MPEG-2 and Principles of the DVB System" (Butterworth-Heinemann, June, 1997) by Herve Benoit.
  • DVB Digital Video Broadcasting is a specification for digital television broadcasting mainly adopted in various countered in Europe adopt. A more extensive explanation of DVB may be found at "DVB: The Family of International Standards for Digital Video Broadcasting" by Ulrich Reimers (see World Wide Web at dvb.org). ATSC is an alternative or adjunct to DVB and is considered or adopted for digital broadcasting used in many countries such as the U.S. and Korea.
  • DVR Digital Video Recorder (DVR) is usually considered a Set Top Box
  • Media time Media time is defined from the start of a video stream or file (0) to the end (the length of video).
  • MPEG-2 Moving Picture Experts Group - Standard 2 (MPEG-2) is a digital video compression standard designed for coding interlaced/noninterlaced frames. MPEG-2 is currently used for DTV broadcast and DVD. A more extensive explanation of MPEG-2 may be found on the World Wide Web at mpeg.org and "Digital Video: An Introduction to MPEG-2 (Digital
  • MPEG-4 Moving Picture Experts Group - Standard 4 (MPEG-4) is a video compression standard supporting interactivity by allowing authors to create and define the media objects in a multimedia presentation, how these can be synchronized and related to each other in transmission, and how users are to be able to interact with the media objects.
  • MPEG-4 Moving Picture Experts Group - Standard 4
  • PCR Program Clock Reference (PCR) in the Transport Stream (TS) indicates the sampled value of the system time clock that can be used for the correct presentation and decoding time of audio and video.
  • TS Transport Stream
  • STB Set-top Box is a display, memory, or interface device intended to receive, store, process, repeat, edit, modify, display, reproduce or perform any portion of a program, including personal computer (PC) and mobile device.
  • PC personal computer
  • STT System Time Table is a small table that provides the current date and time of day information in ATSC.
  • Digital Video Broadcasting has a similar table called a Time and Date Table (TDT).
  • TTT Time and Date Table
  • TV Television generally a picture and audio presentation or output device; common type include cathode ray tube (CRT), plasma, liquid crystal and other projection and direct view systems, usually with associated speakers.
  • CTR cathode ray tube
  • plasma plasma
  • liquid crystal liquid crystal
  • W3 C WorldNet
  • a multimedia bookmark for an AV stream delivered through a broadcasting network or the Internet can be generated by any of a wide variety of devices such as mobile devices, e.g., cell phones, PDAs, laptops, etc, or non-mobile devices such as PCs, phones on landlines, or the like, and can contain one or more of the following bookmark information:
  • AV content information such as reduced-sized frame(s) or signal pattern captured at a bookmarked position
  • Generator of bookmark (unique identifier and/or attributes of the generator (e.g., time zone where the generator lives, etc.))
  • the bookmark information includes not only positional information (1 and
  • Patent Application No. 09/91 1,213 filed 07/23/2001 Publication No.2002/00692178 the entirety of which is hereby incorporated by reference, and U.S. Patent Application No. 11/071,894 filed
  • the positional information can be represented by using one of media locators or timestamps such as broadcasting time (for example, system_time in STT for ATSC) as disclosed in the copending U.S. Patent Application No. 10/369,333 filed 2/19/2003
  • Patent Application No. 1 1/069,750 filed 3/01/2005 Publication No. 2005/0193425) the entirety of which is hereby incorporated by reference
  • U.S. Patent Application No. 1 1/221,397 filed 09/07/2005 Publication No.2006/00647166 the entirety of which is hereby incorporated by reference.
  • the AV content information (3) may be composed of AV low-level features and textual features.
  • the AV features are the information, for example, obtained by capturing or sampling the AV signal at or around the bookmarked position.
  • the AV features can be thumbnail image(s) of the captured video frame(s) which could be encoded in JPEG, for example, and low-level visual feature vectors like color histogram for one or more of the frames.
  • the AV features can also be the sampled audio signal (typically of short duration) and its visualized image.
  • the textual features (4) are text information specified by the user, as well as delivered with the AV stream.
  • the descriptive information of the program (7) can be obtained from broadcast EPG, if the multimedia stream is digitally broadcast, and comprises channel number, TV program title, start date/time/duration of the TV program, episode number if the program belongs to a series, synopsis, etc.
  • the date and time (8) when the bookmark is generated and the information on the generator of a bookmark (9) such as name can be also included.
  • a user can draw an area of interests on a bookmarked frame where such information could be also contained in multimedia bookmark information.
  • the client environment (11) contains information comprising what delivery medium was used for the bookmarked stream, service provider, and device type.
  • the delivery medium could be one of terrestrial digital broadcasting such as ATSC or DVB, digital cable, digital satellite, mobile such as DMB, Internet, DVD (or Blue-Ray), and others.
  • the service provider contains the information on the service provider for a client device such as name(s) of the local digital terrestrial/mobile broadcasting network, IPTV service provider, local digital cable network, and satellite service provider.
  • the two types of information on delivery medium and service provider can be unified to one field which uniquely identify the service available to client, for example, "ATSC: ABC-LA", or other equivalents.
  • the device type contains the information on the client device such as resolution and a type of AV decoder (for example, MPEG-2 or H.264) that are appropriate for the client device.
  • a multimedia bookmark can be sent to a designated address in the form of a multimedia bookmark message containing the multimedia bookmark information along with user information such as sender and receiver. For example, if a multimedia bookmark is delivered via e-mail (or other equivalents such as messaging), email addresses of sender and receiver will be needed. In case a multimedia bookmark is delivered via Bluetooth, a combination of the Personal Identification Number (PIN) and a Bluetooth address can be used to identify other Bluetooth devices.
  • PIN Personal Identification Number
  • Bluetooth can be used to identify other Bluetooth devices.
  • [0065] Suppose that a user of DTV with DVR generates a multimedia bookmark on the broadcast TV video program stored in their local or associated storages or live TV, sends the multimedia bookmark to a user of PC through communication network . In this case, the recipient of the multimedia bookmark cannot simply play the video from the bookmarked position using a broadcasting time unless the same video stream is available in the PC's local storage.
  • bookmarks It is disclosed herein methods and systems of sending, receiving, and processing multimedia bookmarks to access the exact position of the multimedia content indicated by a multimedia bookmark between different types of client devices. In that manner, bookmarks may be coordinated amongst different platforms.
  • FIG. IA illustrates a system for sending, receiving and processing multimedia bookmarks between different types of client devices (DTV with DVR, Mobile, and PC with DTV tuner).
  • the broadcaster 1 10 broadcasts video programs to client devices of TV viewers (client 120, 180 and 130) through the broadcasting network 160 such as the Internet, cable, satellite, mobile and terrestrial networks.
  • client devices of TV viewers client 120, 180 and 130
  • the broadcasting network 160 such as the Internet, cable, satellite, mobile and terrestrial networks.
  • a user of each client device can make a multimedia bookmark on the program where the corresponding bookmark information is generated at the bookmarked position of the program.
  • the generated multimedia bookmark might be stored in its local storage.
  • the user can send a multimedia bookmark to a user of another client with a designated address through the communication network 170 such as the Internet.
  • the client who received the bookmark When the same stream pointed to by the received multimedia bookmark is already recorded in the received client, for example, from DTV to DTV, a user of the client who received the bookmark can access the stream from the bookmarked position by using the bookmark information.
  • the client who received the bookmark sends a request along with the received bookmark information and its information on client environment to the media localization server 150 through the communication network 170. Then, the media localization server 150 returns the information on how or where the same content as the bookmarked stream can be accessed in the form of a URl, for example, and the modified bookmarked position corresponding to the same content back to the client who requested such information.
  • the client who received access information from the media localization server 150 can access the content from one of the media host 140 through the communication network 170.
  • the operation of the media localization server 150 is described in detail in FIG. 6.
  • the media localization server 150 may be thought of as a clearinghouse that maps or translates bookmarks from a first AV stream available to the first user to a second AV stream available to a second user.
  • a user can also create a bookmark by using cellular phone text messaging (e.g., short message service (SMS)), personal computer instant messaging, e-mail, among others.
  • SMS short message service
  • bookmark server 190 While a user is watching a recorded or live TV program, the user can have a bookmark created and sent whenever needed by just sending a text message, e-mail, or the like, to him/herself or the user's friends through the bookmark server 190.
  • the sender of the message is known to the bookmark server 190 via a pre-registered profile.
  • the receiver of the bookmark generated by bookmark server 190 can be the sender him/herself by default or specified by the sender.
  • the descriptive information and the client environment of the bookmark is automatically obtained or filled in at the bookmark server 190 by using a TV program or broadcaster-specific recipient mobile number (or short code, a 5 number text message recipient number) to identify what is being watched.
  • the text itself such as a short word to identify the TV program (for example, "vmarkABC-LA") for a user to bookmark the ABC-LA channel can be used as the descriptive information of the bookmark.
  • the positional information or broadcasting time for the bookmark can be automatically obtained from the short message service (SMS) received time at the bookmark server 190 wherein the positional information can be adjusted by considering the typical delay (1-2 second).
  • SMS short message service
  • the recipient of the bookmark accesses the bookmark server 190 and selects the received bookmark.
  • the recipient of the received bookmark does not have access to the AV steam identified in the bookmark, the recipient can connect to the media localization server 150 to obtain a content URI from which the bookmark program is available (for example, by streaming) and an appropriate timestamp which is mapped from the SMS received time accounting for commercial breaks to determine the actual commercial-free playout time (i.e., what is the position when playing out from the file, which is pointed by the content URI, that has no commercials). Then, the recipient of the bookmark can play from the file pointed to by the content URI without the commercials.
  • FIG. IB illustrates an exemplary system for sending, receiving and processing a multimedia bookmark between two different types of client devices DTV with DVR 120, and PC 130 with an optional DTV tuner 131.
  • the broadcaster 1 10 broadcasts HDTV TV video programs (for example, in 1080i format) to the clients 120 and 130 through the broadcasting network 160 such as the Internet, cable, satellite, and terrestrial networks.
  • the broadcast TV programs might be recorded in local storages 122 and 132 of the clients, and played with media players 124 and 134 on the corresponding display devices 125 and 135 whenever a user wants.
  • a user of DTV 120 can make a multimedia bookmark on the program where the corresponding bookmark information is generated and stored in its local storage 122 by the multimedia bookmark client 126. Also, the user can send the multimedia bookmark to the client PC 130 via the bookmark server 190, which the user selects among other clients, by using the bookmark client 126 through the communication network 170 such as Bluetooth or the Internet.
  • the multimedia bookmark is saved in the local storage 194 at the bookmark server 190. Then, a user of PC 130 can receive the a multimedia bookmark from bookmark manager 193 of the bookmark server 190.
  • the program pointed by the multimedia bookmark sent by the client 120 has already been recorded in the local storage 132 of the client 130, the program can be played from the bookmarked position by using the information such as a URI and bookmarked position included in the multimedia bookmark if the clients 120 and 130 use the same file naming scheme and predefined directory for storing recorded streams. For example, all recorded programs that are broadcast on February 2005 are stored in a directory whose name is "200502," and a recorded program that was broadcast at 9:30 PM on 16 th February 2005 at channel 205 has a file name such as "20050216-2130-205.ts.” The file name is used in the field "URI of a bookmarked file" of the bookmark information in the present disclosure, for example.
  • the search agent (128, 138) at each client (120, 130) can resolve the locations of the stored programs by associating the descriptive information and client environment in the received multimedia bookmark with the corresponding information on the recorded streams in the local storage 132.
  • the program pointed to by the multimedia bookmark sent by the client 120 was not recorded in the local storage 132 of the client 130, the program can be recorded later when it is rebroadcast on the same channel or available from other channels by searching broadcast EPG for the same program based on the descriptive information (for example, program title, episode number, etc) included in the received multimedia bookmark, wherein the searching broadcast EPG for the same program is also performed by the search agent 138, which is requested by the multimedia bookmark client 136.
  • the positional information for example, system_time in STT for ATSC
  • the received multimedia bookmark cannot be simply used to access the same position of the multimedia content because the broadcasting time of the program pointed by the received multimedia bookmark is different from that of the program to be recorded later.
  • the search client 138 it is necessary for the search client 138 to connect to the media localization server 150 (service discovery) and send a request for a modified timestamp along with the received bookmark information and additional information on the client environment for the client 130 to the media localization server 150 through the communication network 170.
  • the media localization server 150 maps or modifies the timestamp in the received bookmark information to the correct timestamp for the stream available at the client 130 and sends it back to the client 130 through the communication network 170.
  • an AV stream corresponding to the same multimedia content as the TV program can be downloaded or streamed to the PC client 130 as follows: First, the search agent 138 connects to the media localization agent of 157 the media localization server 150 (service discovery) and sends a request for a content URI from which the AV stream corresponding to the same multimedia content as the TV program can be available and the modified timestamp for the stream to the media localization server 150 through the communication network 170 wherein the search agent 138 sends the bookmark information received from the user of the client 120 to the media localization server 150 (for example, the time-stamp for bookmarked position, program title, channel number, start date/time of the program, delivery medium such as ATSC and service provider such as ABC-LA) and additional information on the client environment for the client 130 (for example, digital cable and Comcast) in the form of a bookmark message.
  • the search agent 138 sends the bookmark information received from the user of the client 120 to the media localization server 150 (for example, the time-stamp for bookmarked position
  • the media localization server 150 analyzes the received information and sends the content URI and a timestamp or media time corresponding to the bookmarked position represented in broadcasting time (or system time in STT in case of ATSC) back to the search agent 138 in the form of an updated bookmark message.
  • the search agent 138 connects to the media host 140 as directed by the content URI from the media localization server 150 and sends the request for the stream pointed by the content URI to the media host 140 through the communication network 170.
  • the search engine 148 searches its database for an appropriate AV stream or file of the same multimedia content as the TV program that is downloaded or streamed from the storage 142 by the download server 144 or streaming server 146 of media host 140 to the client 130 through the communication network 170, respectively.
  • FIG. 1C is an exemplary block diagram for the media localization server 150 for managing the information on the contents available to users through a variety of the client environments wherein the information on the contents comprises descriptive information of the contents (for example, channel number, program title, start date/time and duration of the program, delivery medium, service provider, device type, etc.) and the location where the contents are available (for example, in the form of URI).
  • the media localization server 150 includes modules for receiving and decoding broadcast streams (for example, tuners 151, and AV decoders 152).
  • the EPG parser 154 extracts the EPG data such as channel number, program title, start date/time, duration, rating (if available) and synopsis from each channel and stores the information into the content & timestamp database 158.
  • the timestamp processor 156 associates timestamps adequate to each program (for example, system_time for ATSC) with frames of the program and stores the information to the content & timestamp database 158.
  • the media localization agent 157 receives a request from a client to the media localization server 150.
  • the content locator 155 locates the stream or program from the content & timestamp database 158 by using the bookmark information such as descriptive information of the content and the information on the client environment of the client 120 as well as the information on the client environment of the client 130 who sent request to the media localization server 150.
  • the content locator 155 outputs the URI of the located program from which the program can be streamed or downloaded.
  • the time stamp processor 156 also maps the timestamp in the received bookmark to the timestamp of the program located by the content locator 155 by using the information in the contents & timestamp database 158.
  • the AV analyzer 153 performs an AV signal pattern matching using one of the existing image processing techniques.
  • the AV analyzer 153 searches the AV stream which is first located by the content locator 155 for the exact frame that is matching to the content information such as reduced-image of the multimedia bookmark, and then outputs a frame-accurate timestamp corresponding to the bookmark information.
  • the requested information such as URI and the mapped (or frame- accurate if needed) timestamp information is then sent back to the client who requested by the media localization agent 157.
  • FIG. 2 illustrates an exemplary timestamp relationships between a variety of AV streams, which correspond to the same content (for example, a football game), that could be in different encoding formats or available through different delivery mediums or service providers.
  • the AV stream 201, 202, 203 and 204 correspond to the same AV content, but are delivered through mobile network by ABCm, terrestrial network by ABC-LA, terrestrial network by ABC- New York, and the Internet by ABCi, respectively.
  • the timestamp for the AV stream 201 is represented by the local time delivered through the mobile network.
  • the timestamp for the AV streams 202 and 203 is represented by the local broadcasting time delivered through the terrestrial network (for example, system_time for ATSC).
  • the timestamp for the AV stream file 204 is represented by media time.
  • the timestamps 205, 207, 209, and 211 correspond to the start position of the same content, and the timestamps 206, 204, 210, and 212 correspond to the end position of the same content.
  • There is an offset 220 between the timestamps 205 and 207 due to the delay caused by the time it takes to transmit the content from a broadcaster to the mobile network although the same content is broadcast in the same local area.
  • the streams 202 and 203 include two commercial breaks 214/215 and 216/217, respectively.
  • the commercial breaks are not included in the AV stream file 204 which could be available later than the local broadcasting time 205 by the amount of delay 222, and thus the intervals (214/216 and 215/217) correspond to the media times 218 and 219, respectively.
  • the frame-accurate relationship between the timestamps can be obtained by using a technique similar to the AV signal pattern matching technique described later.
  • This type of timestamp relationship (start and end times of the programs, start and end time of commercial breaks, and offset) between a variety of AV streams is stored in the content & timestamp database 158 in the media localization server 150 to map the bookmarked position to a timestamp appropriate for the AV stream available to the client 130.
  • FIG. 3A is an exemplary flowchart illustrating a scheme for generating a multimedia bookmark, according to an embodiment of the present disclosure.
  • the operation of multimedia bookmark client 126 of DTV 120 starts at step 302. While a user of DTV 120 is watching a recorded program in its local storage or live TV, a user bookmarks a position of the program via a button on remote control for DTV 120 at step 304. Otherwise, the process returns at step 316.
  • the multimedia bookmark client 126 generates the positional information represented by using one of media locators such as broadcasting time at step 306. Then, the AV content information is generated by capturing or sampling the AV signal at or around the bookmarked position at step 308.
  • the AV features can be thumbnail image(s) of the captured video frame(s) which could be encoded in JPEG, for example, and low-level visual feature vectors like color histogram for one or more of the frames. It is noted that the use of one thumbnail image on the bookmark position could not be enough to find the accurate or unique position since some of AV streams include similarly-looking frames, and thus it would be desirable to additionally utilize the AV features near the bookmarked frame or position, which is described in detail later with FIGs. 4 and 7.
  • the textual annotations could be obtained through the remote metadata server if needed at step 308. Also, the user can input an optional bookmark title at step 308.
  • FIG. 3B is an exemplary flowchart illustrating a scheme for creating and sending a bookmark by using a cellular phone text messaging, according to an embodiment of the present disclosure.
  • the process starts at step 322. While a user of DTV 120 is watching a recorded program in its local storage or live TV, the user can create a cellular phone text message at step 324. Otherwise, the process returns at step 326.
  • a user can input texts representing a short word to identify the TV program.
  • the user can use a TV program or broadcaster-specific recipient mobile number (or short code, a 5 number text message recipient number) to identify what is being watched at step 330. Then, the text message is sent to the bookmark server 190.
  • the recipient of the multimedia bookmark can access the bookmark received from the bookmark server 190 at the client 130 or can play the bookmark by remotely logging into the bookmark server 190.
  • FIG. 4 illustrates an exemplary scheme of generating the AV content information for the multimedia bookmark generated by a user of the client 120 at the frame 403 of a TV program delivered through ATSC.
  • the bookmarked frame is assumed to be in the shot 409 ranging from the frame 401 to the frame 402 wherein shot boundaries are detected by using one of the existing image processing techniques.
  • a reduced-sized or thumbnail image corresponding to the frame 403 is generated by the AV analyzer 127.
  • the temporal positions 406, 407 and 408 correspond to the broadcast system_times in STTs for ATSC, respectively.
  • the system_time for the bookmarked frame 403 could be the system_time at 407 obtained by finding the first- occurring and nearest system_time preceding the frame 403.
  • the frame-accurate temporal localization to a specific frame may be achieved by interpolating the values of system_times in STTs in the resolution of preferably at least or about 30Hz by using a clock, such as PCR, or other equivalents.
  • a clock such as PCR
  • the use of more AV features could contribute to the accurate frame access.
  • the frames 404 and 405 look different from the bookmarked frame 403 since the frames 404 and 405 are included in the shots different from the shot 409.
  • two thumbnail images for the frames 404 and 405 adjacent to the bookmark frame 403 and their respective distances (410 and 411) to the frame 403 in the unit of time or a number of frames are also generated for the multimedia bookmark and stored in the local storage.
  • An exemplary scenario of sending a multimedia bookmark from a user of the client 120 to a user of the client 130 is as follows: If the user of the client 120 wants to send the bookmark selected from a list of multimedia bookmarks stored at the local storage to the user of the client 130, the user opens the stored multimedia bookmarks where the bookmark thumbnails and additional information are displayed. Then, the user selects a bookmark and sends it to other user by e-mail. Then, the multimedia bookmark client 126 generates the message of a multimedia bookmark. Table 1 shows an example of a bookmark message written in XML. After all, the bookmark message is sent to the client PC 130. ⁇ BookmarkMessage> TABLE 1
  • FIG. 5 A is an exemplary flowchart illustrating a scheme for processing a multimedia bookmark received at, for example, client PC 130 from the bookmark server 190 or other device of 120, 180 of FIG. IA, according to an embodiment of the present disclosure.
  • the multimedia bookmark client 136 of client PC 130 starts at step 502.
  • the multimedia bookmark client 136 receives a multimedia bookmark message at step 504.
  • a search agent 138 searches the local storage for the AV stream pointed to by the received bookmark by using the bookmark information such as positional information, descriptive information, and client environment at step 506. If the AV stream is available in the local storage, the received bookmark is saved in the local storage at step 516 and the process returns at step 518.
  • the search agent 138 connects to the media localization agent (157 in FIG. 1C) of the media localization server 150 (service discovery) and sends a request for an AV stream URI from which the stream can be available and the timestamp for the stream to the media localization server 150 through the communication network 170 at step 508.
  • the localization server 150 may be thought of as providing a "clearing house" type of service in which bookmarks created for one source of content are translated or mapped to bookmarks identifying the same location in a different source for the same or comparable (e.g., same program but without commercials) content,.
  • the search agent 138 also sends the bookmark information and additional information on client environment for the PC 130 in the form of bookmark message to the media localization server 150.
  • the search agent 138 receives an updated bookmark message from the media localization agent 157 of the media localization server 150 wherein the updated bookmark message contains the information on the AV stream URI and an appropriate timestamp at step 510.
  • the search agent 138 sends a request for downloading the AV stream from the media host 140 as directed by the AV stream URI received from the media localization server 150 at step 512 wherein an appropriate AV stream of the same multimedia content as the bookmarked stream at the client 120 is downloaded from the media host to the local storage of the PC 130.
  • the updated bookmark message from the media localization server 150 is saved in the local storage of the client 130 at step 514 and the process returns at step 518. Then, the user of the client PC 130 can play the bookmarked AV stream whenever needed.
  • the process shown in FIG. 5 A can be performed when the user of the client 130 wants to play the bookmarked stream wherein the AV stream can be streamed to the client 130 instead of downloading the AV stream as shown at step 512.
  • This process can be easily modified to handle the case when the same content are available in the storages of both the sending and receiving clients, but the same content is delivered in different way via ATSC and mobile, for example.
  • the stream corresponding to the content does not need to be downloaded at step 512 to the bookmark receiving client 130 which just needs to send request for an appropriately modified timestamp for the bookmark position along with the received bookmark information and additional information on client environment for the client 130 to the media localization server 150.
  • FIG. 5B shows an exemplary flowchart illustrating a process of processing a multimedia bookmark from a multimedia bookmark list, according to an embodiment of the present disclosure.
  • the multimedia bookmark client 136 of the client 130 starts at step 522.
  • the multimedia bookmark client 136 reads the list of the multimedia bookmarks which are received from the bookmark server 190 and stored in the local storage 132, and displays it at step 524.
  • the media player 134 of the client 130 starts to play from bookmarked position of the program at step 530. Otherwise, the process returns at step 528.
  • FIG. 5C is an exemplary flowchart illustrating a scheme of processing a multimedia bookmark from a multimedia bookmark list or bookmark BBS (Bulletin Board System) by connecting to the bookmark server 190, according to an embodiment of the present disclosure.
  • the process starts at step 542.
  • a user of client devices without employing the multimedia bookmark client such as 136 can connect (or login) to the web server 191 of the bookmark server 190 at step 544, play the AV program pointed to by a multimedia bookmark or one of multimedia bookmarks posted on the bookmark BBS.
  • the bookmark manager 193 of the bookmark server 190 reads the multimedia bookmark information from the local storage 194 and displays them at step 550. Otherwise, the bookmark BBS server 192 displays the bookmark BBS at step 548.
  • the search agent 195 connects to the media localization agent 157 of the media localization server 150 and sends a request for an AV stream URI from which the stream can be available and the appropriate timestamp for the stream to the media localization server 150 through the communication network 170 at step 556.
  • the search agent 195 also sends the bookmark information and additional information on the client environment in the form of bookmark message to the media localization server 150 at step 556.
  • the search agent 195 then receives an updated bookmark message from the media localization agent 157 of the media localization server 150 wherein the updated bookmark message contains the information on the AV stream URI and an on the client device such as resolution and a type of AV decoder (for example, MPEG-2 or H.264) timestamp at step 558.
  • the search agent 195 sends a request for streaming the AV stream from the media host 140 as directed by the AV stream URI and the timestamp received from the media localization server 150 at step 560.
  • the media player of the client device starts to play from the updated bookmarked position of the AV stream. If a selected multimedia bookmark to play is sent via a text message generated by a cellular phone, the bookmarked position is not usually accurate and thus the media player of the client device can provide the control of auto-rewind to adjust the position.
  • the process is terminated at step 554.
  • FIG. 6 is an exemplary flowchart illustrating a process performed in the media localization server 150 when the bookmarked AV stream is not recorded in the storage of the client 130.
  • the process starts at step 602 with the receipt of a request from the client 130 (i.e., the requesting party) at step 604. If there is a request, the content locator 155 locates the AV stream from the contents & timestamp database 158 at step 606 by using the bookmark information such as descriptive information of the content and the information on the client environment of the client 120 as well as the information on the client environment of the client 130 who sent request to the media localization server 150.
  • the content locator 155 outputs the URI of the located AV stream from which the program can be streamed or downloaded to the client 130.
  • the positional information (for example, broadcasting time) included in the received multimedia bookmark cannot be used to access the corresponding position of the AV stream from a media host directed by the AV stream URI found at step 606 because the AV stream of the same multimedia content that is to be downloaded or streamed from the select media host could use the timestamp different from the recorded TV program or stream in the local storage 122 (for example, broadcasting time versus media time) and further might not contain commercial breaks.
  • the time stamp processor 156 accesses the content & timestamp database 158 and then transforms or maps the timestamp in the bookmark that is originally generated in the client 120 to the timestamp of the corresponding position of the stream located by the AV stream URI at step 608.
  • the timestamp processing step 608 does not provide a frame-accurate timestamp and a frame accurate access to the bookmarked position is needed, the AV analyzer 153 performs an AV signal pattern matching using one of the existing image processing techniques.
  • the AV analyzer 153 searches the local version, which is stored in the local storage 159, of the AV stream located by the AV stream URI for the exact frame that is matching to the content information such as reduced-image of the multimedia bookmark generated in the client 120, and then outputs a frame-accurate timestamp corresponding to the bookmark information at step 610. Then, the requested information such as URI and the mapped timestamp information are sent back to the client 130 who requested at step 612.
  • This process can be easily modified when the client 130 send a request for an appropriately modified timestamp for the bookmark position when the bookmarked AV stream is recorded in the local storage of the client 130 by omitting the step of locating content URI (step 606).
  • the process returns at step 614.
  • FIG. 7 illustrates an exemplary AV pattern matching process for locating a frame accurate bookmarked position that is performed by the AV analyzer 168 of the media localization server 150 by using three thumbnails contained in the AV content information of the multimedia bookmark information although one or more thumbnail images instead of three or other AV low- level features or patterns can be used.
  • the AV file 701 stored in the local storage 159 in the media localization server 150 is the same as the AV stream located by the URI by step 606 (or, it could be the transcoded version in low-bit rate having the same but reduced-size frames).
  • the start of the AV file 701 corresponds to media time zero seconds and the end of the AV file 701 corresponds to the media time 703.
  • the three thumbnails 707 consisting of the bookmarked frame 403 and its two adjacent thumbnails 404 and 405 are the AV content information in the multimedia bookmark.
  • the media time 704 (the timestamp mapped at step 608) is not assumed to be frame accurate.
  • the AV analyzer 168 searches the AV file 701 for three frames best matching three thumbnails 707 subject to the temporal distances 410 (or 708) and 411 (or 709) within the search range 706 centered at the media time 704 by using one of the existing image processing techniques, resulting in the frame accurate media time corresponding to the bookmarked frame 403. To speed up the matching process, only intra- frames of the video stream can be searched at the expense of frame accuracy. This process can be equally applied to the case of audio only streams.
  • the AV pattern matching technique described in FIG. 7 can be performed by the AV analyzer such as 127 in client device to appropriately map the timestamp for a bookmark position between different types of client devices (for example, between DTV and mobile device) when the same content are available in those client devices without connecting to the media localization server 150.

Abstract

En utilisant un téléphone cellulaire ou un autre dispositif mobile pour envoyer un message textuel ou un courrier électronique, ou encore en utilisant un ordinateur pour envoyer un courrier électronique, des procédés HTTP Get et Post ou un message instantané, l'invention permet le marquage d'une scène exacte dans tout flux AV regardé. L'invention permet également à l'utilisateur d'ajuster le signet de la scène exacte pour une sauvegarde en vue d'un usage personnel ultérieur, pour un partage avec des amis ou sur un réseau social, ou pour un partage avec le public général par l'affichage sur un panneau d'affichage. De plus, en utilisant un type de « centrale de signets », un signet lié à un premier flux AV peut être représenté ou transformé en un signet lié à un second flux AV ayant un contenu qui est identique ou comparable au contenu du premier flux AV. Ainsi, un accès au contenu multimédia exactement au même endroit, sur différents dispositifs, peut être réalisé en utilisant l'invention.
PCT/US2008/002200 2007-02-20 2008-02-20 Systèmes et procédés pour envoyer, recevoir et traiter des signets multimedias WO2008103364A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US90222807P 2007-02-20 2007-02-20
US60/902,228 2007-02-20

Publications (1)

Publication Number Publication Date
WO2008103364A1 true WO2008103364A1 (fr) 2008-08-28

Family

ID=39710380

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2008/002200 WO2008103364A1 (fr) 2007-02-20 2008-02-20 Systèmes et procédés pour envoyer, recevoir et traiter des signets multimedias

Country Status (1)

Country Link
WO (1) WO2008103364A1 (fr)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2010043269A1 (fr) * 2008-10-17 2010-04-22 Telefonaktiebolaget Lm Ericsson (Publ) Procédé et appareil destinés à être utilisés dans un réseau de télévision commuté par paquets
CN101860724A (zh) * 2010-05-11 2010-10-13 中兴通讯股份有限公司 交互式网络电视系统中点播节目的方法及装置
WO2011046608A1 (fr) * 2009-10-14 2011-04-21 Thomson Licensing Mise à jour automatique de contenus multimédia par l'intermédiaire d'un réseau social en ligne
WO2011091296A1 (fr) * 2010-01-21 2011-07-28 Interdigital Patent Holdings, Inc. Transfert de session et support de mise en signet pour services de diffusion en flux
WO2011143050A2 (fr) 2010-05-13 2011-11-17 Microsoft Corporation Signets éditables partagés par l'intermédiaire d'un réseau social
EP3401765A1 (fr) * 2009-04-30 2018-11-14 TiVo Solutions Inc. Raccourcis de programme
US11550867B1 (en) * 2021-09-09 2023-01-10 Bank Of America Corporation Apparatus and methods for bookmark sharing

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050210145A1 (en) * 2000-07-24 2005-09-22 Vivcom, Inc. Delivering and processing multimedia bookmark

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050210145A1 (en) * 2000-07-24 2005-09-22 Vivcom, Inc. Delivering and processing multimedia bookmark

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10645456B2 (en) 2007-01-03 2020-05-05 Tivo Solutions Inc. Program shortcuts
WO2010043269A1 (fr) * 2008-10-17 2010-04-22 Telefonaktiebolaget Lm Ericsson (Publ) Procédé et appareil destinés à être utilisés dans un réseau de télévision commuté par paquets
EP3401765A1 (fr) * 2009-04-30 2018-11-14 TiVo Solutions Inc. Raccourcis de programme
KR101774039B1 (ko) * 2009-10-14 2017-09-01 톰슨 라이센싱 온라인 소셜 네트워크에 대한 자동적 미디어 자산 업데이트
WO2011046608A1 (fr) * 2009-10-14 2011-04-21 Thomson Licensing Mise à jour automatique de contenus multimédia par l'intermédiaire d'un réseau social en ligne
CN102714762A (zh) * 2009-10-14 2012-10-03 汤姆森许可贸易公司 经由在线社交网络的自动媒体资产更新
US9769414B2 (en) 2009-10-14 2017-09-19 Thomson Licensing Automatic media asset update over an online social network
CN102714762B (zh) * 2009-10-14 2016-08-31 汤姆森许可贸易公司 经由在线社交网络的自动媒体资产更新
WO2011091296A1 (fr) * 2010-01-21 2011-07-28 Interdigital Patent Holdings, Inc. Transfert de session et support de mise en signet pour services de diffusion en flux
CN101860724A (zh) * 2010-05-11 2010-10-13 中兴通讯股份有限公司 交互式网络电视系统中点播节目的方法及装置
WO2011143050A2 (fr) 2010-05-13 2011-11-17 Microsoft Corporation Signets éditables partagés par l'intermédiaire d'un réseau social
CN102893299B (zh) * 2010-05-13 2016-10-19 微软技术许可有限责任公司 通过社交网络共享的可编辑的书签
EP2569750A4 (fr) * 2010-05-13 2013-11-06 Microsoft Corp Signets éditables partagés par l'intermédiaire d'un réseau social
EP2569750A2 (fr) * 2010-05-13 2013-03-20 Microsoft Corporation Signets éditables partagés par l'intermédiaire d'un réseau social
CN102893299A (zh) * 2010-05-13 2013-01-23 微软公司 通过社交网络共享的可编辑的书签
US11550867B1 (en) * 2021-09-09 2023-01-10 Bank Of America Corporation Apparatus and methods for bookmark sharing
US20230102496A1 (en) * 2021-09-09 2023-03-30 Bank Of America Corporation Apparatus and methods for bookmark sharing
US11762936B2 (en) 2021-09-09 2023-09-19 Bank Of America Corporation Apparatus and methods for bookmark sharing

Similar Documents

Publication Publication Date Title
US10313758B2 (en) Scheduling video content from multiple sources for presentation via a streaming video channel
US9955226B2 (en) Multimedia content search system
US10097885B2 (en) Personal content distribution network
US20200221163A9 (en) Method for receiving enhanced service and display apparatus thereof
US8589973B2 (en) Peer to peer media distribution system and method
US8813127B2 (en) Media content retrieval system and personal virtual channel
US7069579B2 (en) On-demand interactive magazine
US20090070324A1 (en) Related information transmission method, related information transmission server, terminal apparatus and related information transmission system
KR101472013B1 (ko) 부가 데이터를 포함하는 음원 스트리밍을 제공하는 서버 및 방법, 그리고 디바이스
WO2008103364A1 (fr) Systèmes et procédés pour envoyer, recevoir et traiter des signets multimedias
US8661013B2 (en) Method and apparatus for generating and providing relevant information related to multimedia content
KR20070043372A (ko) 홈단말에서 실시간 필터링된 방송 비디오 관리 시스템 및그 방법
KR101095296B1 (ko) 메타데이터를 이용한 하이브리드 방송 서비스 시스템
JP2004173120A (ja) 動画像蓄積装置、動画像配信システム
JP6089969B2 (ja) デジタル放送受信装置
KR100529126B1 (ko) Ｐｖｒ의 영상 서비스 방법
JP2004304306A (ja) Ａｖストリーム情報交換装置、ａｖストリーム受信装置およびａｖストリーム格納装置
WO2002062062A1 (fr) Procede et dispositif servant a creer une sequence video fixe au moyen d'un appareil et transmission de cette sequence a un appareil de communication mobile dans le but de l'utiliser

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 08725796

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 08725796

Country of ref document: EP

Kind code of ref document: A1