BR112015023389B1 - METHOD AND SYSTEM FOR IDENTIFYING VIDEO SEGMENTS TO DISPLAY CONTEXTUALLY RELEVANT CONTENT - Google Patents

Abstract

METHOD, AND, SYSTEM. The invention relates to a system and method for identifying specific video segments in a video program, such as certain TV commercials. A means is disclosed for an accelerated identification of the video segment of interest together with a means for communicating said information to a process in a remote television system so that the remote television system can anticipate the future start of said video segment. video of interest and can display related material contextually near the beginning of said video segment.

Description

PRIORITY CLAIM

[001] This application constitutes a continuation-in-part of the U.S. patent application. 12/788,721, entitled “METHODS FOR IDENTIFYING VIDEO SEGMENTS AND DISPLAYING CONTEXTUAL TARGETED CONTENT ON A CONNECTED TELEVISION”, filed May 27, 2010, and issued November 6, 2013 as a U.S. Patent 8,595,781, which application is a non-provisional application claiming priority of the provisional patent application in the U.S. 61/182,334, entitled “SYSTEM FOR PROCESSING INFORMATION CONTENT IN A TELEVIDEO SIGNAL”, filed May 29, 2009 and which is a non-provisional application claiming priority of the provisional patent application in the U.S. 61/290,714, entitled “CONTEXTUAL TARGETING BASED ON DATA RECEIVED FROM A TELEVISION SYSTEM”, filed on December 29, 2009; this application further constitutes a continuation in part of the U.S. patent application 12/788,748, entitled “METHODS FOR DISPLAYING CONTEXTUALLY TARGETED CONTENT ON A CONNECTED TELEVISION”, filed on May 27, 2010; this application further constitutes a continuation in part of the U.S. patent application 14/089,003, entitled “___”, filed on November 25, 2013; this application further constitutes a continuation in part of the U.S. patent application **/***,***, entitled “SYSTEMS AND METHODS FOR ADDRESSING A MEDIA DATABASE USING DISTANCE ASSOCIATIVE HASHING”, filed on March 17, 2014; this application further constitutes a continuation in part of the U.S. patent application **/***.***, titled “SYSTEMS AND METHODS FOR REAL-TIME TELEVISION AD DETECTION USING AN AUTOMATED CONTENT RECOGNITION DATABASE”, filed March 17, 2014; this application further constitutes a continuation in part of the U.S. patent application **/***.***, entitled “SYSTEMS AND METHODS FOR ONSCREEN GRAPHICS DETECTION”, filed March 17, 2014; this application further constitutes a continuation in part of the U.S. patent application **/***.***, titled “SYSTEMS AND METHODS FOR IMPROVING SERVER AND CLIENT PERFORMANCE IN FINGERPRINT ACR SYSTEMS”, filed on March 17, 2014; this application further constitutes a continuation in part of the U.S. patent application **/***.***, entitled “SYSTEMS AND METHODS FOR MULTI-DIFFUSION DIFFERENTIATION”, filed on March 17, 2014; and this application further constitutes a non-provisional patent application in the U.S. 61/791,578, entitled “SYSTEMS AND METHODS FOR IDENTIFYING VIDEO SEGMENTS BEING DISPLAYED ON REMOTELY LOCATED TELEVISIONS”, filed on March 15, 2013. The aforementioned applications are either currently copending or are applications whose copending application is entitled to the benefit of the date of deposit.

FIELD OF THE INVENTION

[002] This invention relates generally to image recognition and, more particularly, to systems and methods for identifying video segments to display contextually relevant content.

BACKGROUND

[003] Since the initial experiments with interactive television in the early 1990s, numerous approaches have been proposed to provide additional contextually relevant information to TV viewers in consideration of what is displayed on their screen. An example of such an application might be special offers associated with a certain nationally broadcast commercial that enhance the national message with local pricing or availability information. Practical considerations regarding the timing of the in-broadcast ad and the local creation and display of the message to be superimposed on a specific TV set resulted in the contextually relevant information being placed at an unacceptable offset from the actual start of the broadcast segment. associated video.

[004] Using the prior art, detecting and identifying a specific video segment can take approximately 7 to 10 seconds using means known to persons skilled in the art. Examples of such known means include identifying multiple video or audio "tags" or detecting a single "watermark" code previously embedded in the video segment of interest. However, by the time such a detection system can respond and additional related information is retrieved, prepared, and displayed on television, the video segment of interest (e.g., a thirty-second commercial message) will have played for about fifteen seconds or even longer with the result that the overlay message loses its contextual relevance.

[005] It is the object of the invention to essentially eliminate all this delay with the use of a system and a method that anticipate the arrival of a specific interest advertisement so that the various processes required confirm the arrival of the segment and generate and display the graphics locally so that these can be started slightly in advance of the actual arrival of the video segment of interest.

SUMMARY

[006] In some embodiments, an exemplary method related to identifying video segments to display contextually relevant content may include obtaining at least one relationship from at least one customer and at least one channel; receive one or more indications of a targeted video segment on a monitored channel; determining one or more clients associated with the monitored channel, based at least in part on at least one relationship obtained from at least one client and at least one channel; and providing one or more triggers related to the targeted video segment to the one or more designated clients.

[007] In some embodiments, obtaining at least one relationship from at least one customer and at least one channel may include receiving at least some information related to a content of a temporary video display store from at least one customer; determine, at least partially based on the at least some information received, at least one channel associated with the content and the at least one customer; and storing the determined channel and an identifier relating to the at least one customer.

[008] In some embodiments, obtaining at least one relationship from at least one customer and at least one channel may include obtaining at least one relationship that includes at least one latency associated with at least one customer and at least one channel. In some embodiments, obtaining at least one relation that includes at least one latency associated with at least one client and at least one channel may include receiving at least some information related to a content of a temporary video display store of at least one client ; determine, at least partially based on the at least some information received, at least (i) a channel associated with the content and the at least one client, and (ii) a latency associated with a network feed and the channel associated with the content; and storing the determined channel, the determined latency, and an identifier related to the at least one client.

[009] In some embodiments, receiving one or more indications of a targeted video segment in a monitored channel may include receiving one or more indications of a targeted video segment in a monitored channel, wherein an audiovisual stream associated with the monitored channel is received at least some quantum of time in advance of the audiovisual stream that is received by a client receiving the channel. In some embodiments, receiving one or more indications of a targeted video segment in a monitored channel may include receiving one or more of at least one frame, at least one sample of a data stream, or at least one still image broadcast per means of at least one channel; determining one or more hash values associated with the received one or more of at least one frame, at least one sample of a data stream, or at least one still image; receive at least one of one or more candidates or one or more suspects from a data store based at least partially on the determined one or more hash values, the data store including at least some hash values associated with one or more targeted video segments; and probabilistically determining a targeted video segment based on receiving at least one of one or more candidates or one or more suspects.

[010] In some embodiments, determining one or more clients associated with the monitored channel, based at least in part on at least one relationship obtained from at least one client and at least one channel, may include retrieving one or more associated client identifiers the correspondence of one or more relations stored between one or more determined channels and one or more customers with one or more identifiers related to the monitored channel. In some embodiments, determining one or more customers associated with the monitored channel, based at least in part on at least one relationship obtained from at least one customer and at least one channel may include determining one or more customers associated with the monitored channel, where the one or more clients tuned to the same channel as the monitored channel, wherein an audio-visual stream associated with the monitored channel is received prior to the audio-visual stream being received by a client receiving a channel corresponding to the monitored channel.

[011] In some embodiments, providing one or more triggers related to the targeted video segment to the specified one or more client systems may include providing one or more triggers related to the targeted video segment to the specified one or more clients, one or more of which or more triggers include at least one time offset related to one or more latencies obtained. In some embodiments, providing one or more triggers related to the targeted video segment to the specified one or more client systems may include providing one or more triggers related to the targeted video segment to the specified one or more client systems, the one or more or more triggers include at least some data relating to at least one of a video sample, a frame, or a still image of the targeted video segment. In some embodiments, providing one or more triggers related to the targeted video segment to the specified one or more client systems may include providing one or more triggers related to the targeted video segment to the specified one or more client systems, the one or more or more triggers include at least one time offset related to one or more obtained latencies and include at least some data related to at least one of a video sample, a frame, or a still image of the targeted video segment.

[012] In some embodiments, another exemplary method related to identifying video segments to display contextually relevant content may include sending at least some related data to a temporary video display store; receive one or more triggers; and provide, based at least partially on at least one of the one or more triggers, at least some contextually related content. In some embodiments, sending at least some data relating to a video display buffer and receiving one or more triggers may include sending at least some data relating to a video display buffer of a client tuned to a particular channel and receiving one or more triggers associated with a network feed of a channel corresponding to the channel to which the client is tuned, wherein the one or more triggers are related to a targeted video segment detected via an ingest module receiving the feed network, being the one or more triggers received by the client subsequent to detection of the targeted video segment at the ingest module and prior to receipt of the targeted video segment at the client.

[013] In some embodiments, providing, based at least partially on at least one of the one or more triggers, at least some contextually related content may include providing at least some received contextually related content prior to sending at least some data related to a temporary video viewfinder store. In some embodiments, providing, based at least partially on at least one of the one or more triggers, at least some contextually related content may include determining a time at which to provide the contextually related content based at least partially on at least one of at least one of the triggers. the one or more triggers; and provide the related content contextually at the given time. In some embodiments, providing, based at least partially on at least one of the one or more triggers, at least some contextually related content may include identifying a target frame, the target frame including at least one of a video sample, a frame or still image of the targeted video segment, the target frame associated with the received trigger; and comparing at least some data associated with a video display temporary store with at least some data associated with the identified target frame and, if the at least some data associated with a video display temporary store matches the at least some data associated with the identified target frame, provide related content contextually.

[014] In some embodiments, identifying a target frame, where the target frame includes at least one of a video sample, a frame or a still image of the targeted video segment, the target frame associated with the received trigger may include identifying a target frame, wherein the target frame includes at least one of a video sample, frame or still image from the targeted video segment and at least one of those received substantially contemporaneously with the trigger or received prior to sending at least some data related to a temporary video display buffer. In some embodiments, the method may further include interactively comparing at least some data associated with a video display buffer to at least some data associated with the identified target frame for at least a period of time and, if the at least some data associated with a video display buffer does not match at least some data associated with the identified targeted frame within the at least one time period, provide contextually related content.

[015] In some embodiments, an exemplary system related to identifying video segments to display contextually relevant content may include one or more clients associated with one or more temporary video display stores configured to at least send at least some data related to the one or more video display buffers and receive one or more triggers associated with contextually related content; and a central media center, the central media center including at least: a media data store configured to at least hold at least some data having the ability to identify video segments; an ingest module in communication with the media data store, the ingest module being configured to at least receive broadcasts and signal the media data store to maintain at least some identifying data relating to at least some of the received broadcasts; a context data store, the context data store being configured to at least hold at least some data relating to one or more targeted video segments; a video segment detection module in communication with the context data store and the ingest module, the video segment detection module is configured to detect one or more targeted video segments within one or more received broadcasts; a client tracking module in communication with the media data store and the one or more clients, the client tracking module configured to at least receive at least some data associated with one or more temporary video display stores from of the one or more clients and determine, based at least in part on at least some data received and the media data storage, one or more program sources associated with the one or more clients; and a client trigger module in communication with the client tracking module and the video segment detector, the client trigger module configured to at least send one or more triggers associated with content related contextually to at least one of the one or more clients based at least partially on one or more indications received from the video segment detection module relating to one or more targeted video segments and based at least partially on one or more indications received from the segment tracking module. client related to one or more specified program sources associated with one or more clients.

[016] In some embodiments of the exemplary system, the one or more clients and the central media center may include one or more clients tuned to a particular channel and a central media center that includes at least one ingest module that receives a network feed of the particular channel to which the one or more clients are tuned, wherein the ingest module receives an audio-visual stream from the network feed before the corresponding audio-visual stream that is received by the one or more clients tuned to the particular channel that corresponds to the mains power, with the time difference being a latency period, where a duration associated with sending one or more triggers from a trigger module to a central media center client and receiving the one or more triggers by one or more clients is less than the latency period.

[017] In addition to the foregoing, various other embodiments of methods, systems, and/or program products are set forth and described in the teachings as the text (e.g., claims, drawings, and/or the detailed description) and/or drawings of the present disclosure .

[018] The foregoing is a summary and therefore contains, by necessity, simplifications, generalizations and omissions of details; accordingly, those skilled in the art will appreciate that the summary is illustrative only and is NOT intended to be limiting in any way. Other aspects, embodiments, features and advantages of the device and/or methods and/or other matter described herein will become apparent from the teachings set forth herein.

BRIEF DESCRIPTION OF THE DRAWINGS

[019] Certain embodiments of the present invention are described in detail below with reference to the following drawings:

[020] Figure 1 is a block diagram of the system for detecting specific commercials or any other video segment in a broadcast streaming transmission at a time prior to its reception by a home TV, and deploying synchronous creation and display in screen of additional graphical information associated with the segment.

[021] Figure 2 is a flowchart of the discrete steps of the method implemented by the system.

[022] Figure 3 is a high-level diagram of national television distribution through regional and local distribution points, then to the headend of a cable TV system, and finally to the viewer's home.

[023] Figure 4 is a block diagram of an exemplary media processing center and an exemplary client, in accordance with an alternative embodiment of the invention.

[024] Figure 5 illustrates an operational flow that represents exemplary operations related to identifying video segments to display contextually relevant content.

[025] Figure 6 illustrates an alternative modality of the operational flow of Figure 5.

[026] Figure 7 illustrates an alternative modality of the operational flow of Figure 5.

[027] Figure 8 illustrates an alternative modality of the operational flow of Figure 5.

[028] Figure 9 illustrates an alternative modality of the operational flow of Figure 5.

[029] Figure 10 illustrates an alternative modality of the operational flow of Figure 5.

[030] Figure 11 illustrates a different operational flow that represents exemplary operations related to identifying video segments to display contextually relevant content.

[031] Figure 12 illustrates an alternative modality of the operational flow of Figure 11.

[032] Figure 13 illustrates an alternative modality of the operational flow of Figure 11.

[033] Figure 14 illustrates an alternative modality of the operational flow of Figure 11.

[034] Figure 15 illustrates a portion of an example system related to identifying video segments to display contextually relevant content.

[035] Figure 16 illustrates a portion of an example system related to identifying video segments to display contextually relevant content.

DETAILED DESCRIPTION

[036] Figure 1 describes a system to provide accelerated detection of a video segment of interest and also to detect a sequence of events and cause a triggering event to be sent to a software application of the invention residing in a television receiver connected causes said application to display contextually targeted information for at least the duration of the video segment of interest.

[037] Due to its location in the distribution network, the central server system 101 of the invention receives television programming from a regional processing center before it is received by the home television system. This time difference allows the system to prepare appropriate responses prior to the arrival and display of a particular video segment, such as a television commercial or the like, so that contextually targeted information can be displayed with an unremarkable delay upon the appearance of said video segment on television display in a household 107.

[038] It is well known to the person skilled in the art that there are significant latencies inherent in the national television distribution system as signals are generally processed through regional and local television distribution centers. The primary means of television program distribution is via fiber optic networks which themselves impose only slight delays. However, the many means of television distribution include statistical multiplexing, rate matching and other processing known to the skilled person. Each of these processes entails delays that accumulate and count towards a total delay of about three to five seconds from the regional center to the viewer's home. When satellite distribution is used on segments of the distribution system instead of fiber optic cable, a few additional seconds are added to the distribution chain for the hop from Earth to the geostationary satellite and back, plus signal processing .

[039] An element of the system and method of this invention as described herein must obtain access to the program video stream as far upstream of the household as possible. This method allows the system described in this document to simulate an apparent ability to actually "look ahead" and identify video segments significantly in advance of their actual receipt by the TV set in the viewer's home that is located at the end of a long distribution chain.

[040] Typically, television programming will be received at the media processing center medium 100 in approximately 3 to 5 seconds before the same programming passes through the rest of the distribution network on its way to the viewer's home. For satellite television consumers, a few additional seconds are added due to sending and returning to a geosynchronous satellite (in addition to any previous satellite links in the distribution system). The system 101 processes said television signals, which thereby creates a database of IDs 102 to be used as a reference database for the purpose of matching unknown video or audio programming 103 from a television monitor 107 at a remote location such as a private home of a user of said system.

[041] The Low Latency Video Segment Detector 105 uses a database for television video segments 104 that must display content contextually related to the trigger on said remote television medium. Said database is of a reduced size with respect to the primary television programming database of the invention 102. This allows accelerated detection of video segments stored in the video segment database 104 when the given video segment is present on a monitored television channel displayed by the low latency television receiver system 101. The video segment match output 105 is a data trigger for the event manager 106 to inform 106 of the presence of a particular video segment of interest. Event handler 106 is also provided with TV match information that indicates currently displayed television programming. The event manager 106 is further provided with the detected result of the currently displayed channel from the home television system 107. The output of 106 is an event trigger data message to the connected TV 107 which will cause 107 to recover, format and be ready to display the contextually targeted information appropriately. Additionally, a clock time for displaying said information is also provided as part of the data message. The television system 107 then displays said information in the time-specified instruction received from the event manager 106.

[042] This invention specifies a system for ingesting and processing television content prior to its distribution and arrival at the home through the current invention and also specifies a second television video segment matching system that provides low-latency detection that monitors one or more television channels located in a processing center 100 of the invention.

[043] The television video segment detection system 105 is programmed to detect the presence of a number of specific television advertisements and is optimized for high-speed, low-delay (latency) detection. The invention includes an event handler 106 that processes event scripts for the purpose of detecting a sequence of events and then issuing an event trigger to a remote software application of the invention 108. Said scripts contain event sequences that must be satisfied before triggering a local event by the television application of the invention 108.

[044] For example, a script can specify that the television be tuned to a specific channel as detected by the TV matching system 103 from media IDs sent over a data network from the application of the invention 109 from a connected television 107. Event manager 106 will additionally detect that a certain confidence interval has been reached, e.g. television 107 has been tuned continuously to a desired channel for more than five seconds and that the television program as specified by the event script is 6 minutes and 30 seconds from the start of the program and finally the video segment detection system 105 must have detected the presence of the video segment of interest. When all the scripting rules are satisfied, the event handler 106 will send an event trigger to the television application of the invention 108 which will cause said application to display an overlay window containing addition information in synchrony with the segment of incoming video.

[045] Figure 4 depicts an alternative embodiment of the invention, which seeks a video matching function that is enabled by the event manager 106 of the system described above. If television 107 is tuned to a television channel that has upcoming events prepared by the system, television contextual event manager 108 is provided with a video frame of a nearby video segment within a few seconds of its display based on the time code from the elapsed time since the beginning of the program currently being shown on said television. Said video frame is retrieved from the central media processing center 100 from the video segment database 104. Said reference video frame is passed to a TV frame store 403 and stores in said frame in the memory. Next, a TV frame comparison process 402 of examining each video frame from the television video display buffer 107 through the TV frame grabber 401 is initiated. The TV frame comparison 402 compares video frames until a reference is found. Upon matching the frame stored at 403 with live video from TV frame grabber 401, application 108 is triggered to display contextually targeted information in a similar manner to the means described above.

[046] Various means of matching video segments in TV frame comparison 402 can be applied, including comparing relative values of pixel packet differences where an array of perhaps ten uniformly distributed pixel packets is compared with the reference video frame. If the average difference between all packet comparisons is within a predetermined threshold, then a match is assumed.

[047] Another means of matching within the television systems internal processor 107 may involve using the perception hashing algorithm to compare the reference frame 403 to the unknown video frames coming from the TV frame grabber 401. Perception Hash is a software medium well known to the person skilled in the art that is excellent at matching single frame images. Perception hashing algorithms describe a class of comparable hashing functions. Features in the image are used to generate a distinct but not unique ID and these IDs are comparable.

[048] Perception hashes are a different concept compared to cryptographic hash functions like MD5 and SHA1. With cryptographic hashes, the hash values are random and cannot be compared. In contrast, perception hashes can be compared, which provides some measure of similarity between the two datasets of hash values.

[049] Each perception hash algorithm has the same basic properties: images can be scaled up or down, have different aspect ratios, and even minor color differences that include changes in contrast and brightness will still match. to similar images. As the perceptual hashing algorithms are in the public domain and well known to the person skilled in the art, the algorithm will not be described in the present context.

[050] In any case of employing video sequence matching performed by the processor of the connected television system, the system of the invention gains the advantage of displaying contextually targeted information within the first few frames of a video segment of interest, the which thus keeps the display latency of contextually relevant material to an acceptable brief level.

[051] As a precaution, if the single square match by the television invention application 107 fails to find a match for whatever reason, the process may be returned by the first method using the Event Manager 106 to send a trigger of event over the network to the connected television and from there trigger the event (contextually targeted display or other events) as described above, the only penalty to the process is the delay of 2 or 3 seconds.

[052] A flowchart of the system and method described above can be seen in Figure 2 and the sources of the numerous latencies that accumulate as video programming is distributed by various known means of television distribution is diagrammed in Figure 3.

[053] Figure 5 illustrates an operational flow 500 representing exemplary operations related to identifying video segments to display contextually relevant content. In Figure 5 and the following figures which include various examples of operational flows, discussion and explanation can be provided in relation to the examples described above in Figures 1 to 4, and/or in relation to other examples and contexts. However, it should be understood that the operational flows can be executed in a variety of other environments and contexts, and/or in modified versions of Figures 1 to 4. Also, although the various operational flows are presented in the sequence(s) illustrated, it should be understood that the various operations may be performed in orders other than those illustrated, or may be performed simultaneously.

[054] After an initial operation, operational flow 500 moves to operation 502. Operation 502 depicts getting at least one relationship from at least one customer and at least one channel. For example, as shown in and/or described in relation to Figures 1 to 4, a channel to which a customer television is tuned can be determined by means of a determination video segment viewable by a customer television and querying a media database to determine a channel on which this video segment is broadcast. The operations by which the video segment is determined may include operations described in a primary application, U.S. Patent Application 12/788,721 (now U.S. Patent 8,595,781), “METHODS FOR IDENTIFYING VIDEO SEGMENTS AND DISPLAYING CONTEXTUAL TARGETED CONTENT ON A CONNECTED TELEVISION” (“the '781 patent”) and/or in the related patent application in the U.S. - “SYSTEMS AND METHODS FOR ADDRESSING A MEDIA DATABASE USING DISTANCE ASSOCIATIVE HASHING” filed concurrently with this application (“the related application”).

[055] Then, operation 504 depicts receiving one or more indications of a targeted video segment on a monitored channel. For example, as shown and/or described in relation to Figures 1 to 4, an ingest system that monitors one or more channels via a network feed, for example, can detect a video segment of interest that is shown in one of the channels. A video segment of interest could be, for example, a particular commercial for which an entity affiliated with the commercial has contracted with an entity that operates a central media center to supplement the commercial in real-time or near real-time (i.e., within the first three to five seconds after the commercial starts playing on the customer's TV, ideally within the first few frames of the commercial) with additional content for all customer televisions that the central media center has detected tuned to any channel by the which the particular commercial is broadcast. Operations by which the video segment of interest may be detected may include operations described in the '781 patent and/or the related application, whereby operations may additionally be applied to a context database containing at least hash information related to the video segments of interest (the commercials mentioned above, for example).

[056] Then, operation 506 depicts determining one or more customers associated with the monitored channel, based at least in part on at least one relationship obtained from at least one customer and at least one channel. For example, as shown and/or described in relation to Figures 1 to 4, once a video segment of interest is detected as being broadcast on a particular channel as described above, a determination of all client televisions tuned to that particular channel, using data that may have been determined in operation 502, is evaluated.

[057] Then, operation 508 depicts providing one or more triggers related to the targeted video segment to the one or more designated clients. For example, as shown and/or described in relation to Figures 1 to 4, using the customer list of all customer televisions tuned to that particular channel, one or more data packets are sent to those particular customer televisions, the one or more data packets containing one or more instructions which, when received and/or executed by the client televisions, would cause the client televisions, for example, to display supplemental content relating to the video segment of interest. In some embodiments, the client television may, once the video segment of interest (commercial) is received and begins to form on the television, overlay the commercial on the video display for contextually relevant information. For example, the overlay information may include commercial-related supplemental information, such as a special offer associated with a nationally broadcast commercial, where the special offer includes established pricing for a geographic division within which the TV is included - a first selling price of the advertised item shown by televisions in one state, for example, and a second selling price of the item that is different from the first selling price shown by televisions in a different state. Such geographic divisions can be performed, for example, by means of an IP address of the client television which is compared with an ARIN database. Other types and/or methods of supplemental content for determining a subset of client televisions to receive particular supplemental content are possible. Also, providing supplementary content can mean, for example, instantiating a window on the television display that overlays a program that forms on that display. In different embodiments, providing the supplemental content may mean rendering the content on a different display, or delivering the content to a computing device associated with a client television either wirelessly or over wire(s), or sending the content to a printer associated with a customer television, or burn a CD-ROM or DVD-ROM using a recorder associated with a customer television, or record the contents on a USB key, hard disk or other storage medium associated with a client television, or any other means of displaying supplemental content on one or more devices within the client television range.

[058] Figure 6 illustrates alternative embodiments of the exemplary operational flow 500 of Figure 5. Figure 6 illustrates an exemplary embodiment in which operation 502 may include at least one additional operation. Additional operations may include the 602 operation, the 604 operation, and/or the 606 operation.

[059] Operation 602 illustrates receiving at least some information related to a content of a video display temporary store of at least one client. For example, as shown in and/or described in relation to Figures 1 to 4, the client television may periodically send "indication data", or indications that allow indication data to be determined. For example, a customer television may send one or more indications relating to content from a client television video display buffer ten times per second, the one or more indications allowing a determination of a particular program that is displayed on that television, using, for example, the operations disclosed in the '781 patent and/or the related application, the resulting cue data and/or hash values further permitting a determination of a video segment that the television of client displays.

[060] Additionally, operation 604 illustrates determining, at least partially based on the at least some information received, at least one channel associated with the content and the at least one customer. For example, as shown in and/or described in relation to Figures 1 to 4, once a video segment that is formed by a customer television is determined, perhaps at operation 602, a central media center can determine which channel a client television displays by comparing cue data with a media data store (perhaps through operations disclosed in the '781 patent and/or the related application) to determine the name of a video segment, and then examine stored metadata associated with the given video segment, which metadata includes a particular channel derived from the program guides associated with channels monitored by the ingest system (also perhaps through operations disclosed in the '781 patent and/or the related application ) to determine the channel to which the client televisions are tuned.

[061] Additionally, operation 606 illustrates storing the determined channel and an identifier related to at least one customer. For example, as shown in and/or described in relation to Figures 1 to 4, a real-time or near real-time listing of all customer televisions and all channels to which those customer televisions are tuned may be maintained at a data store, using perhaps outputs from operation 602 and/or operation 604.

[062] Figure 7 illustrates alternative embodiments of the exemplary operational flow 500 of Figure 5. Figure 7 illustrates an exemplary embodiment in which operation 502 may include at least one additional operation. Additional operations may include the 702 operation, the 704 operation, the 706 operation, and/or the 708 operation.

[063] Operation 702 illustrates obtaining at least one relation that includes at least one latency associated with at least one client and at least one channel. For example, as shown in and/or described in relation to Figures 1 to 4, a central media center can be established "upstream" in the distribution chain from the customer televisions, whereby the central media center can, for example, downlink programming directly from a satellite to which a networked national distribution center uplinks that programming. In contrast, by the time programming is received by a client system (television with OTA and/or STB antenna, satellite receiver, cable card, etc.), programming may have been downlinked by, for example , a regional affiliate, which rebroadcasts the same on-air, or the regional affiliate's broadcast may, in turn, be rebroadcast by a cable and/or satellite television provider. As a result, given the upstream position of the central media center relative to client systems, the central media center "knows" of broadcast content several seconds before a client system, whereas the central media center does not is subject to intervention transmissions. The time difference between a central media center getting an indication that a particular video segment was broadcast by a national broadcaster and a client system getting an indication that the particular video segment was broadcast can be several seconds. These several seconds can be referred to as the latency. Client televisions can signal content-related data from their video display buffers ten times per second to the central media server over the Internet, for example. This data can be used at the central media center to determine an exact or near-exact time difference between a video segment that is received by the central media server and the same video segment that is received by the client system (perhaps through of operations disclosed in the '781 patent and/or related application). The exact or nearly exact time difference can be related to latency between the central media center and the client systems.

[064] Additionally, operation 704 illustrates receiving at least some information related to a content of a video display temporary store of at least one client; for example, as shown in and/or described in relation to Figures 1 to 4, the client television may periodically send "indication data", or indications that allow indication data to be determined. For example, a customer television may send one or more indications relating to content from a client television video display buffer ten times per second, the one or more indications allowing a determination of a particular program that is displayed on that television, using, for example, the operations disclosed in the '781 patent and/or the related application, the resulting cue data and/or hash values further permitting a determination of a video segment that the television of client displays.

[065] Additionally, operation 706 illustrates determining, at least partially based on the at least some information received, at least (i) a channel associated with the content and the at least one client, and (ii) a latency associated with a feed and the channel associated with the content. For example, as shown in and/or described in relation to Figures 1 to 4, once a video segment that is formed by a customer television is determined, perhaps at operation 704, a central media center can determine which channel a client television displays by comparing cue data with a media data store (perhaps through operations disclosed in the '781 patent and/or the related application) to determine the name of a video segment, and then examine stored metadata associated with the given video segment, which metadata includes a particular channel derived from the program guides associated with channels monitored by the ingest system (also perhaps through operations disclosed in the '781 patent and/or the related application ) to determine the channel to which the client televisions are tuned. Additionally, the central media center can also determine the time difference, either exactly or nearly exactly, between the ingest system receiving a video segment and the client system receiving the video segment. The time difference is useful when providing triggers, where the trigger data can include an indication of a time offset after the trigger is received at which time the supplemental content should be provided, the time offset that counts towards the difference of time between the ingest system receiving a video segment and the client system receiving the video segment (“latency”), and also a quantum of time that will cause the trigger to be received at the client system after it is sent (the last quantum of time, in other contexts, which is called “network latency”, where the network is the Internet and latency is the time, usually measured in milliseconds, for a transmission from an Internet node, for example, a central media center to be received at another Internet node, for example a customer television).

[066] Additionally, operation 708 illustrates storing the determined channel, the determined latency, and an identifier related to at least one client. For example, as shown in and/or described in relation to Figures 1 to 4, a real-time or near real-time listing of all customer televisions and all channels to which those customer televisions are tuned may be maintained at a data store, using perhaps outputs from operation 602 and/or operation 604.

[067] Figure 8 illustrates alternative embodiments of the exemplary operational flow 500 of Figure 5. Figure 8 illustrates an exemplary embodiment in which operation 504 may include at least one additional operation. Additional operations may include the 802 operation, the 804 operation, the 806 operation, and/or the 808 operation.

[068] Operation 802 illustrates receiving one or more of at least one frame, at least one sample of a data stream, or at least one still picture transmission via at least one channel. For example, as shown in and/or described in relation to Figures 1 to 4, an ingest system that monitors one or more channels, perhaps via downlink transmissions from a satellite to which the national network broadcast centers. the uplink programming can receive such transmissions that received downlink as a sequence of video frames, the video frames are alternatively referred to as sequential still pictures, 30 still pictures per second in sequence comprising NTSC, PAL , SECAM, HDTV, or other current and/or future standards for video data. In different embodiments, different frame rates are possible, for example 24 frames per second, 60 frames per second, or other rates at which a data stream is sampled. The ingest system, in short, examines each frame received from one or more channels, which can be 10, 200, 500, 1000, or more channels, all of which receive downlink simultaneously at the central media center.

[069] Additionally, operation 804 illustrates determining one or more hash values associated with the one or more received from at least one frame, at least one sample of a data stream, or at least one still picture. For example, as shown and/or described in relation to Figures 1 to 4, a hash value associated with one or more of at least one frame, at least one sample of a data stream, or at least one still image is determined. , perhaps through operations disclosed in the '781 patent and/or the related application, the hash value that has the ability to identify a particular video segment, perhaps uniquely or perhaps in conjunction with a plurality of associated hash values to other frames of a particular downlinked feed.

[070] Additionally, operation 806 illustrates receiving at least one of one or more candidates or one or more suspects from a data store based at least partially on the determined one or more hash values, the data store including at least some hash values associated with the one or more targeted video segments. For example, as shown and/or described in relation to Figures 1 through 4, a hash value can be used to retrieve one or more suspects or candidates from a data store, perhaps through operations disclosed in the '781 patent. and/or the related order. Differently, in the present operational flow, suspects and/or candidates can be retrieved from a specific data store for targeted video segments. As described above, a targeted video segment and/or a video segment of interest may be, for example, a particular commercial for which an entity affiliated with the commercial has contracted with an entity that operates a central media center to supplement the commercial. in real-time or near real-time (that is, within the first three to five seconds after the commercial starts playing on the customer's TV, ideally within the first few frames of the commercial) with additional content for all customer televisions that the central media center has detected tuned to whatever channel the particular commercial is broadcast on. Data associated with such commercials may be stored in a data store, perhaps, for example, a context data store and/or a video segment database.

[071] Additionally, operation 808 illustrates probabilistically determining a targeted video segment based on receiving at least one of one or more candidates or one or more suspects. For example, as shown and/or described in relation to Figures 1 to 4, the one or more candidates or suspects can be placed into bins and/or time bins and values associated with the bin or time bins can be increased and/or diminished, perhaps through tokens, and perhaps through operations disclosed in the '781 patent and/or the related application and applied to a context data store and/or a video segment database to identify a video segment. targeted private video of interest that is broadcast on a monitored channel via feeds received downlink at the central media center.

[072] Figure 9 illustrates alternative embodiments of the exemplary operational flow 500 of Figure 5. Figure 9 illustrates an exemplary embodiment in which the operation 504 may include at least one additional operation, which includes the operation 902. Figure 9 also illustrates a exemplary embodiment in which operation 506 may include at least one additional operation. Additional operations may include operation 904 and/or operation 906.

[073] Operation 902 illustrates receiving one or more indications of a targeted video segment on a monitored channel, wherein an audiovisual stream associated with the monitored channel is received at least some time quantum in advance of the audiovisual stream that is received by a client that receives the channel. For example, as shown and/or described in relation to Figures 1 to 4, due to the latency associated with the central media center being "upstream" from the customer televisions, a segment of video broadcast on one channel may be received by via a downlink of a network feed of that channel at the central media center before the same video segment is received on a customer television tuned to that channel. Latency results in the video segment being received at the central media center some quantum of time before the video segment is received at a client television. In that time quantum, using systems and/or methods described elsewhere in this document and/or in an application or patent, whether primary or related, it is possible for the central media center to: determine a video segment of interest that is broadcast on any of the channels it monitors through downlink formation on the network uplink; determine all TVs tuned to the channel using cue data that were transmitted earlier and/or substantially contemporaneously (within the last five, ten, fifteen or more seconds) to the central media center, perhaps ten times per second per TV ; and output one or more indications associated with the trigger to all TVs tuned to that channel. In some embodiments, a trigger may include at least a portion of supplemental content. In different embodiments, the supplemental content may have been previously received by the client or system television (perhaps several seconds earlier, perhaps the night before, perhaps in production of the client or system television, or perhaps sometime in between).

[074] Additionally, operation 904 illustrates retrieving one or more customer identifiers associated with the corresponding one or more relationships stored between one or more determined channels and one or more customers with one or more identifiers related to the monitored channel. For example, as shown and/or described in relation to Figures 1 through 4, once the video segment of interest is determined to be broadcast on a particular channel that is monitored, a list of all customer televisions tuned to that particular channel is retrieved and/or provided. It is important to note that the customer televisions tuned to that particular channel do not receive the channel via the same network feed as the central media center, but instead receive the channel via the network feed that has been re-cast. transmitted downstream from the satellite from which the central media center forms a downlink with the feed.

[075] Additionally, operation 906 illustrates determining one or more clients associated with the monitored channel, the one or more clients tuned to the same channel as the monitored channel, wherein an audiovisual stream associated with the monitored channel is received prior to the audiovisual stream is received by a client that receives a channel corresponding to the monitored channel. For example, as shown and/or described in relation to Figures 1 to 4, since the customer televisions tuned to that particular channel do not receive the channel through the same network feed as the central media center, but instead instead, receive the channel via this network feed which has been re-transmitted downstream from the satellite from which the central media center forms downlink with the feed, a given monitored channel schedule is received at the central media center before the programming is received by the client television via a corresponding channel. In the present case, "channel" refers to a particular programming string and not necessarily to a particular signal transmission or frequency. In fact, the present invention is operable due to the fact that the central media center and the clients receive the programming streams through different frequencies or through different signal transmissions. The central media center can, for example, receive the programming stream (the ESPN “channel”, for example), via a satellite dish tuned to a frequency in the C-band satellite frequency band (3.7 GHz , for example), whereby a customer television system can receive the same programming stream (the ESPN “channel”) by way of receiving it by a cable television distribution network operating at a frequency of North American television of, for example, 409.25MHz. In the context of this application, "channel" refers to the name of a programmer and/or broadcaster and/or affiliate, whereby a frequency refers to the portion of the electromagnetic spectrum over which a broadcast streaming audio-visual transmission comprising a channel in (ESPN, so to speak) be transmitted by a broadcaster and/or received by a client television either wirelessly or over wire(s).

[076] Figure 10 illustrates alternative embodiments of the exemplary operational flow 500 of Figure 5. Figure 10 illustrates an exemplary embodiment in which operation 508 may include at least one additional operation. Additional operations may include operation 1002, operation 1004 and/or operation 1006.

[077] Operation 1002 illustrates providing one or more triggers related to the targeted video segment to the given one or more clients, the one or more triggers including at least one time offset related to one or more latencies obtained. For example, as shown and/or described in relation to Figures 1 to 4, using a customer list of all customer televisions tuned to that particular channel, one or more data packets are sent to those particular customer televisions. , the one or more data packets containing one or more instructions that, when received and/or executed by the client televisions, would cause the client televisions, for example, to display supplemental content related to the video segment of interest. In some embodiments, the client television may, once the video segment of interest (commercial) is received and begins to form on the television, overlay the commercial on the video display for contextually relevant information. For example, the overlay information may include commercial-related supplemental information, such as a special offer associated with a nationally broadcast commercial, where the special offer includes established pricing for a geographic division within which the TV is included - a first selling price of the advertised item shown by televisions in one state, for example, and a second selling price of the item that is different from the first selling price shown by televisions in a different state. Such geographic divisions can be performed, for example, by means of an IP address of the client television which is compared with an ARIN database. Other types and/or methods of supplemental content for determining a subset of client televisions to receive particular supplemental content are possible. Particularly, the trigger data may additionally include and/or be associated with a particular time quantum and/or a particular time in which the client must provide the supplemental content. The particular time quantum and/or time may be established in view of the latency determined to exist between the central media center and the customer TV which is at least in part due to an upstream location of the central media center relative to the customer TV.

[078] Additionally, operation 1004 illustrates providing one or more triggers related to the targeted video segment to the determined one or more client systems, the one or more triggers including at least some data related to at least one of a sample of video, a frame, or a still image of the targeted video segment. For example, as shown and/or described in relation to Figures 1 through 4, an indication is sent to client systems (televisions, set top boxes, computing devices, media players, etc. that have the client application and/or widget present, where the application or widget is in communication with a central media center) of a video segment of interest that is broadcast on the channel that is anticipated to arrive on client systems during or at the end of the latency period. The data may include, for example, a time at which the client system must provide supplemental content, or a time delay after which the trigger is received at the client before supplemental content is provided.

[079] Additionally, operation 1006 illustrates providing one or more triggers related to the targeted video segment to the determined one or more client systems, the one or more triggers including at least one time offset related to one or more latencies obtained and includes at least some data related to at least one of a video sample, a frame, or a still image of the targeted video segment. For example, as shown and/or described in relation to Figures 1 to 4, a trigger can be associated with data relating to a particular frame of the video segment of interest. In some embodiments, the trigger may include pixel data associated with a frame, a subset of the pixel data associated with the frame, or a value encoded as a hash value associated with the frame. Data relating to a particular frame of the video segment of interest can be, for example, related to the immediately first frame (or a frame in close proximity to the first frame - within the first ten frames, for example) of the video segment of interest. The data can be used by the client system to detect the actual start of the video segment of interest locally. For example, the client system can use the data and a matching algorithm, which checks the received data about the video segment of interest against frames that are received subsequently in the data stream. In an alternative embodiment, data relating to the particular frame of the video segment of interest may have been previously transmitted to the application and/or widget with an identifier, and the subsequent trigger, rather than including pixel data or hash values, may merely include the identifier.

[080] Figure 11 illustrates an operational flow 1100 representing exemplary operations related to identifying video segments to display contextually relevant content. In Figure 11 and the following figures which include various examples of operational flows, discussion and explanation can be provided in relation to the examples described above in Figures 1 to 4, and/or in relation to other examples and contexts. However, it should be understood that the operational flows can be executed in a variety of other environments and contexts, and/or in modified versions of Figures 1 to 4. Also, although the various operational flows are presented in the sequence(s) illustrated, it should be understood that the various operations may be performed in orders other than those illustrated, or may be performed simultaneously.

[081] After an initial operation, operational flow 1100 moves to operation 1102. Operation 1102 depicts sending at least some data related to a video display temporary store. For example, as shown and/or described in relation to Figures 1 to 4, a client system may send one or more indications relating to content on the video display temporary store, the content representing a frame, a still image or a sample of a video data stream that is displayed or is to be displayed on the television display. The one or more indications may refer to pixel data from the frame, still picture, or sample of a video data stream, or a subset of the pixel data, or be an encoded value relating to the frame, still picture, or sample. of a stream of video data, as a hash value determined, perhaps, through operations disclosed in the '781 patent and/or the related application. Data may be sent periodically, perhaps 10 times per second (every third frame). Data can be sent non-periodically, randomly, according to bandwidth availability, or at some other regular or irregular interval. Data does not have to be sent as much data is stored by an ingest system on a central media server.

[082] Then, operation 1104 depicts receiving one or more triggers. For example, as shown and/or described in relation to Figures 1 to 4, a client system will receive one or more indications from a central media center of a video segment of interest that is to be received by the client system. , the one or more triggers that include at least an indication of the supplemental to provide and a time at which the supplemental content is to be provided. Supplementary content may have been received before the television was turned on by a user, during a period of low usage, for example, where the trigger could indicate what supplemental content was previously received to provide. Or, the trigger can include at least a portion of supplemental content.

[083] Then, operation 1106 depicts providing, based at least partially on at least one of one or more triggers, at least some portion of the contextually related content. For example, as shown and/or described in relation to Figures 1 to 4, the client system may provide content related to the video segment of interest, which may include local pricing related to a geographic division in which the client is located that is intended to supplement an advertisement of interest that is to be received and formed by the client television (ie, the content refers to the context of the advertisement). Supplemental content may be provided in any manner that includes those described in connection with operation 508 and includes, but is not limited to, instantiating a window overlay on the video, where the window includes a message related to the underlying ad or other content contextually related.

[084] Figure 12 illustrates alternative embodiments of the exemplary operational flow 1100 of Figure 11. Figure 12 illustrates an exemplary embodiment in which the operation 1102 may include at least one additional operation, which includes the operation 1202. Figure 12 also illustrates a Exemplary embodiment in which operation 1106 may include at least one additional operation, which includes operation 1204.

[085] Operation 1202 illustrates sending at least some data related to a video display buffer of a client tuned to a particular channel and receiving one or more triggers associated with a network feed of a channel corresponding to the channel to which the client is tuned in, where the one or more triggers are related to a targeted video segment detected by an ingest module receiving the network feed, where the one or more triggers received by the client subsequent to detection of the segment targeted video on the ingest module and before receiving the targeted video segment on the client. For example, as shown and/or described in relation to Figures 1 to 4,

[086] Additionally, operation 1204 illustrates providing at least a portion of contextually related content to a video display buffer. For example, as shown and/or described in relation to Figures 1 to 4, a latency period may exist between the time that a particular frame of an audio-visual data stream on a channel is received in an ingest system of a central media, and the time that the same particular frame of the audio-visual data stream on the channel is received by a client television. Between these two points in time, where a particular video segment of is broadcast on a channel and can be detected at the central media center, a listing of televisions tuned to that channel can be generated, and triggers related to the video segment of of interest are sent to all televisions in the list, with triggers being received prior to the time the video segment of interest is received at the client systems. This may be, for example, due to the fact that it takes several seconds for the video segment of interest to propagate through various means of distribution and/or retransmission between the network uplink and the client television, where the system client and central media center can communicate over the Internet, which provides end-to-end data ratios faster than distribution and/or relay media. If the trigger arrives after the start of the video segment on the client system (perhaps due to Internet latency), the instructions associated with the trigger can still be executed by the client system as long as the video segment of interest is still formed by the system. of customer.

[087] Figure 13 illustrates alternative embodiments of the exemplary operational flow 1100 of Figure 11. Figure 13 illustrates an exemplary embodiment in which operation 1106 may include at least one additional operation. Additional operations may include operation 1302, operation 1304, operation 1306 and/or operation 1308. Figure 13 further illustrates an exemplary embodiment in which operation 1306 may include at least one additional operation, which includes operation 1310.

[088] Operation 1302 illustrates determining a time at which to provide at least partially contextually related content in the at least one or more triggers. For example, as shown and/or described in relation to Figures 1 to 4, the trigger may include an indication of a time at which supplemental content is to be delivered based on a detected time difference between a central media center receiving a frame and the client system subsequently receives that frame. In some embodiments, the trigger may include a literal time at which to provide the supplemental content (for example 8 hours, 30 seconds, and 100 milliseconds Greenwich Mean Time on March 17, 2014). In different embodiments, the trigger can include a time delta (for example, which instructs the client system to provide the supplemental content exactly 2500 milliseconds after the trigger is received, where 2500 milliseconds is a unique value for the client system that set by the central media center to consider broadcast path latency based on a determination of how far the client system lags behind the central media center in receiving content from a particular channel).

[089] Additionally, operation 1304 illustrates providing contextually related content at the given time. For example, as shown and/or described in relation to Figures 1 to 4, based on the data coming from operation 1302, supplemental and/or contextually related content is provided at the given time. In some embodiments, the designated time is set to be synchronized with the start of a commercial of interest (e.g. simultaneously with the first frame, where substantially simultaneously with the first frame means within the first ten frames, first 30 frames, first five seconds, etc.). In different embodiments, the determined time is set to be synchronized with a particular point within the commercial of interest about halfway through, for example, or after a particular scene has been formed, or towards the end of the commercial.

[090] Additionally, operation 1306 illustrates identifying a targeted frame, wherein the targeted frame includes at least one of a video sample, a frame, or a still image of the targeted video segment, wherein the targeted frame is associated with the received trigger. For example, as shown and/or described in relation to Figures 1 to 4, a trigger can include data associated with the first frame of the video segment of interest as the client system. In different embodiments, a frame of the video segment of interest other than the first frame can be selected as a target frame. For example, the target frame can be selected as the first frame of the video segment that allows positive identification in the client system. The target frame can be, for example, the first non-black frame in the video segment of interest. In other embodiments, the target frame may be selected from the middle of the video segment of interest, the target frame selected to coincide with a particular point during the video segment of interest at which supplemental content is to be provided.

[091] Additionally, operation 1308 illustrates comparing at least some data associated with a video display buffer to at least some data associated with the identified targeted frame and, if the at least some data associated with a video display buffer match at least some data associated with the identified targeted frame, provide contextually related content. For example, as shown and/or described in relation to Figures 1 to 4, the client system can compare data associated with each frame subsequently received via the channel with data associated with the first frame of the video segment of interest. If a match is found, it may indicate the start of the video segment of interest on the client system at which point the client system would deliver the supplemental content.

[092] Additionally, operation 1310 illustrates identifying a targeted frame, wherein the targeted frame includes at least one of a video sample, frame, or still image of the targeted video segment and at least one received substantially contemporaneously with the trigger or received before sending at least some data related to a video display buffer. For example, as shown and/or described in relation to Figures 1 to 4, as shown elsewhere in the present document, data associated with the target frame may be received with the trigger (substantially contemporaneous) and may be pixel data taken from the frame, or they can be an identifier that attaches to frame data about all video segments of interest received at the client system before the client system is monitored (the night before, for example, or in production of the client system, or during a client system network upgrade).

[093] Figure 14 illustrates alternative embodiments of the exemplary operational flow 1100 of Figure 11. Figure 14 illustrates an exemplary embodiment in which operations 1306 and 1308 may include at least one additional operation, which includes operation 1402.

[094] Operation 1402 illustrates interactively comparing at least some data associated with a video display buffer to at least some data associated with the identified target frame for at least a period of time and, if the at least some associated data a video display buffer does not match at least some data associated with the identified targeted frame within the at least one time period, provide contextually related content. For example, as shown and/or described in relation to Figures 1 through 4, the two different ways of indicating when to provide the supplemental content (instructing the client system to provide it at a particular time, or instructing the client system to monitor a particular frame and then provide the supplemental content) can guarantee each other. For example, the client system, which has received an indication of a target frame which, when received, would indicate to the client system that it is time to provide the supplemental content, can match each frame in the received data stream for a period of time. If a match is not made within the time period, the client system can provide the supplemental content as fail-safe against matching processes that did not match, or perhaps matching processes that started after the target frame had been received. The time period can be, for example, a delta time associated with the trigger and based at least partially on the detected latency. In different embodiments, one or the other of two different ways of indicating when to provide supplemental content may provide a more reliable means of providing content closer to the start of the video segment of interest. For example, if internet latency is an issue and triggers are received at inconsistent points relative to the start of the video segment of interest, target frame matching can provide a more reliable indication of the start of the segment of interest.

[095] Figure 15 illustrates a portion of an example system related to identifying video segments for displaying contextually relevant content. Figure 15 depicts one or more clients 1500 as shown and/or described with respect to Figures 1 to 4. The clients may be connected to the Internet 1590. In some embodiments, a client is a television 1504. The client television may be a smart television, which includes a 1536 network connection, which can be a wired network connection (an RJ-45 Ethernet connection over Cat 5, for example) or a wireless connection (802.11 Wi-Fi, for example). The smart television may have at least one of a widget or application 1548 that runs concurrently with the smart television forming the received program content, perhaps through operations disclosed in the '781 patent and/or the related application. The widget or application may connect using the network connection to the Internet and be in communication with a central media server 1600 (not depicted in Figure 15 but shown in Figure 16). The smart television may have a video display 1546 (e.g. a 55” LED display), the video display being dockable with a video display buffer 1542, the video display buffer being a memory that contains data relating to one or more frames that are to be displayed through the viewfinder. Data can include pixel data related to one or more frames. The smart television may also have a user interface 1544 for controlling the television and/or for receiving and/or interacting with supplemental content. In some embodiments, the user interface can be formed by means of a window that overlays a portion of the image data from the program received on the display (the top right corner, for example), or by means of a window that overlays the entire until then. In other embodiments, the user interface can also be formed by means of an auxiliary display coupled with the television, or in a device operationally coupled to the smart television (tablet computer, printer, computing device, audio system, second television, etc.).

[096] In some embodiments, when the television 1504 is a smart television, the television will form content associated with a programming source internal to the smart television. For example, the smart television can form content received by an over-the-air antenna 1508, a cable card 1510, or the Internet 1512 (through an application such as Aereo, for example).

[097] In some embodiments, the smart television can form the programming content received from an external system through a smart television input connection, for example an HDMI 1514 input source; via a legacy input source 1516 such as composite video, S-Video or component video; or via a future input source (HDMI 2.0, MHL, etc.). In other embodiments, the smart television can form the programming content through a DVR 1520 that is internal to the smart television, or through another legacy audiovisual player 1522 such as a DVD player or VCR internal to the television, or a future audiovisual 1524 that is internal to television. The smart television may render the content through a computing device 1526, such as a Media Center PC or a PC with a tuner card installed, the computing device connected to the television through HDMI, DVI, VGA, or other proper connection. The smart television can also form the content through a game console 1528 that has a tuner, or an application such as Aereo, etc. Smart television can also form the content received through a Set-top Box (STB) 1534, which can be a DVR, external tuner like a satellite or cable receiver, or it can be an external digital HDTV receiver used to receive and form content. digital on an older television designed to receive analog stations. Smart television can form content received via USB 1530 or Bluetooth 1535, which includes dongles or devices that include tuners or apps like Aereo, or program content stored on a USB drive, for example. In other embodiments, smart television can form the content received through an “other” source 1532.

[098] In different embodiments, the television 1504 may be a television without a network connection. The television may have an external tuner 1550, which may have at least one of a resident widget or application 1552. The external tuner can have a 1554 network connection, which can be wireless 1556 or wired 1558, and has a capability of outputting 1560 to television (HDMI, for example). The external tuner may have a buffer 1562 that is analogous to the video display buffer 1542 in a smart television (eg, being a memory through which pixel data relating to video frames passes). The external tuner can also have a 1564 program source, such as a digital tuner that receives signals from OTA, cable card, satellite receiver, Aereo application, etc. In such embodiments, the external tuner coupled to the television without a network connection is in communication via its network connection with a central media center 1600. In some embodiments, the external tuner may also have the capability of playing tapes of DVDs. or VHS, be a game console, or a Media Center PC, or a PC with a tuner card installed, etc.

[099] In other embodiments, a computing device 1570 forming the content is in communication with the central media center 1600 via network connection 1576. In some embodiments, a computing device has a display 1572, such as a tablet iPad with a Retina display, or a personal computer with an attached VGA LCD monitor. The computing device may have a programming source 1580. The computing device may run an application such as Aureo, or may receive streaming video content relating to a particular channel. For example, the tablet or PC type computer could be used by a female traveler who has a streaming connection to a cable box in the user's home, or to a Slingbox, or whose cable contract for the television in the user's home also allows the user to log in with a computing device to receive cable channels. There may be temporary storage 1578 associated with pixel data that must be formed via display 1572. The computing device may also have a widget or application 1574 in communication with the central media center, which allows supplemental content to be provided in conjunction with programming content received on the computing device, perhaps through a window superimposed on an Aureo window on the tablet computer.

[0100] Figure 16 illustrates a portion of an example system related to identifying video segments for displaying contextually relevant content. Figure 16 depicts a central media center 1600 as shown and/or described with respect to Figures 1 to 4. The central media center may be connected to the Internet 1590, which facilitates communications with clients 1500. There may be a single media center central media, or there may be multiple central media centers. There could be a disaster recovery site, for example, or central media centers could be geographically distributed so that communications with a subset of customers are geographically distributed over the central media centers. Or, central media centers can be divided based on subsets of channels that are monitored. Alternatively, some combination of the above can be employed to provide one or more central media hubs.

[0101] The central media center 1600 has an ingest module 1602, which receives programming from a number of channels, perhaps by forming a downlink of the content via a satellite connection 1616, the downlink being of an uplink provided by a national television distribution center 301 for particular channels, the uplink and downlink being via satellite 1614. Also the downlink formation from the satellite 1614 may be an affiliate network, for example, or a cable system headend, one of which may use a 1618 satellite connection. In different embodiments, programming data from the national television distribution center may be delivered over a fiber connection optical or other suitable non-satellite connection. Distribution 1620 is intended to represent the various intermediate uplinks, downlinks, retransmissions, and local distribution of programming content between the national television distribution center and the television client. An example of the distribution is depicted in Figure 3, and a latency associated with the distribution can typically be measured in seconds (3, 5, 7, 10 seconds or more, for example).

[0102] Data associated with schedule streams received via ingest module 1602 is held in media data 1604. Frames of schedule streams may be hashed, perhaps through operations disclosed in the '781 patent and/or the related request, and metadata associated with the programming streams (program titles coming from a channel guide, and/or an indication of the channel from which the programming was received, for example) can be associated with the hash values so that a particular hash value can be used to identify a program title or a channel on which the program is broadcast. Data associated with programming streams received via the ingest module is also processed by the video segment detection module 1608, which is in communication with the context data store 1606. The context data store, or database video segment data 104 of Figures 1 and 4, may have information relating to video segments of interest (commercials, for example). The video segment detection module examines data associated with frames received through the ingest module to determine whether the frame is associated with a video segment of interest using data associated with the video segments of interest stored in video storage. context data.

[0103] Separately, the client tracking module 1610 receives data from clients 1500 over the Internet. The data may be associated with the contents of the clients' video display buffers, and may be used to determine a channel to which the client is tuned using operations disclosed elsewhere herein. The client tracking module, in communication with media data store 1604, can therefore maintain a listing of all clients that are active and the channels to which the clients are tuned.

[0104] If a video segment of interest is detected by the video segment detection module 1608, the client trigger module 1612 may send one or more trigger indications to the clients 1500, perhaps using a client listing tuned to a particular channel showing the video segment of interest, the customer listing being received from customer tracking module 1610 or compiled using data associated with customer tracking module 1610. Triggers are sent through Internet connection 1590. Communication between the central media center and the clients can be sent and received in a millisecond period (the Internet latency), whereby a time difference between a segment of video of interest that is received at the central media center 1600 and that the video segment that is received at the clients can be measured in seconds (the broadcast latency) due at least partially to between the distribution, redistribution, transmission, retransmission, uplinking, and downlinking that occurs between the time that a national television distribution center 301 uplinks a program and the time that the program that has received upstream is eventually received by a client.

[0105] The system and methods, flowcharts and block structure diagrams described in this specification can be implemented in computer processing systems that include program code comprising program instructions that are executable by a computer processing system computer. Other deployments can also be used. Additionally, the flowcharts and block structure diagrams described herein describe particular methods and/or corresponding acts in favor of the corresponding steps and functions in support of the disclosed structural means, may also be used to implement corresponding and equivalent software structures and algorithms of the same.

[0106] The embodiments of the matter described in this specification may be implemented as one or more computer program products, that is, one or more modules of computer program instructions encoded on a tangible program carrier for execution by, or for control the operation of the data processing apparatus. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, or a combination of one or more thereof.

[0107] A computer program (also known as a program, software, software application, writing or code) can be written in any form of programming language, which includes compiled or interpreted languages, declarative or process languages and the same it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program but need not necessarily correspond to a file on a file system. A program can be stored in a portion of a file that holds other programs or data (for example, one or more or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple files. of coordinates (for example, files that store one or more modules, subprograms or pieces of code). A computer program can be deployed to run on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected through a suitable communication network.

[0108] The processes and logical flows described in this descriptive report can be performed by one or more programmable processors that execute one or more computer programs to perform functions operating on input data and generating output. Logic processes and flows can also be realized, and the apparatus can also be implemented as special purpose logic circuitry, for example, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0109] The essential elements of a computer are a processor to carry out instructions and one or more memory devices to store instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from, or transfer data to, or both, one or more mass storage devices for storing data, for example, magnetic or magnetic-optical disks or disks. optics. However, a computer need not have such devices. Processors suitable for running a computer program include, by way of example only and without limitation, both general purpose and special purpose microprocessors, and any one or more processors of any type of digital computer. Generally, a processor will receive instructions and data from either read-only memory or random-access memory or both.

[0110] To provide for interaction with a user or manager of the system described in this document, embodiments of the matter described in this descriptive report can be implanted in a computer that has a display device, for example, a CRT monitor (ray tube cathode) or LCD (liquid crystal display) to display information to the user, and a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can be used to predict interaction with a user as well. For example, the feedback provided to the user can be any form of sensory feedback, for example visual feedback, auditory feedback, or tactile feedback; and user input can be received in any form, including haptic, voice control, or acoustic input.

[0111] The embodiments of the matter described in this specification can be implemented in a computing system that includes end-to-end component(s) that include one or more data servers, or that includes one or more middleware components such as data servers application, or that includes a front-end component such as a client computer that has a graphical user interface or a Web browser through which a user or administrator can interact with some deployment of the subject matter described is that descriptive report, or any combination of one or more such back-end, middleware, or front-end components. System components may be interconnected by any form or means of digital data communication, such as a communication network. The computing system can include clients and servers. A client and a server are generally remote from each other and typically interact over a communication network. The client and server relationship arises by virtue of computer programs running on the respective computers and having a client and server relationship with each other.

[0112] Although this descriptive report contains several specific implementation details, this should not be construed as limitations on the scope of any invention or what can be claimed, but rather as descriptions of features that may be specific to particular embodiments of inventions private. Certain features that are described in this report described in the context of separate modalities may also be deployed in combination with a single modalities.

[0113] Conversely, multiple features that are described in the context of a single modality may also be deployed across multiple modalities whether separately or in any suitable subcombination. Furthermore, while features may be described above as acting in certain combinations and even initially claimed as such, one or more features of a claimed combination may, in some cases, be removed from the combination and the claimed combination may be directed to a subcombination. or variation of a subcombination.

[0114] Similarly, although operations are shown in the drawings in a particular order, this should not be understood as a requirement that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed to achieve desirable results. In certain circumstances, parallel processing and multithreading can be advantageous. Furthermore, the separation of various system components in the embodiments described above is not to be construed as requiring such separation in all embodiments, and it is to be understood that the systems and program components described may generally be integrated into a single software product or packaged in multiple software products.

[0115] This written description sets forth the best mode of the invention and provides examples to describe the invention and enable a person of ordinary skill in the art to make and use the invention. This written description does not limit the invention to the precisely defined terms. Thus, although the invention has been described in detail with reference to the examples defined above, those of ordinary skill in the art can make changes, modifications and variations to the examples without departing from the scope of the invention.

Claims (15)

1. METHOD, characterized by comprising: identifying (202) an unknown video segment being displayed on a device, wherein the unknown video segment is identified based on comparing the pixel information received from the device with the stored pixel information of known video segments (203) and wherein the unknown video segment is provided to the device by a source; determining (204) a particular programming transmission from which the identified video segment is being sent; the server receiving a private programming stream from the source, wherein the server receives content of the private programming stream before the device receives the content of the private programming stream; the server identifying (206) a target video segment from the content of the particular programming stream to be displayed by the device, wherein the target video segment is identified based on comparing the pixel information received from the source with the stored pixel information of known video segments (203); determining a time to display additional content on the device, wherein the additional content relates to the target video segment; and the server sending (208) a trigger in response to identifying (206) the target video segment, the trigger causing the additional content to be displayed by the device at the designated time. 2. METHOD, according to claim 1, characterized in that it further comprises: identifying a latency for receiving the content of the particular programming transmission between the server and the device, in which the trigger is based on said latency. 3. METHOD, according to claim 2, characterized in that the trigger causes the additional content to be presented by the device with an offset based on latency. 4. METHOD, according to claim 1, characterized by the additional content being contextually related to the target video segment. 5. METHOD, according to claim 1, characterized by the server determining the particular programming transmission by comparing the identified video segment displayed on the device with a content schedule of one or more programming transmissions. 6. METHOD, according to claim 1, characterized in that additional content is displayed by the device when the initial frame of the target video segment is displayed by the device. 7. METHOD, according to claim 1, characterized by the server monitoring a plurality of devices, in which the first device is watching a first programming transmission, and in which the second device is watching a second programming transmission. 8. METHOD, according to claim 1, characterized in that the device is configured to identify the content being presented by the device, in which the trigger includes the identification of information of a frame of the target video segment, and in which the trigger makes additional content to be displayed by the device based on the device ID (206) of the target video segment. 9. METHOD, according to claim 1, characterized in that additional content is stored in the device before the trigger is received. 10. METHOD, according to claim 1, characterized in that the trigger includes at least one or more additional contents. 11. SYSTEM, characterized by comprising: one or more processors; and a non-transient computer-readable medium including instructions that, when executed by one or more processors, cause one or more processors to perform operations including: identifying (202) an unknown video segment being displayed on a device, in that the unknown video segment is identified based on comparing the pixel information received from the device with the stored pixel information of known video segments (203), and wherein the unknown video segment is provided to the device by a source; determining (204) a particular programming transmission in which the identified video segment is being sent; receiving a private programming transmission from the source, wherein the system receives content of the private programming transmission before the device receives the content of the private programming transmission; identifying (206) a target video segment from the content of the particular programming stream to be displayed by the device, wherein the target video segment is identified based on comparing the pixel information received from the source with the pixel information stored known video segments (203); determining a time to display additional content on the device, wherein the additional content relates to the target video segment; and sending (208) a trigger in response to identifying (206) the target video segment, the trigger causing the additional content to be displayed by the device at the designated time. 12. SYSTEM, according to claim 11, characterized in that the non-transient computer readable medium also includes instructions that when executed by one or more processors, cause one or more processors to perform operations including: identifying a receipt latency of particular programming transmission content between the system and the device, where the trigger is based on said latency. 13. SYSTEM, according to claim 12, characterized in that the trigger causes the additional content to be presented by the device with an offset based on latency. 14. SYSTEM, according to claim 11, characterized in that the additional content is contextually related to the target video segment. 15. SYSTEM, according to claim 11, characterized in that the system determines the particular programming transmission by comparing the identified video segment displayed on the device with a content schedule of one or more programming transmissions.

Images