Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
  • H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
  • H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
  • H04N21/43074—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV program
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
  • G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
  • G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
  • G06F16/7847—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using low-level visual features of the video content
  • G06F16/785—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using low-level visual features of the video content using colour or luminescence
• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
  • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
  • G11B27/02—Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
  • G11B27/031—Electronic editing of digitised analogue information signals, e.g. audio or video signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04H—BROADCAST COMMUNICATION
  • H04H60/00—Arrangements for broadcast applications with a direct linking to broadcast information or broadcast space-time; Broadcast-related systems
  • H04H60/35—Arrangements for identifying or recognising characteristics with a direct linkage to broadcast information or to broadcast space-time, e.g. for identifying broadcast stations or for identifying users
  • H04H60/37—Arrangements for identifying or recognising characteristics with a direct linkage to broadcast information or to broadcast space-time, e.g. for identifying broadcast stations or for identifying users for identifying segments of broadcast information, e.g. scenes or extracting programme ID
  • H04H60/375—Commercial
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04H—BROADCAST COMMUNICATION
  • H04H60/00—Arrangements for broadcast applications with a direct linking to broadcast information or broadcast space-time; Broadcast-related systems
  • H04H60/56—Arrangements characterised by components specially adapted for monitoring, identification or recognition covered by groups H04H60/29-H04H60/54
  • H04H60/59—Arrangements characterised by components specially adapted for monitoring, identification or recognition covered by groups H04H60/29-H04H60/54 of video
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
  • H04N21/23418—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
  • H04N21/254—Management at additional data server, e.g. shopping server, rights management server
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
  • H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
  • H04N21/2668—Creating a channel for a dedicated end-user group, e.g. insertion of targeted commercials based on end-user profiles
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
  • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
  • H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/47—End-user applications
  • H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
  • H04N21/4722—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for requesting additional data associated with the content
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
  • H04N21/81—Monomedia components thereof
  • H04N21/812—Monomedia components thereof involving advertisement data
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
  • H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
  • H04N21/835—Generation of protective data, e.g. certificates
  • H04N21/8358—Generation of protective data, e.g. certificates involving watermark
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
  • H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
  • H04N21/845—Structuring of content, e.g. decomposing content into time segments
  • H04N21/8456—Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
  • H04N21/85—Assembly of content; Generation of multimedia applications
  • H04N21/858—Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot

Definitions

• This invention generally relates to the matching of unknown media data, such as video or audio segments, against a massive database of reference media files.
• ACR automatic content recognition
• index is often too large to reside in its entirety in the main memory of a computer server as used in a typical ACR system.
• said database cannot fit entirely in the memory of a computer system, it is typically stored on magnetic disk storage and parts of said database are then read into memory in blocks corresponding to the index value providing the address. Said means of recalling partial database information is also known to one skilled in the art as "paging" which is a process common to many different computer software systems.
• the present invention is an extension of the invention referenced above and is a system and method for matching unknown digital media such as television programing to a database of known media using a signal processing means employing a modified path pursuit algorithm, as described in the first invention.
• Another novel aspect of the system and method as disclosed herein is its distance associative hash indexing means which can be subdivided into a plurality of independently addressable segments where each of said segments can address a portion of the database each of which can reside in its entirety in the main memory of a server means.
• the resulting cluster of servers of the indexing means each hosts a sector of the index addressing associated data of a larger database of searchable audio or video information.
• This indexing means of the invention results in a significant improvement in the speed and accuracy of the ACR system so enabled as to identify unknown media even when the television display is showing content where a user is changing channels, rewinding, fast-forwarding or even pausing video from a digital video recorder.
• an exemplary method related to addressing a media database using distance associative hashing may include receiving one or more indications of a sample of a video segment; determining, for at least one patch of the sample of a video segment including at least one or more pixels of the at least one patch, an algorithmically-derived value of the one or more pixels of each patch; subtracting a median point value established for each patch from the mean value for each patch; transforming the values resulting from the subtraction using a function pre-derived to distribute the values evenly; constructing a hash value from the transformed values; referencing a number of most significant bits of the constructed hash value to determine a database sector; and storing at least the hash value on the determined database sector.
• At least one of the receiving, determining, subtracting, transforming, constructing, referencing, or storing of the foregoing exemplary method is at least partially implemented using one or more processing devices.
• receiving one or more indications of a sample of a video segment may include receiving one or more indications of at least one of a frame or a still image.
• receiving one or more indications of a sample of a video segment may include receiving one or more indications of a sample of a video segment, the one or more indications of a sample of a video segment associated with at least one indication of a channel, at least one indication of a video segment, and at least one indication of a timecode offset from the beginning of the video segment.
• determining, for at least one patch of the sample of a video segment including at least one or more pixels of the at least one patch, an algorithmically-derived value of the one or more pixels of each patch includes at least determining, for at least one patch of the sample of a video segment including at least one or more pixels of the at least one patch, a mean value of the one or more pixels of each patch.
• subtracting a median point value established for each patch from the mean value for each patch may include subtracting a median point value established for each patch from the mean value for each patch, the median point value established for each patch having been previously determined utilizing data from each patch for a plurality of channels over at least one period of time.
• transforming the values resulting from the subtraction using a function pre-derived to distribute the values evenly may include forming a variable matrix including at least the values resulting from the subtraction; obtaining a static matrix which, when crossed with the variable matrix, will more evenly distribute the transformed values; and computing a dot product of the variable matrix and the static matrix, the dot product including at least the more evenly-distributed transformed values.
• obtaining a static matrix which, when crossed with the variable matrix, will more evenly distribute the transformed values may include determining, using locality-sensitive hashing at least partially based on one or more previously obtained hash values, a static matrix which, when crossed with a variable matrix, will more evenly distribute the transformed values of the variable matrix.
• constructing a hash value from the transformed values may include constructing a hash value from the transformed values, including at least reducing the fidelity of the transformed values via reducing each transformed value to a binary representation.
• reducing the fidelity of the transformed values via reducing each transformed value to a binary representation may include determining for each transformed value whether the transformed value is a positive number and, if the transformed value is a positive number, assigning a one to the hash value and otherwise assigning a zero to the hash value.
• referencing a number of most significant bits of the constructed hash value to determine a database sector may include referencing a number of most significant bits of the constructed hash value to determine a database server, wherein the number of most significant bits is pre-determined to address a plurality of database servers, wherein a number of database servers associated with the number of most significant bits is established to enable at least one index associated with a database sector to reside entirely in memory of a corresponding database server.
• storing at least the hash value on the determined database sector may include storing at least the hash value on the determined database sector, including at least storing at least one indication of a channel, at least one indication of a video segment, and at least one indication of a timecode offset from the beginning of the video segment at a database location at least partially based on the hash value.
• circuitry and/or programming for effecting the herein-referenced method embodiments; the circuitry and/or programming can be virtually any combination of hardware, software, and/or firmware configured to effect the herein-referenced method aspects depending upon the design choices of the system designer.
• an exemplary method related to addressing a media database using distance associative hashing may include receiving a cue, the cue constructed via one or more operations associated with a media storage operation; referencing a number of most significant bits of the received cue to determine a database sector; and returning at least one indication of at least one candidate from the database sector based at least partially on the received cue.
• receiving a cue, the cue constructed via one or more operations associated with a media storage operation may include receiving a cue associated with a sample of a video buffer of a client system, including at least receiving one or more indications related to an epoch time associated with the sample of the video buffer of the client system.
• receiving a cue, the cue constructed via one or more operations associated with a media storage operation may include receiving a cue, the cue associated with a sample of a video buffer of a client system, the cue at least partially determined by hashing at least some values associated with the video buffer.
• receiving a cue, the cue associated with a sample of a video buffer of a client system, the cue at least partially determined by hashing at least some values associated with the video buffer may include receiving a cue, the cue associated with a sample of a video buffer of a client system, the cue at least partially determined by hashing at least some values associated with the video buffer, the hashing based at least partially one or more of at least one operand or at least one algorithm also utilized in an associated media storage operation.
• receiving a cue, the cue constructed via one or more operations associated with a media storage operation may include receiving a cue, the cue determined via one or more operations including at least receiving one or more indications of at least one content of a video buffer of a client system; determining, for at least one patch of the at least one content of the video buffer including at least one or more pixels of the at least one patch, an algorithmically- derived value of the one or more pixels of each patch; subtracting a median point value from the mean value for each patch; transforming the values resulting from the subtraction; constructing a hash value from the transformed values; and associating the cue at least partially with the constructed hash value, wherein at least one of the determining, subtracting, transforming, or constructing operations utilize one or more of at least one operand or at least one algorithm also utilized in an associated media storage operation.
• returning at least one indication of at least one candidate from the database sector based at least partially on the received cue may include returning at least one indication of at least one candidate from the database sector based at least partially on a probabilistic point location in equal balls ("PPLEB") algorithm as a function of the received cue.
• returning at least one indication of at least one candidate from the database sector based at least partially on the received cue may include returning at least one indication of at least one candidate from the database sector based at least partially on the received cue, the at least one candidate being within a predetermined inverse percentage distribution radius of the received cue.
• circuitry and/or programming for effecting the herein-referenced method embodiments; the circuitry and/or programming can be virtually any combination of hardware, software, and/or firmware configured to effect the herein-referenced method aspects depending upon the design choices of the system designer.
• an exemplary method related to addressing a media database using distance associative hashing may include receiving at least one indication of at least one candidate and at least one indication of at least one cue; adding a token to a bin associated with at least one received candidate; and determining whether a number of tokens in a bin exceeds a value associated with a probability that a client system is displaying a particular video segment associated with at least one cue and, if the number of tokens in a bin exceeds a value associated with a probability that a client system is display a particular video segment associated with at least one cue, returning at least some data associated with the particular video segment based at least partially on the bin.
• adding a token to a bin associated with at least one received candidate may include adding a token to a time bin associated with at least one received candidate.
• adding a token to a bin associated with at least one received candidate may include determining a relative time, including at least subtracting a candidate time associated with the at least one candidate from an arbitrary time associated with the at least one cue; and adding a token to a time bin associated with the candidate based at least partially on the determined relative time.
• the method may include removing one or more tokens from a time bin based at least partially on a time period elapsing.
• circuitry and/or programming for effecting the herein-referenced method embodiments; the circuitry and/or programming can be virtually any combination of hardware, software, and/or firmware configured to effect the herein-referenced method aspects depending upon the design choices of the system designer.
• an exemplary system related to addressing a media database using distance associative hashing may include, but is not limited to, one or more computing devices; and one or more instructions that, when executed on at least some of the one or more computing devices, cause at least some of the one or more computing devices to at least receive at least one stream of rasterized video; create at least one hash value associated with at least one sample of at least one received rasterized video stream; determine at least one database sector for storing a created at least one hash value; and store a created at least one hash value on at least one determined database sector.
• an exemplary system related to addressing a media database using distance associative hashing may include, but is not limited to, one or more computing devices; and one or more instructions that, when executed on at least some of the one or more computing devices, cause at least some of the one or more computing devices to at least receive one or more indications associated with at least one video buffer of at least one client system; determine a cue based at least partially on the at least one video buffer and at least one epoch time associated with the at least one video buffer, wherein one or more of at least one operand or at least one function associated with determining the cue is also utilized in an associated media storage operation; reference a number of most significant bits of a determined cue to determine a database sector; and return at least one indication of at least one candidate from a determined database sector based at least partially on a determined cue.
• an exemplary system related to addressing a media database using distance associative hashing may include, but is not limited to, one or more computing devices; and one or more instructions that, when executed on at least some of the one or more computing devices, cause at least some of the one or more computing devices to at least receive at least one indication of at least one candidate and at least one indication of at least one cue; add a token to a bin associated with at least one received candidate; and determine whether a number of tokens in a bin exceeds a value associated with a probability that a client system is receiving a particular video segment associated with at least one received cue and, if the number of tokens in a bin exceeds a value associated with a probability that a client system is receiving a particular video segment associated with at least one received cue, returning at least some data associated with the particular video segment based at least partially on the bin.
• Figure 1 illustrates the construction of a sectored video matching database as taught by this invention which begins with initial video ingest or capture process which is then continuously updated.
• a television display system 101 and its corresponding television display memory buffer 103 are shown for a potential embodiment of the system.
• the allocation of pixel patches 102 and a calculation of a value 105, using certain algorithmic means known to those skilled in the art, is made for each pixel patch and a resulting data structure is created and then time-stamped make a "cue" 106 which may also have additional metadata associated with it.
• Figure 2 illustrates the processing of the cue data 201 and the generation of the hash index 202 using the distance associative hashing process, further illustrating the sectored addressing scheme 203 to store data in related groups (buckets) 206.
• Figure 3 illustrates the real-time capture of unknown television content for recognition from a connected television monitor or the like 301.
• a pixel patch is defined as typically a square pixel area of the video buffer 303 with dimensions of perhaps ten pixels by ten rows of pixels 304, however, any reasonable shape and dimension may be used.
• the number of pixel patch positions can be any number between ten and fifty locations within said video buffer and is processed 305 to send cue data 306 to the central server means.
• Figure 4 illustrates the extraction of candidate cue values 401 from the reference (matching) database bucket 404 and supplying said cue values 403 to the path pursuit content matching process 402 as taught in the first invention referenced above.
• Figure 5 illustrates the data structure of bins which hold tokens for scoring candidate values from the matching database. Said bins are "leaky” and tokens expire over time as the search process progresses through time.
• Figure 6 illustrates a typical memory paging scheme as taught by prior art for accessing large databases.
• Figure 7 illustrates the creation of a hash value involving several steps beginning with computing the median value of each of the multiplicity of points which make up the samples from a frame of video.
• Figure 8 illustrates how the hash value is computed.
• Figure 9 illustrates the beneficial results of using the median values of a pixel location as part of the process of computing the hash values.
• Figure 9a illustrates the problem of not using a media value when partitioning a multi-dimensional dataset.
• Figure 9b illustrates the benefit of finding a media value of a dataset.
• Figure 10 illustrates an operational flow representing example operations related to addressing a media database using distance associative hashing.
• Figure 11 illustrates an alternative embodiment of the operational flow of Figure 10.
• Figure 12 illustrates an alternative embodiment of the operational flow of Figure 10.
• Figure 13 illustrates an alternative embodiment of the operational flow of Figure 10.
• Figure 14 illustrates an alternative embodiment of the operational flow of Figure 10.
• Figure 15 illustrates an alternative embodiment of the operational flow of Figure 10.
• Figure 16 illustrates an alternative embodiment of the operational flow of Figure 10.
• Figure 17 illustrates an alternative embodiment of the operational flow of Figure 10.
• Figure 18 illustrates an alternative embodiment of the operational flow of Figure 10.
• Figure 19 illustrates an alternative embodiment of the operational flow of Figure 10.
• Figure 20 illustrates a different operational flow representing example operations related to addressing a media database using distance associative hashing.
• Figure 21 illustrates an alternative embodiment of the operational flow of Figure 20.
• Figure 22 illustrates an alternative embodiment of the operational flow of Figure 20.
• Figure 23 illustrates an alternative embodiment of the operational flow of Figure 20.
• Figure 24 illustrates another operational flow representing example operations related to addressing a media database using distance associative hashing.
• Figure 25 illustrates an alternative embodiment of the operational flow of Figure 24.
• Figure 26 illustrates an alternative embodiment of the operational flow of Figure 24.
• Figure 27 illustrates a system related to addressing a media database using distance associative hashing.
• Figure 28 illustrates another system related to addressing a media database using distance associative hashing.
• Figure 29 illustrates yet another system related to addressing a media database using distance associative hashing.
• the first invention which relates to this invention is a system and method of matching unknown video to a database of known video using a novel signal processing means employing a modified path pursuit algorithm, among other means, as described in the aforementioned publication.
• a novel means of the new invention is its Distance Associated Hashing with its attendant provision of utilizing a sectored-index database access.
• Said indexing means provides a highly computationally-efficient means for matching an unknown media segment to a reference database of known media, such as audio or video content.
• This indexing means of the invention results in a significant improvement in the speed and accuracy of the ACR system so enabled as to track the identity of media even when the television display is showing content where a user is changing channels, rewinding, fast- forwarding or even pausing video from a digital video recorder.
• Both the building, updating, and the subsequent accessing of the media matching database will describe a system capable of generating and addressing a sectored database such that the database sectors can each reside in the main memory of a respective multiplicity of server means without resorting to a paging means within each of the respective server means.
• This collective means of addressing a sectored database through locality sensitive hashing provides a significant improvement in efficiency of operation.
• the construction of a sectored video matching database begins with the process as illustrated in Figure 1.
• a television system 101 decodes a television signal and places the contents of each frame of video into a video frame buffer in preparation for the display or further processing of pixel information of the frame of video.
• Said television system can be any television decoding system that can decode a television signal whether from a baseband or modulated television source and fill a video frame buffer with the decoded RGB values at the respective frame size as specified by the video signal.
• Such systems are well known to one skilled in the art.
• the system of the invention first builds and then continuously updates a reference database of television programming fingerprints described in the original application as cues or cue values.
• the invention performs the acquisition of one or more patches of video 102 which are read from the video frame buffer 103.
• Said video patches can be any arbitrary shape or pattern but for the purposes of this example shall be 10 pixels horizontally by 10 pixels vertically. Also for the sake of this example, assume that there are 25 pixel patch positions within the video frame buffer that are evenly distributed within the boundaries of said buffer, though they do not have to be evenly distributed.
• Each pixel shall consist of a red, a green and a blue value, 104, typically represented by an eight bit binary value for each color for a total of 24 bits or three bytes per patch location.
• This composite data structure is populated with the average pixel values from a number of pixel patch positions from the video buffer.
• a pixel patch is defined as a typically square pixel area of the video buffer with dimensions of perhaps ten pixels by ten rows of pixels 304. The number of pixel patch positions might typically be between ten and fifty locations within the video buffer.
• Epoch time is defined as the time in fractions of a second that have elapsed since midnight, January 1,1970 which is an accepted convention in computing systems, particularly with Unix (or Linux)-based systems.
• Metadata may be included and together a data structure 106 is defined called a tagged fingerprint, "cue”, or a "point”, as taught in the original patent application.
• metadata attributes might be derived from closed-captioning data from the currently displaying video program or it could be keywords extracted by means of a speech recognition system operating within the processor means of the television system which converts audio from the respective television program into text information. Said textual information may then be searched for relevant key words or sent in its entirety as part of the cue data structure to the central server means for further processing.
• the cue records 201 are passed in Figure 2 to a hash function 202 that generates a hash value 203 using a locality sensitive hashing algorithm based on Probabilistic Point Location in Equal Balls algorithm (PPLEB).
• PPLEB Probabilistic Point Location in Equal Balls algorithm
• the ten by ten pixel patch 302 shown in this particular example would have one hundred pixels and is mathematically averaged resulting in a mean pixel value 305 for red, green and blue values, respectively.
• any averaging function can be used in place of a simple mean.
• a plurality of such pixel patches are extracted from the video frame. If, by way of example, 25 such pixel patches are extracted from the video frame, then the result will be a point representing a position in a 75-dimension space. The skilled person will know that such a large search space could require extensive computing resources to later locate, even approximately, said value in combination with the other 74 values representing one frame of video.
• the creation of the hash value involves several steps beginning with computing an algorithmically-derived value of each point as shown in Figure 7, 701 to 775.
• One useful means of algorithmically deriving said median value is found by summing each point of every frame of every program stream or channel of video maintained by the matching database over perhaps a 24 hour period. The median of each point is found from the summation process.
• the next step in deriving the final hash value is to subtract the mean value from the point value of each respective point, row 801 minus row 802 equals row 803.
• the result is a plus or minus values to which a pre-derived hashing function is applied.
• the result of the point values minus the mean values of the respective points are arranged in a matrix to which a dot- product is calculated using a similar matrix constituting the pre-derived hash value (or key).
• the result of the dot product of the two matrices is then further transformed to a one or zero value based on the sign of the product matrix element.
• the skilled person would set positive values to one and negative values to zero.
• the resulting hash value points to more or less evenly distributed values across the data storage area.
• the hash value 203 can be further divided, Figure 2, such that the 'n' most significant bits 205 addresses one of the 2 n (2 A n) sectors of the database.
• the remaining bits 206 address individual 'buckets' of the addressed sector of the database, which will be described in more detail later.
• Figure 6 illustrates a typical paging approach.
• Figure 6 assume the example system is attempting to match unknown data to a database of known data.
• An index 602 is used to address only the portion of the data 605 that can fit in the main memory of the CPU 606. This data is searched and, if results are negative, then another segment of data is fetched into main memory 603 and searching continues.
• Google search engine A notable example might be the considerable Google search engine.
• the skilled person knows this system to be one of the largest computer systems built to date. The speed and accuracy of the Google search process is remarkable.
• the Google search means is considerably different and not at all applicable to matching unknown media to a database of known media even if the two databases were the same very large size. This is because the Google search means employs the map-reduce algorithm which is designed for searching large databases of essentially unassociated data. While an advance over paging system, map-reduce is a computationally-intense process which also requires significant data communications bandwidth between the participating computer systems. In contrast, this invention is efficient in the use of processing and communications resources.
• the distance associative hashing function provides a means to address a database in sectors such that the data of said addressing means fits in the main memory of an individual server means of group of servers.
• Said grouping is accomplished by grouping the data related by distance in a multi-dimensional array into the same sector using the distance associative hashing step as a means to achieve said grouping.
• the sector identification for addressing a data element is calculated from the hash index generated from said process by extracting a subset of the total bits of said hash function and using said subset to address the desired sector in which to store data in the reference database.
• the hash index subset is the address of the sector that contains the distance associated hash values, called buckets in the first invention.
• the remainder of the hash address is then used to address a bucket of the sector for storing the new data.
• the sector address can be found by means of re-hashing the first hash value.
• the distance associative hashing provides a means to address a very complex (multi-dimensional) database quickly by finding data that is not an exact match but rather is within a predetermined radius (distance associative) of the value sought. Importantly, sometimes this addressing means will result in no match at all. Where a business-oriented database cannot tolerate inaccuracy, a media matching system can readily tolerate missed matches and will simply continue the matching process upon the arrival of the next data received and taught in the first patent.
• Data arrival from the unknown source that is to be determined by the ACR system is periodic, of course, but can be commanded by the system of the invention to arrive at differing intervals based on the requirement for accuracy or by requirements imposed by the state of the system such as when the system might be nearing overload and the sending clients are then commanded to send a lower sample rate.
• a typical data reception rate might be 1/10 second intervals, for example.
• the group of pixel values are derived from every frame of video from every video source that is to be part of the reference database.
• the group of pixel values and are then appended with the broadcast time of the video program as well as with certain metadata, which is information about the program such as the content identification (ID), title of the program, actors name, time of airing, short synopsis, etc.
• Said metadata is generally acquired from commercial electronic program guide sources.
• Said array of processed pixel values with the addition of the timecode plus the metadata are then stored in the reference database and the address of said stored data is then added to the hash index at the respective hash value and sector ID value.
• a second database index is built and maintained by using the content ID from the metadata as another means for addressing the reference database.
• the process of building and continuously updating the database is continuous and the number of days of data maintained by the database is based on the needs of the user but for example might range from one day to one month.
• the process of identifying an unknown video segment from data received from a multiplicity of client devices begins with a procedure similar to that used above for building the reference database.
• this procedure involves a television monitor 301, such as a popular flat-screen HDTV typically of the type known as the smart TV wherein the TV contains a processing means with memory capable of executing application programs similar to the type found on common smartphones of today.
• the system of the invention samples regions 302 of a video frame buffer 301 in typically a multiplicity of places. Said samples are of an identical size, shape and position to the pixel patches used in the process of building the reference database.
• Each of the collected pixel patches is then algorithmically processed to produce a computed value for the red, green and blue values of each patch in a manner identical to the method used to create the reference database.
• Said system of the invention calculates a distance associative hash index of the collected mean values identical to the content ingest function described above.
• the resulting sector identification (ID) value is extracting as a subset of the total bits of the hash index also identically to the ingest system described above.
• the remainder of the hash index is used to address the desired sector in which to search for all candidate (potential) matches belonging to the same bucket as the unknown data point.
• the system of the invention will also collect candidates from the database responsible for said sector belonging to the potential content ID, using the content ID index, created during the ingest process as described above, to address reference cues around time radius r' of the timestamp (of the successfully matched candidate). Duplicate candidates are next removed as well as candidates that are too far from the unknown point by radius r, as taught in the first patent.
• Each matching candidate 501 is assigned a data structure 502 in the memory of the matching system of the invention.
• the data structure consists of, among other things, arbitrary time bins grouped by some arbitrary amount (e.g. approximately one second). For the sake of example, assume said data structure consists of one hundred bins representing ten seconds of video cue points. The bins are generally not equally spaced in time.
• a relative time is calculated by subtracting candidate time from the arbitrary time of the unknown video.
• Candidate time is the time of broadcast of each video cue associated with the candidate during the reference program airing.
• Epoch time is well known to the skilled person and is typically employed in computer systems. Said time is calculated as the current number of units of time since January 1 of 1970.
• the relative time of the actually matching candidates should be close to that value. Likewise, candidates that are not a good match are not likely to have relative times close to the 100 seconds of this example.
• the system of the invention adds a token to the respective bin of the candidate data structure. Said system then repeats the process for the next candidate as described in the previous paragraph.
• Another, and important, step for the scoring of results is to apply time discounting to all bins.
• This is a relatively simple process that decrements the value in all bins by a small amount for each cycle of time.
• the skilled person would recognize this as a "leaky bucket" method of scoring.
• bins that are no longer being filled by means of matching cue points will ultimately decrement to zero over a number of cycles of said process.
• bins that are filled slowly by random noise in the system will likewise be decremented.
• time discounting ultimate clears bins that are filled by false-positive matches and random noise.
• the skilled person would also clearly see that without said time discount binning, all bins would eventually fill to capacity and no results could be obtained from the process.
• Said time discounting also decrements to zero any bins with levels, such as 503, that are above the matching threshold 504 when the video stream from the client television monitor is in any way changed by any of the following: changing channels, rewind, fast forward, pausing video, etc.
• Figure 8 illustrates how the hash value is computed. First a median value of each pixel location contributing to the video fingerprint is found by summing the values of said location over a period of many days of collection values at said location from a plurality of television channels representative of the typical television programming to be identified by the invention. Once the median value is determined is can be used indefinitely as a constant without further calculation or adjustment.
• the pixel value sent from the client to the server matching system is first processed by subtracting the median value of said pixel location.
• the resulting value is stored with the other pixels locations of a video frame in matrix and an appropriate hashing function is applied to said matrix. Hash values are then derived from the resulting dot product.
• Figure 9 illustrates the beneficial results of using the median values of a pixel location as part of the process of computing the hash values.
• Chart 901 shows the resulting curve of the output of a typical un-optimized hash function with a relatively small number of hash values occupying a relatively narrow range on the left edge of the curve.
• the resulting median value 902 is relatively low.
• Chart 903 shows the favorable redistribution of hash values as a resulting of computing the median of each pixel location that participates in the matching process and applying said median value as part of the hashing function.
• the distribution of hash values is more spread out with an attendant rise in the median value of all hash keys 904.
• Figure 9a illustrates what happens to a dataset when a median value is not found prior to partitioning said dataset. If the system sampled sixteen pixel locations of each video frame and if each pixel location had a red, green and blue pixel value, there would be 64 dimensions (or axis) to the graph. For the sake of illustration, in this example, the dataset includes just two pixel sample points of a single video frame 906 and 908. Further, the example assumes just a single luminance value is obtained at each pixel point.
• Figure 9b illustrates the benefit of finding the median value of each pixel location.
• This example continues to use the assumption that the pixel values are a single luminance value from zero to 255, although absolute value is of no consequence to this method.
• This example illustrates a simplistic assumption of the median value is 128 for both pixel locations.
• Diagonal slice 909 moves to 909'. It is clear from the illustration that now all eight sectors contain data.
• the data can be spliced more than once around each median point of the 48 dimension graph as required to partition said data such that said dataset resulting from said slice can be made to fit within the operational constraints of an individual computer server of the system. In any case, data will be found most of the time on the clockwise and counterclockwise side of each partition slice.
• Figure 10 illustrates an operational flow 1000 representing example operations related to addressing a media database using distance associative hashing.
• discussion and explanation may be provided with respect to the above-described examples of Figures 1 through 9, and/or with respect to other examples and contexts.
• the operational flows may be executed in a number of other environments and contexts, and/or in modified versions of Figures 1 through 9.
• the various operational flows are presented in the sequence(s) illustrated, it should be understood that the various operations may be performed in other orders than those which are illustrated, or may be performed concurrently.
• Operation 1002 depicts receiving one or more indications of a sample of a video segment.
• the indications may be associated with one or more pixel patches from an ingest system.
• operation 1004 depicts determining, for at least one patch of the sample of a video segment including at least one or more pixels of the at least one patch, an algorithmically-derived value of the one or more pixels of each patch. For example, as shown in and/or described with respect to Figures 1 through 9, a mean value of the red pixels in each patch, the green pixels in each patch, and the blue pixels in each patch may be computed.
• operation 1006 depicts subtracting a median point value established for each patch from the mean value for each patch.
• a median value of each pixel location contributing to the video fingerprint may be found by summing the values of said location over a period of many days of collection values at said location from a plurality of television channels.
• operation 1008 depicts transforming the values resulting from the subtraction using a function pre-derived to distribute the values evenly.
• the values resulting from the subtraction populate a matrix.
• a dot product of that matrix and a pre-derived static matrix may be computed.
• the pre-derived static matrix may be determined prior to operational flow 1000 being instantiated, and may be optimized mathematically based on past ingested data such that matrices crossed with it will produce more evenly distributed results than results coming directly from the subtraction operation.
• operation 1010 depicts constructing a hash value from the transformed values.
• a hash value may be a string of bits.
• operation 1012 depicts referencing a number of most significant bits of the constructed hash value to determine a database sector.
• a number of bits may be predetermined so that the predetermined number of bits of a hash value are used for addressing one or more database sectors.
• operation 1014 depicts storing at least the hash value on the determined database sector.
• the hash value may be stored in a bucket, the bucket including other hash values which are mathematically near, where the hash values are associated at least with particular video segments and offsets.
• Figure 11 illustrates alternative embodiments of the example operational flow 1000 of Figure 10.
• Figure 11 illustrates an example embodiment where operational flow 1000 may include at least one additional operation. Additional operations may include operation 1102.
• Operation 1102 illustrates at least one of the receiving 1002, determining 1004, subtracting 1006, transforming 1008, constructing 1010, referencing 1012, or storing 1014 operations being at least partially implemented using one or more processing devices.
• one of the foregoing operations may be at least partially implemented using one or more computer processors.
• Other processing devices may include Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), digital signal processors (DSPs), or any other circuitry configured to effect the result of at least one of the foregoing operations.
• ASICs Application Specific Integrated Circuits
• FPGAs Field Programmable Gate Arrays
• DSPs digital signal processors
• Figure 12 illustrates alternative embodiments of the example operational flow 1000 of Figure 10.
• Figure 12 illustrates an example embodiment where operation 1002 may include at least one additional operation. Additional operations may include operation 1202, and/or operation 1204.
• Operation 1202 illustrates receiving one or more indications of at least one of a frame or a still image.
• a sample of a video segment may be comprised of an individual frame of a video stream. Such a frame may be one 30fps video frame.
• a sample of a video segment may be a still image, or a portion of a video segment that may be imaged at a rate other than 30 times a second.
• operation 1204 illustrates receiving one or more indications of a sample of a video segment, the one or more indications of a sample of a video segment associated with at least one indication of a channel, at least one indication of a video segment, and at least one indication of a timecode offset from the beginning of the video segment.
• data associated with a video segment (which may be a program title and/or other metadata associated with a video segment), the channel from which the program was ingested, and an offset in time from the start of the program may be received, from, for example, a channel guide associated with a channel which is being monitored by the ingest system.
• Figure 13 illustrates alternative embodiments of the example operational flow 1000 of Figure 10.
• Figure 13 illustrates an example embodiment where operation 1004 may include at least one additional operation 1302.
• Operation 1302 illustrates determining, for at least one patch of the sample of a video segment including at least one or more pixels of the at least one patch, a mean value of the one or more pixels of each patch.
• the algorithmic operation used to reduce the one or more pixels in a patch to a single value may be, for example, an arithmetic mean.
• Figure 14 illustrates alternative embodiments of the example operational flow 1000 of Figure 10.
• Figure 14 illustrates an example embodiment where operation 1006 may include at least one additional operation 1402.
• Operation 1402 illustrates subtracting a median point value established for each patch from the mean value for each patch, the median point value established for each patch having been previously determined utilizing data from each patch for a plurality of channels over at least one period of time.
• a median value may be determined, the median value determined for each patch, wherein medians are established for the same patches at ingest as in the operation of determining a segment on a client system, the median being established as a constant value derived from monitoring the same patches across many channels for a long time (a month, a year, etc.).
• Figure 15 illustrates alternative embodiments of the example operational flow 1000 of Figure 10.
• Figure 15 illustrates an example embodiment where operation 1008 may include at least one additional operation. Additional operations may include operation 1502, operation 1504, and/or operation 1506.
• Operation 1502 illustrates forming a variable matrix including at least the values resulting from the subtraction. For example, as shown in and/or described with respect to Figures 1 through 9, values are arranged in a matrix, the values resulting from the subtraction operation, wherein the subtraction operation subtracts the median value established over time for each patch from the mean value of the instant frame being ingested.
• Operation 1504 illustrates obtaining a static matrix which, when crossed with the variable matrix, will more evenly distribute the transformed values.
• a matrix may be determined based upon mathematical analysis of previously-obtained data sets related to hash values.
• the matrix may be optimized mathematically such that, when used as an operand in a dot product operation with successive variable matrices, the corresponding successive result matrices will include values that are more evenly spread along a distribution curve than the variable matrices prior to the dot product operation.
• Operation 1506 illustrates computing a dot product of the variable matrix and the static matrix, the dot product including at least the more evenly-distributed transformed values.
• the variable matrix containing values resulting from the subtraction operation may be crossed with a static matrix that has been predetermined to distribute data represented by a variable matrix more evenly, such that the resulting matrices are more spread out instead of being bunched about a particular portion of the distribution.
• Figure 16 illustrates alternative embodiments of the example operational flow 1000 of Figure 10.
• Figure 16 illustrates an example embodiment where operation 1504 may include at least one additional operation 1602.
• Operation 1602 illustrates determining, using locality-sensitive hashing at least partially based on one or more previously obtained hash values, a static matrix which, when crossed with a variable matrix, will more evenly distribute the transformed values of the variable matrix.
• a locality-sensitive hashing technique may be used to analyze previously-ingested video samples, producing a matrix such that, when used as an operand in a dot product operation with successive variable matrices, the corresponding successive result matrices will include values that are more evenly spread along a distribution curve than the variable matrices prior to the dot product operation.
• Figure 17 illustrates alternative embodiments of the example operational flow 1000 of Figure 10.
• Figure 17 illustrates an example embodiment where operation 1010 may include at least one additional operation. Additional operations may include operation 1702, and/or operation 1704.
• Operation 1702 illustrates constructing a hash value from the transformed values, including at least reducing the fidelity of the transformed values via reducing each transformed value to a binary representation.
• each value of the resultant matrix from the dot product operation may be reduced from, for example, an 8-bit value from 0 to 255 (or from -127 to 128) to a single bit, being either a one or a zero.
• Operation 1702 may include operation 1704.
• Operation 1704 illustrates determining for each transformed value whether the transformed value is a positive number and, if the transformed value is a positive number, assigning a one to the hash value and otherwise assigning a zero to the hash value. For example, as shown in and/or described with respect to Figures 1 through 9, each value of the resultant matrix from the dot product operation between 1 and 128 may be reduced to a bit value of 1, and each value of the resultant matrix from the dot product operation between -127 and 0 may be reduced to a bit value of 0.
• Figure 18 illustrates alternative embodiments of the example operational flow 1000 of Figure 10.
• Figure 18 illustrates an example embodiment where operation 1012 may include at least one additional operation 1802.
• Operation 1802 illustrates referencing a number of most significant bits of the constructed hash value to determine a database server, wherein the number of most significant bits is pre-determined to address a plurality of database servers, wherein a number of database servers associated with the number of most significant bits is established to enable at least one index associated with a database sector to reside entirely in memory of a corresponding database server.
• a number of most significant bits of 2 may be selected, whereby the 2 bits may provide four different values (00, 01, 10, and 11), each of which may be assigned to a different database sector.
• the number of most significant bits of a hash value may be established to provide a sufficient number of servers such that a content associated with a plurality of hash values may fit entirely in the memory of a particular database sector, which may be a database server, a cluster partner, a virtual machine, and/or another type of database node.
• the number of bits does not have to, but may, exactly represent the maximum number of database sectors at any given time (i.e. while 6 bits may be selected to provide for addressing of up to 64 database sectors, the system may be operable with fewer servers e.g. 60 sectors, or with the maximum 64 sectors).
• Figure 19 illustrates alternative embodiments of the example operational flow 1000 of Figure 10.
• Figure 19 illustrates an example embodiment where operation 1014 may include at least one additional operation 1902.
• Operation 1902 illustrates storing at least the hash value on the determined database sector, including at least storing at least one indication of a channel, at least one indication of a video segment, and at least one indication of a timecode offset from the beginning of the video segment at a database location at least partially based on the hash value.
• data associated with a video segment (which may be a program title and/or other metadata associated with a video segment), the channel from which the program was ingested, and an offset in time from the start of the program may be stored, either along with the hash value or in a location associated with and/or referenceable by the hash value, the storage being in the same or different sector, server, or database as the hash value.
• Figure 20 illustrates an operational flow 2000 representing example operations related to addressing a media database using distance associative hashing.
• discussion and explanation may be provided with respect to the above-described examples of Figures 1 through 9, and/or with respect to other examples and contexts.
• the operational flows may be executed in a number of other environments and contexts, and/or in modified versions of Figures 1 through 9.
• the various operational flows are presented in the sequence(s) illustrated, it should be understood that the various operations may be performed in other orders than those which are illustrated, or may be performed concurrently.
• Operation 2002 depicts receiving a cue, the cue constructed via one or more operations associated with a media storage operation.
• a cue the cue constructed via one or more operations associated with a media storage operation.
• at least some data is received which is associated with a sample of video data taken from a particular client system.
• the data may be associated with exactly the same patches of the client system as are defined by the ingest operation.
• the data may be algorithmically processed to arrive at a hash value using the same operations as the ingest operation.
• the same hashing operations as applied to the ingested frame will result in the same hash value as resulted from the hashing operations on the ingested frame.
• the cue of operation 2002 represents data associated with a sample of video data from a particular client system.
• a cue may be received via, for example, an HTTP request.
• operation 2004 depicts referencing a number of most significant bits of the received cue to determine a database sector.
• the same bits of the cue are examined as defined by the number of most significant bits used to reference a database sector during ingest. For example, if the first two bits of the hash value at ingest are used for storing the hash value at a particular database sector, the same first two bits of the cue associated with a sample of video data from a client system are used for addressing a particular database sector.
• operation 2006 depicts returning at least one indication of at least one candidate from the database sector based at least partially on the received cue. For example, as shown in and/or described with respect to Figures 1 through 9, hash values which exactly match the cue, or are nearby the cue, are returned as one or more of suspects or candidates. Candidates may be returned within a particular percentage radius. Candidates may be returned according to a nearest neighbor algorithm or a modified nearest neighbor algorithm.
• Figure 21 illustrates alternative embodiments of the example operational flow 2000 of Figure 20.
• Figure 21 illustrates an example embodiment where operation 2002 may include at least one additional operation. Additional operations may include operation 2102, operation 2104, and/or operation 2106.
• Operation 2102 illustrates receiving a cue associated with a sample of a video buffer of a client system, including at least receiving one or more indications related to an epoch time associated with the sample of the video buffer of the client system.
• a cue may include, or be associated with, a time offset from an arbitrary time. The time offset may be computed from January 1, 1970, for example.
• Operation 2104 illustrates receiving a cue, the cue associated with a sample of a video buffer of a client system, the cue at least partially determined by hashing at least some values associated with the video buffer.
• patches associated with a video buffer may be reduced to a bit string via one or more mathematical operations or algorithms using one or more operands as constants, the constants pre-derived via operations described elsewhere herein with respect to hashing, for example.
• Operation 2106 illustrates receiving a cue, the cue associated with a sample of a video buffer of a client system, the cue at least partially determined by hashing at least some values associated with the video buffer, the hashing based at least partially one or more of at least one operand or at least one algorithm also utilized in an associated media storage operation.
• at least some data associated with a sample of a video buffer representing what is displayed by a television screen at a particular quantum of time is processed via operations utilized by the ingest process and/or in conjunction with data locations common to the ingest process and/or involving constant values for operands utilized by the ingest process.
• the number of patches analyzed at ingest may also be utilized in providing a cue associated with a particular client system.
• the size of pixel patches analyzed at ingest may also be utilized in providing a cue associated with a particular client system.
• the same pre-derived static matrix used to more evenly distribute hash values at ingest may also be used during hashing of the data associated with a particular client system.
• Figure 22 illustrates alternative embodiments of the example operational flow 2000 of Figure 20.
• Figure 22 illustrates an example embodiment where operation 2002 may include at least one additional operation. Additional operations may include operation 2202, operation 2204, operation 2206, operation 2208, operation 2210, operation 2212, and/or operation 2214.
• Operation 2202 illustrates receiving one or more indications of at least one content of a video buffer of a client system.
• pixel values for red, green, and blue pixels at every pixel location at every pre-defined patch of the video buffer of the client system may be read, for every frame, or for every third frame, or for every tenth frame, or for every second, or at some other interval.
• the indications (pixel values or other data) may be received by a widget on the television, by control logic on the television, by a system coupled with the media server, or elsewhere.
• Operation 2204 illustrates determining, for at least one patch of the at least one content of the video buffer including at least one or more pixels of the at least one patch, an algorithmically-derived value of the one or more pixels of each patch. For example, as shown in and/or described with respect to Figures 1 through 9, pixel values for red, green, and blue pixels at every pixel location at every pre-defined patch of the video buffer of the client system may be averaged.
• Operation 2206 illustrates subtracting a median point value from the mean value for each patch.
• median point values at each patch established through analysis of ingested content are determined.
• the median point values for each patch may, for example, be provided to the client system once determined by a system associated with the media database and ingest system.
• the median point values may be updated from time to time (hourly, daily, monthly, yearly).
• the median point values provided for hashing data associated with a video buffer of a client system may be the same median point values utilized to hash incoming content at ingest.
• Operation 2208 illustrates transforming the values resulting from the subtraction.
• values resulting from the subtraction are populated in a matrix and crossed with a pre-defined static matrix.
• the dot-product operation crossing the two matrices may be conducted at the client system during a process of converting pixel patch data associated with a frame in a video buffer to a cue, such that a cue is sent in an HTTP request rather than the actual pixel patch data, resulting in a compact HTTP message.
• the pre-defined static matrix may be provided to the client system in advance of the transform, and may be the same matrix as was produced to distribute hashed values at ingest more evenly.
• the pre-defined static matrix may be updated at the client system from time to time.
• patch data may be sent, with or without other metadata, from a client system (television, e.g.) to a different system for processing and/or hashing.
• Operation 2210 illustrates constructing a hash value from the transformed values.
• the values in the matrix resulting from crossing the matrix with values associated with the video buffer with the pre-derived static matrix may be reduced to bits, with a single bit replacing each 8-bit value in the matrix.
• the constructed hash value may include a different number of bits for each value in the matrix.
• the constructed hash value may have the same number of bits as the values in the matrix, or may be a direct representation of the values in the matrix.
• Operation 2212 illustrates associating the cue at least partially with the constructed hash value.
• the string of bits constructed from the transformed matrix may be a cue, or may associate the constructed string of bits with a time (such as an epoch time) to form a cue, or may associate other data such as an IP address or other identifier associated with the client television or a widget of the client television to form a cue.
• the cue may include or otherwise be associated with any other metadata associated with audiovisual content at the client system.
• Operation 2214 illustrates at least one of the determining 2204, subtracting 2206, transforming 2208, or constructing 2210 operations utilize one or more of at least one operand or at least one algorithm also utilized in an associated media storage operation.
• one or more parameters including one or more of a definition of a number of pixel patches, a definition of a size of pixel patches, a pre-defined median value associated with pixel patches, or a pre-defined static matrix may be provided to a client TV, the one or more parameters also utilized by the ingest process such that operations applied to a sample from a video buffer will result in the same hash value that would result when that frame (e.g.
• Figure 23 illustrates alternative embodiments of the example operational flow 2000 of Figure 20.
• Figure 23 illustrates an example embodiment where operation 2006 may include at least one additional operation. Additional operations may include operation 2302, and/or operation 2304.
• Operation 2302 illustrates returning at least one indication of at least one candidate from the database sector based at least partially on a probabilistic point location in equal balls (“PPLEB”) algorithm as a function of the received cue.
• PPLEB probabilistic point location in equal balls
• at least one of candidates or suspects representing path points close to a cue are returned from a media database constructed and/or modified via an ingest process.
• Operation 2304 illustrates returning at least one indication of at least one candidate from the database sector based at least partially on the received cue, the at least one candidate being within a predetermined inverse percentage distribution radius of the received cue. For example, as shown in and/or described with respect to Figures 1 through 9, at least one of candidates or suspects associated with locality sensitive hashing related to at least one of a cue or a hash value are returned.
• Figure 24 illustrates an operational flow 2400 representing example operations related to addressing a media database using distance associative hashing.
• discussion and explanation may be provided with respect to the above-described examples of Figures 1 through 9, and/or with respect to other examples and contexts.
• the operational flows may be executed in a number of other environments and contexts, and/or in modified versions of Figures 1 through 9.
• the various operational flows are presented in the sequence(s) illustrated, it should be understood that the various operations may be performed in other orders than those which are illustrated, or may be performed concurrently.
• Operation 2402 depicts receiving at least one indication of at least one candidate and at least one indication of at least one cue. For example, as shown in and/or described with respect to Figures 1 through 9, a hash value related to a video buffer of a client system, along with one or more associated candidates or suspects is determined.
• operation 2404 depicts adding a token to a bin associated with at least one received candidate.
• scoring of candidates is performed via tokens added to bins corresponding to candidates/suspects, the token being, for example, a value which is incremented each time a token is added.
• operation 2406 depicts determining whether a number of tokens in a bin exceeds a value associated with a probability that a client system is displaying a particular video segment associated with at least one cue and, if the number of tokens in a bin exceeds a value associated with a probability that a client system is display a particular video segment associated with at least one cue, returning at least some data associated with the particular video segment based at least partially on the bin.
• a determination of a particular video segment and particular offset of the video segment is probabilistically determined via the scoring associated with the bins.
• Figure 25 illustrates alternative embodiments of the example operational flow 2400 of Figure 24.
• Figure 25 illustrates an example embodiment where operation 2404 may include at least one additional operation 2502.
• Operation 2502 illustrates adding a token to a time bin associated with at least one received candidate.
• a data structure associated with a candidate/suspect may include an arbitrary time bin grouped by an arbitrary time.
• Figure 26 illustrates alternative embodiments of the example operational flow 2400 of Figure 20.
• Figure 26 illustrates an example embodiment where operation 2404 may include at least one additional operation. Additional operations may include operation 2602, and/or operation 2604. Further, operational flow 2400 may include at least one additional operation 2606.
• Operation 2602 illustrates determining a relative time, including at least subtracting a candidate time associated with the at least one candidate from an arbitrary time associated with the at least one cue. For example, as shown in and/or described with respect to Figures 1 through 9, a time offset of a video segment associated with a candidate is subtracted from an arbitrary time associated with an epoch time related to the cue received from a client system (television, set-top box, or article, machine, or composition of matter displaying and/or providing and/or receiving video content).
• a client system television, set-top box, or article, machine, or composition of matter displaying and/or providing and/or receiving video content.
• Operation 2604 illustrates adding a token to a time bin associated with the candidate based at least partially on the determined relative time. For example, as shown in and/or described with respect to Figures 1 through 9, when a cue point associated with the client system matches or nearly matches a reference cue point associated with a media database, a token may be added to a bin, which may include incrementing a value associated with a bin or another means of tracking bin operations.
• Operation 2606 illustrates removing one or more tokens from a time bin based at least partially on a time period elapsing.
• a bin may be leaky such that data and/or tokens associated with old suspects/candidates may be release from the bin, which may include decrementing a value associated with a bin or another means of tracking bin operations.
• pixel locations may relate to one or many colors and/or color spaces/models (e.g. red, blue, green; red, blue, green, and yellow; cyan, magenta, yellow, and black; a single pixel value uniquely identifying a color e.g. a 24-bit value associated with a pixel location; hue, saturation, brightness; etc.). Differing numbers of pixels in a patch may be used, and the patch does not have to be a square patch. Further, resolution of the video buffer of the client system may vary. Resolutions and/or color densities at the client system and the ingest system may vary.
• the system may be operable with various raster resolutions, including but not limited to 1920 by 1080, 3840 by 2160, 1440 x 1080, 1366 x 768, or other resolutions. It is expected that over the next two decades, increases in pixel resolution of common programming, televisions, and/or client systems will occur; the same basic operations may be utilized although pixel patch number, size, sampling rate, or other aspects may vary. Further, an up-conversion, down-conversion, or other transformation operation associated with resolution and/or color density may occur and/or be interposed between other operations described herein. [00163]
• Figure 27 illustrates an example system 2700 in which embodiments may be implemented.
• the system 2700 includes one or more computing devices 2702.
• the system 2700 also illustrates a fabric 2704 for facilitating communications among one or more computing devices and one or more client devices 2706.
• the system 2700 also illustrates one or more client devices 2706.
• the one or more client devices may be among the one or more computing devices.
• the system 2700 also illustrates at least one non-transitory computer- readable medium 2708.
• 2708 may include one or more instructions 2710 that, when executed on at least some of the one or more computing devices, cause at least some of the one or more computing devices to at least receive at least one stream of rasterized video; create at least one hash value associated with at least one sample of at least one received rasterized video stream; determine at least one database sector for storing a created at least one hash value; and store a created at least one hash value on at least one determined database sector.
• the one or more instructions may be executed on a single computing device.
• some portions of the one or more instructions may be executed by a first plurality of the one or more computing devices, while other portions of the one or more instructions may be executed by a second plurality of the one or more computing devices.
• Figure 28 illustrates an example system 2800 in which embodiments may be implemented.
• the system 2800 includes one or more computing devices 2802.
• the system 2800 also illustrates a fabric 2804 for facilitating communications among one or more computing devices and one or more client devices 2806.
• the system 2800 also illustrates one or more client devices 2806.
• the one or more client devices may be among the one or more computing devices.
• the system 2800 also illustrates at least one non-transitory computer- readable medium 2808.
• 2808 may include one or more instructions 2810 that, when executed on at least some of the one or more computing devices, cause at least some of the one or more computing devices to at least receive one or more indications associated with at least one video buffer of at least one client system; determine a cue based at least partially on the at least one video buffer and at least one epoch time associated with the at least one video buffer, wherein one or more of at least one operand or at least one function associated with determining the cue is also utilized in an associated media storage operation; reference a number of most significant bits of a determined cue to determine a database sector; and return at least one indication of at least one candidate from a determined database sector based at least partially on a determined cue.
• the one or more instructions may be executed on a single computing device. In other embodiments, some portions of the one or more instructions may be executed by a first plurality of the one or more computing devices, while other portions of the one or more instructions may be executed by a second plurality of the one or more computing devices.
• Figure 29 illustrates an example system 2900 in which embodiments may be implemented.
• the system 2900 includes one or more computing devices 2902.
• the system 2900 also illustrates a fabric 2904 for facilitating communications among one or more computing devices and one or more client devices 2906.
• the system 2900 also illustrates one or more client devices 2906.
• the one or more client devices may be among the one or more computing devices.
• the system 2900 also illustrates at least one non-transitory computer- readable medium 2908.
• 2908 may include one or more instructions 2910 that, when executed on at least some of the one or more computing devices, cause at least some of the one or more computing devices to at least receive at least one indication of at least one candidate and at least one indication of at least one cue; add a token to a bin associated with at least one received candidate; and determine whether a number of tokens in a bin exceeds a value associated with a probability that a client system is receiving a particular video segment associated with at least one received cue and, if the number of tokens in a bin exceeds a value associated with a probability that a client system is receiving a particular video segment associated with at least one received cue, returning at least some data associated with the particular video segment based at least partially on the bin.
• the one or more instructions may be executed on a single computing device. In other embodiments, some portions of the one or more instructions may be executed by a first plurality of the one or more computing devices, while other portions of the one or more instructions may be executed by a second plurality of the one or more computing devices.
• Certain aspects of the present invention include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions of the present invention could be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by real-time network operating systems.
• the present invention also relates to an apparatus for performing the operations herein.
• This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer.
• a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus.
• computers or computing means referred to in the specification may include a single processor or may employ multiple-processor designs for increased computing capability.
• Embodiments of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible program carrier for execution by, or to control the operation of, data processing apparatus.
• the computer readable medium can be a machine readable storage device, a machine readable storage substrate, a memory device, or a combination of one or more of them.
• a computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
• a computer program does not necessarily correspond to a file in a file system.
• a program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code).
• a computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a suitable communication network.
• the essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data.
• a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
• mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
• processors suitable for the execution of a computer program include, by way of example only and without limitation, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer.
• a processor will receive instructions and data from a read only memory or a random access memory or both.
• a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.
• a display device e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor
• a keyboard and a pointing device e.g., a mouse or a trackball
• Other kinds of devices can be used to provide for interaction with a user as well.
• feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback
• input from the user can be received in any form, including acoustic, speech, or tactile input.
• Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back end component(s) including one or more data servers, or that includes one or more middleware components such as application servers, or that includes a front end component such as a client computer having a graphical user interface or a Web browser through which a user or administrator can interact with some implementations of the subject matter described is this specification, or any combination of one or more such back end, middleware, or front end components.
• the components of the system can be interconnected by any form or medium of digital data communication, such as a communication network.
• the computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client server relationship to each other.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Databases & Information Systems (AREA)
• Theoretical Computer Science (AREA)
• Library & Information Science (AREA)
• Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Data Mining & Analysis (AREA)
• Human Computer Interaction (AREA)
• Business, Economics & Management (AREA)
• Marketing (AREA)
• Computer Security & Cryptography (AREA)
• Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• Television Signal Processing For Recording (AREA)

Priority Applications (23)

Applications Claiming Priority (18)

Publications (1)

Family

ID=54258939

Family Applications (1)

Country Status (7)

Cited By (29)

Families Citing this family (1)

Citations (6)

Family Cites Families (21)

• 2014
  • 2014-03-17 CN CN201480017043.9A patent/CN105144141B/zh active Active
  • 2014-03-17 MX MX2015012510A patent/MX356884B/es active IP Right Grant
  • 2014-03-17 BR BR112015023369-4A patent/BR112015023369B1/pt active IP Right Grant
  • 2014-03-17 MX MX2020001441A patent/MX2020001441A/es unknown
  • 2014-03-17 WO PCT/US2014/030782 patent/WO2014145929A1/en active Application Filing
  • 2014-03-17 CA CA2906199A patent/CA2906199C/en active Active
  • 2014-03-17 CN CN201811395356.4A patent/CN110083739B/zh active Active
• 2015
  • 2015-09-11 CL CL2015002621A patent/CL2015002621A1/es unknown
• 2016
  • 2016-05-20 HK HK16105782.3A patent/HK1217794A1/zh unknown

Patent Citations (6)

Also Published As

Similar Documents

Legal Events